Argument list too long error for rm, cp, mv comman

Question

I have several hundred PDFs under a directory in UNIX. The names of the PDFs are really long (approx. 60 chars).

When I try to delete all PDFs together using the following command:

rm -f *.pdf

I get the following error:

/bin/rm: cannot execute [Argument list too long]

What is the solution to this error? Does this error occur for mv and cp commands as well? If yes, how to solve for these commands?

Answer 1

I only know a way around this. The idea is to export that list of pdf files you have into a file. Then split that file into several parts. Then remove pdf files listed in each part.

ls | grep .pdf > list.txt
wc -l list.txt

wc -l is to count how many line the list.txt contains. When you have the idea of how long it is, you can decide to split it in half, forth or something. Using split -l command For example, split it in 600 lines each.

split -l 600 list.txt

this will create a few file named xaa,xab,xac and so on depends on how you split it. Now to "import" each list in those file into command rm, use this:

rm $(<xaa)
rm $(<xab)
rm $(<xac)

Sorry for my bad english.

Answer 2

I ran into this problem a few times. Many of the solutions will run the rm command for each individual file that needs to be deleted. This is very inefficient:

find . -name "*.pdf" -print0 | xargs -0 rm -rf

I ended up writing a python script to delete the files based on the first 4 characters in the file-name:

import os
filedir = '/tmp/' #The directory you wish to run rm on 
filelist = (os.listdir(filedir)) #gets listing of all files in the specified dir
newlist = [] #Makes a blank list named newlist
for i in filelist: 
    if str((i)[:4]) not in newlist: #This makes sure that the elements are unique for newlist
        newlist.append((i)[:4]) #This takes only the first 4 charcters of the folder/filename and appends it to newlist
for i in newlist:
    if 'tmp' in i:  #If statment to look for tmp in the filename/dirname
        print ('Running command rm -rf '+str(filedir)+str(i)+'* : File Count: '+str(len(os.listdir(filedir)))) #Prints the command to be run and a total file count
        os.system('rm -rf '+str(filedir)+str(i)+'*') #Actual shell command
print ('DONE')

This worked very well for me. I was able to clear out over 2 million temp files in a folder in about 15 minutes. I commented the tar out of the little bit of code so anyone with minimal to no python knowledge can manipulate this code.

Answer 3

If you need to keep a server or system responsive while deleting a huge amount of files, sleep between each delete statement can be a good approach.

find . -name "*.pdf" -print0 | while read -d $'\0' file
do
    rm "$file"
    sleep 0.005 # Sleeps for 5ms, tweak as needed
done

Answer 4

Or you can try:

find . -name '*.pdf' -exec rm -f {} \;

Answer 5

You can create a temp folder, move all the files and sub-folders you want to keep into the temp folder then delete the old folder and rename the temp folder to the old folder try this example until you are confident to do it live:

mkdir testit
cd testit
mkdir big_folder tmp_folder
touch big_folder/file1.pdf
touch big_folder/file2.pdf
mv big_folder/file1,pdf tmp_folder/
rm -r big_folder
mv tmp_folder big_folder

the rm -r big_folder will remove all files in the big_folder no matter how many. You just have to be super careful you first have all the files/folders you want to keep, in this case it was file1.pdf

Answer 6

A bit safer version than using xargs, also not recursive: ls -p | grep -v '/$' | grep '\.pdf$' | while read file; do rm "$file"; done

Filtering our directories here is a bit unnecessary as 'rm' won't delete it anyway, and it can be removed for simplicity, but why run something that will definitely return error?