Argument list too long error for rm, cp, mv comman

Question

I have several hundred PDFs under a directory in UNIX. The names of the PDFs are really long (approx. 60 chars).

When I try to delete all PDFs together using the following command:

rm -f *.pdf

I get the following error:

/bin/rm: cannot execute [Argument list too long]

What is the solution to this error? Does this error occur for mv and cp commands as well? If yes, how to solve for these commands?

Answer 1

I found that for extremely large lists of files (>1e6), these answers were too slow. Here is a solution using parallel processing in python. I know, I know, this isn't linux... but nothing else here worked.

(This saved me hours)

# delete files
import os as os
import glob
import multiprocessing as mp

directory = r'your/directory'
os.chdir(directory)


files_names = [i for i in glob.glob('*.{}'.format('pdf'))]

# report errors from pool

def callback_error(result):
    print('error', result)

# delete file using system command
def delete_files(file_name):
     os.system('rm -rf ' + file_name)

pool = mp.Pool(12)  
# or use pool = mp.Pool(mp.cpu_count())


if __name__ == '__main__':
    for file_name in files_names:
        print(file_name)
        pool.apply_async(delete_files,[file_name], error_callback=callback_error)

Answer 2

I had the same problem with a folder full of temporary images that was growing day by day and this command helped me to clear the folder

find . -name "*.png" -mtime +50 -exec rm {} \;

The difference with the other commands is the mtime parameter that will take only the files older than X days (in the example 50 days)

Using that multiple times, decreasing on every execution the day range, I was able to remove all the unnecessary files

Answer 3

If you have similar problems with grep, the easiest solution is stepping one dir back and do a recursive search.

So instead of

grep "something" *

you can use:

cd ..
grep "something" -R search_in_this_dir/

Note it will recursively search subfolders of "search_in_this_dir" directory as well.

Answer 4

tl;dr

It's a kernel limitation on the size of the command line argument. Use a for loop instead.

Origin of problem

This is a system issue, related to execve and ARG_MAX constant. There is plenty of documentation about that (see man execve, debian's wiki).

Basically, the expansion produce a command (with its parameters) that exceeds the ARG_MAX limit. On kernel 2.6.23, the limit was set at 128 kB. This constant has been increased and you can get its value by executing:

getconf ARG_MAX
# 2097152 # on 3.5.0-40-generic

Solution: Using for Loop

Use a for loop as it's recommended on BashFAQ/095 and there is no limit except for RAM/memory space:

for f in *.pdf; do rm "$f"; done

Also this is a portable approach as glob have strong and consistant behavior among shells (part of POSIX spec).

Note: As noted by several comments, this is indeed slower but more maintainable as it can adapt more complex scenarios, e.g. where one want to do more than just one action.

Solution: Using find

If you insist, you can use find but really don't use xargs as it "is dangerous (broken, exploitable, etc.) when reading non-NUL-delimited input":

find . -maxdepth 1 -name '*.pdf' -delete 

Using -maxdepth 1 ... -delete instead of -exec rm {} + allows find to simply execute the required system calls itself without using an external process, hence faster (thanks to @chepner comment).

References

• I'm getting "Argument list too long". How can I process a large list in chunks? @ wooledge
• execve(2) - Linux man page (search for ARG_MAX) ;
• Error: Argument list too long @ Debian's wiki ;
• Why do I get “/bin/sh: Argument list too long” when passing quoted arguments? @ SuperUser

Answer 5

i was facing same problem while copying form source directory to destination

source directory had files ~3 lakcs

i used cp with option -r and it's worked for me

cp -r abc/ def/

it will copy all files from abc to def without giving warning of Argument list too long

Answer 6

To delete all *.pdf in a directory /path/to/dir_with_pdf_files/

mkdir empty_dir        # Create temp empty dir

rsync -avh --delete --include '*.pdf' empty_dir/ /path/to/dir_with_pdf_files/

---

To delete specific files via rsync using wildcard is probably the fastest solution in case you've millions of files. And it will take care of error you're getting.

---

(Optional Step): DRY RUN. To check what will be deleted without deleting. `

rsync -avhn --delete --include '*.pdf' empty_dir/ /path/to/dir_with_pdf_files/

. . .

Click rsync tips and tricks for more rsync hacks