read the files one by one in a zip file using bash

Question

I want to open the files inside a .zip file and read them. In this zip file, I have numerous .gz files, like a.dat.gz, b.dat.gz, and so on.

My code so far:

for i in $(unzip -p sample.zip)
do
    for line in $(zcat "$i")
    do
        # do some stuff here
    done
done

Answer 1

You are correct in needing two loops. First, you need a list of files inside the archive. Then, you need to iterate within each of those files.

unzip -l sample.zip |sed '
  /^ *[0-9][0-9]* *2[0-9-]*  *[0-9][0-9]:[0-9][0-9]  */!d; s///
' |while IFS= read file
  unzip -p sample.zip "$file" |gunzip -c |while IFS= read line
    # do stuff to "$line" here
  done
done

This assumes that each file in the zip archive is itself a gzip archive. You'll otherwise get an error from gunzip.

Code walk

unzip -l archive.zip will list the contents. Its raw output looks like this:

Archive:  test.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        9  2017-08-24 13:45   1.txt
        9  2017-08-24 13:45   2.txt
---------                     -------
       18                     2 files

We therefore need to parse it. I've chosen to parse with sed because it's fast, simple, and preserves whitespace properly (what if you have files with tabs in their names?) Note, this will not work if files have line breaks in them. Don't do that.

The sed command uses a regex (explanation here) to match the entirety of lines containing file names except for the file names themselves. When the matcher fires, sed is told not to delete (!d), which really tells sed to skip anything that does not match (like the title line). A second command, s///, tells sed to replace the previously matched text with an empty string, therefore the output is one file name per line. This gets piped into a while loop as $file. (The IFS= part before read prevents spaces from being stripped from either end, see the comments below.)

We can then unzip just the file we're iterating on, again using unzip -p to get it printed to standard output so it can be stored in the inner while loop as $line.

Experimental simplification

I'm not sure how reliable this would be, but you might be able to do this more simply as:

unzip -p sample.zip |gunzip -c |while read line
  # do stuff to "$line"
done

This should work because unzip -p archive spits out the contents of each file in the archive, all concatenated together without any delimiters or metadata (like the file name) and because the gzip format accepts concatenating archives together (see my notes on concatenated archives), so the gunzip -c pipeline command sees raw gzip data and decompresses it out on the console, which is then passed to the shell's while loop. You will lack file boundaries and names in this approach, but it's much faster.

Answer 2

This is harder than you might think to do robustly in shell. (The existing answer works in the common case, but archives with surprising filenames included will confuse it). The better option is to use a language with native zip file support -- such as Python. (This can also have the advantage of not needing to open your input file more than once!)

If the individual files are small enough that you can fit a few copies of each in memory, the following will work nicely:

read_files() {
  python -c '
import sys, zipfile, zlib

zf = zipfile.ZipFile(sys.argv[1], "r")
for content_file in zf.infolist():
    content = zlib.decompress(zf.read(content_file), zlib.MAX_WBITS|32)
    for line in content.split("\n")[:-1]:
        sys.stdout.write("%s\0%s\0" % (content_file.filename, line))
' "$@"
}

while IFS= read -r -d '' filename && IFS= read -r -d '' line; do
  printf 'From file %q, read line: %s\n' "$filename" "$line"
done < <(read_files yourfile.zip)

---

If you really want to split the file-listing and file-reading operations off from each other, doing that robustly might look like:

### Function: Extract a zip's content list in NUL-delimited form
list_files() {
  python -c '
import sys, zipfile, zlib

zf = zipfile.ZipFile(sys.argv[1], "r")
for content_file in zf.infolist():
    sys.stdout.write("%s\0" % (content_file.filename,))
' "$@"
}

### Function: Extract a single file's contents from a zip file
read_file() {
  python -c '
import sys, zipfile, zlib

zf = zipfile.ZipFile(sys.argv[1], "r")
sys.stdout.write(zf.read(sys.argv[2]))
' "$@"
}

### Main loop
process_zip_contents() {
  local zipfile=$1
  while IFS= read -r -d '' filename; do
    printf 'Started file: %q\n' "$filename"
    while IFS= read -r line; do
      printf '  Read line: %s\n' "$line"
    done < <(read_file "$zipfile" "$filename" | gunzip -c)
  done < <(list_files "$zipfile")
}

To smoketest the above -- if an input file is created as follows:

printf '%s\n' '1: line one' '1: line two' '1: line three' | gzip > one.gz
printf '%s\n' '2: line one' '2: line two' '2: line three' | gzip > two.gz
cp one.gz 'name
with
newline.gz'
zip test.zip one.gz two.gz $'name\nwith\nnewline.gz'
process_zip_contents test.zip

...then we have the following output:

Started file: $'name\nwith\nnewline.gz'
  Read line: 1:line one
  Read line: 1:line two
  Read line: 1:line three
Started file: one.gz
  Read line: 1: line one
  Read line: 1: line two
  Read line: 1: line three
Started file: two.gz
  Read line: 2: line one
  Read line: 2: line two
  Read line: 2: line three