Does fread fail for large files?

Question

I have to analyze a 16 GB file. I am reading through the file sequentially using fread() and fseek(). Is it feasible? Will fread() work for such a large file?

Answer 1

If you're on a POSIX-ish system, you'll need to make sure you've built your program with 64-bit file offset support. POSIX mandates (or at least allows, and most systems enforce this) the implementation to deny IO operations on files whose size don't fit in off_t, even if the only IO being performed is sequential with no seeking.

On Linux, this means you need to use -D_FILE_OFFSET_BITS=64 on the gcc command line.

Answer 2

It depends on what you want to do. If you want to read the whole 16GB of data in memory, then chances are that you'll run out of memory or application heap space.

Rather read the data chunk by chunk and do processing on those chunks (and free resources when done).

But, besides all this, decide which approach you want to do (using fread() or istream, etc.) and do some test cases to see which works better for you.

Answer 3

If implemented correctly this shouldn't be a problem. I assume by sequentially you mean you're looking at the file in discrete chunks and advancing your file pointer.

Check out http://www.computing.net/answers/programming/using-fread-with-a-large-file-/10254.html

It sounds like he was doing nearly the same thing as you.

Answer 4

You don't mention a language, so I'm going to assume C.

I don't see any problems with fread, but fseek and ftell may have issues.

Those functions use long int as the data type to hold the file position, rather than something intelligent like fpos_t or even size_t. This means that they can fail to work on a file over 2 GB, and can certainly fail on a 16 GB file.

You need to see how big long int is on your platform. If it's 64 bits, you're fine. If it's 32, you are likely to have problems when using ftell to measure distance from the start of the file.

Consider using fgetpos and fsetpos instead.

Answer 5

Thanks for the response. I figured out where I was going wrong. fseek() and ftell() do not work for files larger than 4GB. I used _fseeki64() and _ftelli64() and it is working fine now.