Large file support not working in C programming

Question

I'm trying to compile a shared object (that is eventually used in Python with ctypes). The command-line used to build the object is:

gcc -Wall -O3 -shared -Wl,-soname,borg_stream -lm -m128bit-long-double -fPIC \
    -D_FILE_OFFSET_BITS=64 -o borg_stream.so data_stream.c data_types.c \
    file_operations.c float_half.c channels.c statistics.c index_stream.c helpers.c

The library builds properly on a 32-bit OS and it does what it needs to for small files. However, it fails the unit tests for files larger than 4GB. In addition, it sets errno to EOVERFLOW when doing an fseek/ftell. However, if I printf sizeof(off_t), it returns 8. If I remove -D_FILE_OFFSET_BITS=64, then it prints 4. So it seems like -D_FILE_OFFSET_BITS is properly doing its job.

Why does large file support still not work? What am I doing wrong?

Answer 1

Use fseeko and ftello. Not fseek and ftell. And certainly not any function with 64 in its name.

Answer 2

Add the option -D_LARGE_FILE_SOURCE=1 to gcc compilation.

fseek64 is a C function. To make it available you'll have to define _FILE_OFFSET_BITS=64 before including the system headers. That will more or less define fseek to behave as actually fseek64. Or you could do it in the compiler arguments e.g. gcc -D_FILE_OFFSET_BITS=64, that you are already doing.

http://www.suse.de/~aj/linux_lfs.html has a good information for large file support on linux:

Compile your programs with gcc -D_FILE_OFFSET_BITS=64. This forces all file access calls to use the 64 bit variants. Several types change also, e.g. off_t becomes off64_t. It's therefore important to always use the correct types and to not use e.g. int instead of off_t in your C code. For portability with other platforms you should use getconf LFS_CFLAGS which will return -D_FILE_OFFSET_BITS=64 on Linux platforms but might return something else on for e.g. on Solaris. For linking, you should use the link flags that are reported via getconf LFS_LDFLAGS. On Linux systems, you do not need special link flags. Define _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE. With these defines you can use the LFS functions like open64 directly. Use the O_LARGEFILE flag with open to operate on large files.

Hope this helps.

Answer 3

fseek and ftell take in an int which is 32-bit, so it is cast and you lose the ability to address space larger than 4GB. Instead use fseeko64 and ftello64 which take a long.