Reading files larger than 4GB using c++ stl

Question

A few weeks back I was using std::ifstream to read in some files and it was failing immediately on open because the file was larger than 4GB. At the time I couldnt find a decent answer as to why it was limited to 32 bit files sizes, so I wrote my own using native OS API.

So, my question then: Is there a way to handle files greater than 4GB in size using std::ifstream/std::ostream (IE: standard c++)

EDIT: Using the STL implementation from the VC 9 compiler (Visual Studio 2008). EDIT2: Surely there has to be standard way to support file sizes larger than 4GB.

Answer 1

I ran into this problem several years ago using gcc on Linux. The OS supported large files, and the C library (fopen, etc) supported it, but the C++ standard library didn't. I turned out that I had to recompile the C++ standard library using the correct compiler flags.

Answer 2

From the Standard point of view, there is nothing that prevents this. However, in reality, most 32bit implementations use 32bit for std::size_t. Now, the C++ standard mandates that the standard allocator in the C++ Standard Library uses std::size_t as the size quantity. Thus, you are limited to 2^32 bytes of storage for containers, strings and stuff. The situation could be another for std::off_t, i don't know exactly what is going on there.

You have to use the native API of the OS directly, or some library wrapping it, to be able to do that, without having to trust the Standard Library implementations, which are largely implementation-dependent.

Answer 3

At least in VS2013, the C++ standard filestream works with large files(>4GBytes) well.

I tested on VS2013 (with update3).

int64_t file_pos = 4LL * 1024 * 1024 * 1024 + 1;
file.seekp( file_pos, SEEK_SET );
assert( file );
cout << "cur pos: " << file.tellp() << endl; // the output is: 4294967297(4GB + 1)

The following link is an additional confirm that it's a bug and has been fixed: https://connect.microsoft.com/VisualStudio/feedback/details/627639/std-fstream-use-32-bit-int-as-pos-type-even-on-x64-platform

for short: Stephan T. Lavavej (Visual C++ Libraries Developer) said

We've fixed it, and the fix will be available in VC11 ... so large file support should work correctly now (regardless of x86/x64 platform)

Answer 4

Apparently it depends on how off_t is implemented by the library.

#include <streambuf>
__int64_t temp=std::numeric_limits<std::streamsize>::max();

gives you what the current max is.

STLport supports larger files.

Answer 5

If you can move yourself away from using only standard C++, then you might be interested in boost::iostreams.