Way to deal with large data files in Wolfram Mathe

Question

I wonder if there exists way to work with large files in Mathematica ? Currently I have a file about 500Mb with table data.

Import["data.txt","Table"];

What is alternate way?

Answer 1

Use OpenRead["file"] which gives you an InputStream object on which you can use Read[stream]. Depending on the formatting of the data file you may need to set custom option values in Read[] for RecordSeparators.

Example:

In[1]:= str = OpenRead["ExampleData/USConstitution.txt"]    
Out[1]= InputStream["ExampleData/USConstitution.txt", 24]   

In[2]:= Read[str, Word]    
Out[2]= "We"    
In[3]:= Read[str, Word]
Out[3]= "the"    
In[4]:= Read[str, Record]
Out[4]= "People of the United States, in Order to form a more perfect Union,"

Answer 2

The function DumpSave can also be helpful for saving large datasets. It saves data in Mathematica's internal format, so it's more efficient in both time and file size.

Answer 3

You could also load your data into a database (for example MySQL) and access it from Mathematica using DatabaseLink