I'm working on building a C# program to read the RDF data in the Google Freebase data dump. To start out, I've written a simple loop to simply read the file and get a count of the Triples. However, instead of getting the 1.9 billion count as stated in the documentation page (referred above), my program is counting only about 11.5 million and then exiting. The relevant portion of the source code is given below (takes about 30 seconds to run).
What am I missing here?
// Simple reading through the gz file
try
{
using (FileStream fileToDecompress = File.Open(@"C:\Users\Krishna\Downloads\freebase-rdf-2014-02-16-00-00.gz", FileMode.Open))
{
int tupleCount = 0;
string readLine = "";
using (GZipStream decompressionStream = new GZipStream(fileToDecompress, CompressionMode.Decompress))
{
StreamReader sr = new StreamReader(decompressionStream, detectEncodingFromByteOrderMarks: true);
while (true)
{
readLine = sr.ReadLine();
if (readLine != null)
{
tupleCount++;
if (tupleCount % 1000000 == 0)
{ Console.WriteLine(DateTime.Now.ToShortTimeString() + ": " + tupleCount.ToString()); }
}
else
{ break; }
}
Console.WriteLine("Tuples: " + tupleCount.ToString());
}
}
}
catch (Exception ex)
{ Console.WriteLine(ex.Message); }
(I tried using GZippedNTriplesParser
in dotNetRdf
to read the data by building on this recommendation, but that seems to be choking on an RdfParseException
right at the beginning (Tab delimiters? UTF-8??). So, for the moment, trying to roll my own).