Objective-C: Reading a file line by line

Question

What is the appropriate way of dealing with large text files in Objective-C? Let's say I need to read each line separately and want to treat each line as an NSString. What is the most efficient way of doing this?

One solution is using the NSString method:

+ (id)stringWithContentsOfFile:(NSString *)path 
      encoding:(NSStringEncoding)enc 
      error:(NSError **)error 

and then split the lines with a newline separator, and then iterate over the elements in the array. However, this seems fairly inefficient. Is there no easy way to treat the file as a stream, enumerating over each line, instead of just reading it all in at once? Kinda like Java's java.io.BufferedReader.

Answer 1

Just like @porneL said, the C api is very handy.

NSString* fileRoot = [[NSBundle mainBundle] pathForResource:@"record" ofType:@"txt"];
FILE *file = fopen([fileRoot UTF8String], "r");
char buffer[256];
while (fgets(buffer, 256, file) != NULL){
    NSString* result = [NSString stringWithUTF8String:buffer];
    NSLog(@"%@",result);
}

Answer 2

As others have answered both NSInputStream and NSFileHandle are fine options, but it can also be done in a fairly compact way with NSData and memory mapping:

BRLineReader.h

#import <Foundation/Foundation.h>

@interface BRLineReader : NSObject

@property (readonly, nonatomic) NSData *data;
@property (readonly, nonatomic) NSUInteger linesRead;
@property (strong, nonatomic) NSCharacterSet *lineTrimCharacters;
@property (readonly, nonatomic) NSStringEncoding stringEncoding;

- (instancetype)initWithFile:(NSString *)filePath encoding:(NSStringEncoding)encoding;
- (instancetype)initWithData:(NSData *)data encoding:(NSStringEncoding)encoding;
- (NSString *)readLine;
- (NSString *)readTrimmedLine;
- (void)setLineSearchPosition:(NSUInteger)position;

@end

BRLineReader.m

#import "BRLineReader.h"

static unsigned char const BRLineReaderDelimiter = '\n';

@implementation BRLineReader
{
    NSRange _lastRange;
}

- (instancetype)initWithFile:(NSString *)filePath encoding:(NSStringEncoding)encoding
{
    self = [super init];
    if (self) {
        NSError *error = nil;
        _data = [NSData dataWithContentsOfFile:filePath options:NSDataReadingMappedAlways error:&error];
        if (!_data) {
            NSLog(@"%@", [error localizedDescription]);
        }
        _stringEncoding = encoding;
        _lineTrimCharacters = [NSCharacterSet whitespaceAndNewlineCharacterSet];
    }

    return self;
}

- (instancetype)initWithData:(NSData *)data encoding:(NSStringEncoding)encoding
{
    self = [super init];
    if (self) {
        _data = data;
        _stringEncoding = encoding;
        _lineTrimCharacters = [NSCharacterSet whitespaceAndNewlineCharacterSet];
    }

    return self;
}

- (NSString *)readLine
{
    NSUInteger dataLength = [_data length];
    NSUInteger beginPos = _lastRange.location + _lastRange.length;
    NSUInteger endPos = 0;
    if (beginPos == dataLength) {
        // End of file
        return nil;
    }

    unsigned char *buffer = (unsigned char *)[_data bytes];
    for (NSUInteger i = beginPos; i < dataLength; i++) {
        endPos = i;
        if (buffer[i] == BRLineReaderDelimiter) break;
    }

    // End of line found
    _lastRange = NSMakeRange(beginPos, endPos - beginPos + 1);
    NSData *lineData = [_data subdataWithRange:_lastRange];
    NSString *line = [[NSString alloc] initWithData:lineData encoding:_stringEncoding];
    _linesRead++;

    return line;
}

- (NSString *)readTrimmedLine
{
    return [[self readLine] stringByTrimmingCharactersInSet:_lineTrimCharacters];
}

- (void)setLineSearchPosition:(NSUInteger)position
{
    _lastRange = NSMakeRange(position, 0);
    _linesRead = 0;
}

@end

Answer 3

I found response by @lukaswelte and code from Dave DeLong very helpful. I was looking for a solution to this problem but needed to parse large files by \r\n not just \n.

The code as written contains a bug if parsing by more than one character. I've changed the code as below.

.h file:

#import <Foundation/Foundation.h>

@interface FileChunkReader : NSObject {
    NSString * filePath;

    NSFileHandle * fileHandle;
    unsigned long long currentOffset;
    unsigned long long totalFileLength;

    NSString * lineDelimiter;
    NSUInteger chunkSize;
}

@property (nonatomic, copy) NSString * lineDelimiter;
@property (nonatomic) NSUInteger chunkSize;

- (id) initWithFilePath:(NSString *)aPath;

- (NSString *) readLine;
- (NSString *) readTrimmedLine;

#if NS_BLOCKS_AVAILABLE
- (void) enumerateLinesUsingBlock:(void(^)(NSString*, BOOL *))block;
#endif

@end

.m file:

#import "FileChunkReader.h"

@interface NSData (DDAdditions)

- (NSRange) rangeOfData_dd:(NSData *)dataToFind;

@end

@implementation NSData (DDAdditions)

- (NSRange) rangeOfData_dd:(NSData *)dataToFind {

    const void * bytes = [self bytes];
    NSUInteger length = [self length];

    const void * searchBytes = [dataToFind bytes];
    NSUInteger searchLength = [dataToFind length];
    NSUInteger searchIndex = 0;

    NSRange foundRange = {NSNotFound, searchLength};
    for (NSUInteger index = 0; index < length; index++) {
        if (((char *)bytes)[index] == ((char *)searchBytes)[searchIndex]) {
            //the current character matches
            if (foundRange.location == NSNotFound) {
                foundRange.location = index;
            }
            searchIndex++;
            if (searchIndex >= searchLength)
            {
                return foundRange;
            }
        } else {
            searchIndex = 0;
            foundRange.location = NSNotFound;
        }
    }

    if (foundRange.location != NSNotFound
        && length < foundRange.location + foundRange.length )
    {
        // if the dataToFind is partially found at the end of [self bytes],
        // then the loop above would end, and indicate the dataToFind is found
        // when it only partially was.
        foundRange.location = NSNotFound;
    }

    return foundRange;
}

@end

@implementation FileChunkReader

@synthesize lineDelimiter, chunkSize;

- (id) initWithFilePath:(NSString *)aPath {
    if (self = [super init]) {
        fileHandle = [NSFileHandle fileHandleForReadingAtPath:aPath];
        if (fileHandle == nil) {
            return nil;
        }

        lineDelimiter = @"\n";
        currentOffset = 0ULL; // ???
        chunkSize = 128;
        [fileHandle seekToEndOfFile];
        totalFileLength = [fileHandle offsetInFile];
        //we don't need to seek back, since readLine will do that.
    }
    return self;
}

- (void) dealloc {
    [fileHandle closeFile];
    currentOffset = 0ULL;

}

- (NSString *) readLine {
    if (currentOffset >= totalFileLength)
    {
        return nil;
    }

    @autoreleasepool {

        NSData * newLineData = [lineDelimiter dataUsingEncoding:NSUTF8StringEncoding];
        [fileHandle seekToFileOffset:currentOffset];
        unsigned long long originalOffset = currentOffset;
        NSMutableData *currentData = [[NSMutableData alloc] init];
        NSData *currentLine = [[NSData alloc] init];
        BOOL shouldReadMore = YES;


        while (shouldReadMore) {
            if (currentOffset >= totalFileLength)
            {
                break;
            }

            NSData * chunk = [fileHandle readDataOfLength:chunkSize];
            [currentData appendData:chunk];

            NSRange newLineRange = [currentData rangeOfData_dd:newLineData];

            if (newLineRange.location != NSNotFound) {

                currentOffset = originalOffset + newLineRange.location + newLineData.length;
                currentLine = [currentData subdataWithRange:NSMakeRange(0, newLineRange.location)];

                shouldReadMore = NO;
            }else{
                currentOffset += [chunk length];
            }
        }

        if (currentLine.length == 0 && currentData.length > 0)
        {
            currentLine = currentData;
        }

        return [[NSString alloc] initWithData:currentLine encoding:NSUTF8StringEncoding];
    }
}

- (NSString *) readTrimmedLine {
    return [[self readLine] stringByTrimmingCharactersInSet:[NSCharacterSet whitespaceAndNewlineCharacterSet]];
}

#if NS_BLOCKS_AVAILABLE
- (void) enumerateLinesUsingBlock:(void(^)(NSString*, BOOL*))block {
    NSString * line = nil;
    BOOL stop = NO;
    while (stop == NO && (line = [self readLine])) {
        block(line, &stop);
    }
}
#endif

@end

Answer 4

I am adding this because all other answers I tried fell short one way or another. The following method can handle large files, arbitrary long lines, as well as empty lines. It has been tested with actual content and will strip out newline character from the output.

- (NSString*)readLineFromFile:(FILE *)file
{
    char buffer[4096];
    NSMutableString *result = [NSMutableString stringWithCapacity:1000];

    int charsRead;
    do {
        if(fscanf(file, "%4095[^\r\n]%n%*[\n\r]", buffer, &charsRead) == 1) {
            [result appendFormat:@"%s", buffer];
        }
        else {
            break;
        }
    } while(charsRead == 4095);

    return result.length ? result : nil;
}

Credit goes to @Adam Rosenfield and @sooop

Answer 5

That's a great question. I think @Diederik has a good answer, although it's unfortunate that Cocoa doesn't have a mechanism for exactly what you want to do.

NSInputStream allows you to read chunks of N bytes (very similar to java.io.BufferedReader), but you have to convert it to an NSString on your own, then scan for newlines (or whatever other delimiter) and save any remaining characters for the next read, or read more characters if a newline hasn't been read yet. (NSFileHandle lets you read an NSData which you can then convert to an NSString, but it's essentially the same process.)

Apple has a Stream Programming Guide that can help fill in the details, and this SO question may help as well if you're going to be dealing with uint8_t* buffers.

If you're going to be reading strings like this frequently (especially in different parts of your program) it would be a good idea to encapsulate this behavior in a class that can handle the details for you, or even subclassing NSInputStream (it's designed to be subclassed) and adding methods that allow you to read exactly what you want.

For the record, I think this would be a nice feature to add, and I'll be filing an enhancement request for something that makes this possible. :-)

---

Edit: Turns out this request already exists. There's a Radar dating from 2006 for this (rdar://4742914 for Apple-internal people).

Answer 6

from @Adam Rosenfield's answer, the formatting string of fscanf would be changed like below:

"%4095[^\r\n]%n%*[\n\r]"

it will work in osx, linux, windows line endings.