I'm trying to write a CSV parser using Scala parser combinators. The grammar is based on RFC4180. I came up with the following code. It almost works, but I cannot get it to correctly separate different records. What did I miss?
object CSV extends RegexParsers {
def COMMA = ","
def DQUOTE = "\""
def DQUOTE2 = "\"\"" ^^ { case _ => "\"" }
def CR = "\r"
def LF = "\n"
def CRLF = "\r\n"
def TXT = "[^\",\r\n]".r
def file: Parser[List[List[String]]] = ((record~((CRLF~>record)*))<~(CRLF?)) ^^ {
case r~rs => r::rs
}
def record: Parser[List[String]] = (field~((COMMA~>field)*)) ^^ {
case f~fs => f::fs
}
def field: Parser[String] = escaped|nonescaped
def escaped: Parser[String] = (DQUOTE~>((TXT|COMMA|CR|LF|DQUOTE2)*)<~DQUOTE) ^^ { case ls => ls.mkString("")}
def nonescaped: Parser[String] = (TXT*) ^^ { case ls => ls.mkString("") }
def parse(s: String) = parseAll(file, s) match {
case Success(res, _) => res
case _ => List[List[String]]()
}
}
println(CSV.parse(""" "foo", "bar", 123""" + "\r\n" +
"hello, world, 456" + "\r\n" +
""" spam, 789, egg"""))
// Output: List(List(foo, bar, 123hello, world, 456spam, 789, egg))
// Expected: List(List(foo, bar, 123), List(hello, world, 456), List(spam, 789, egg))
Update: problem solved
The default RegexParsers ignore whitespaces including space, tab, carriage return, and line breaks using the regular expression [\s]+
. The problem of the parser above unable to separate records is due to this. We need to disable skipWhitespace mode. Replacing whiteSpace definition to just [ \t]}
does not solve the problem because it will ignore all spaces within fields (thus "foo bar" in the CSV becomes "foobar"), which is undesired. The updated source of the parser is thus
import scala.util.parsing.combinator._
// A CSV parser based on RFC4180
// http://tools.ietf.org/html/rfc4180
object CSV extends RegexParsers {
override val skipWhitespace = false // meaningful spaces in CSV
def COMMA = ","
def DQUOTE = "\""
def DQUOTE2 = "\"\"" ^^ { case _ => "\"" } // combine 2 dquotes into 1
def CRLF = "\r\n" | "\n"
def TXT = "[^\",\r\n]".r
def SPACES = "[ \t]+".r
def file: Parser[List[List[String]]] = repsep(record, CRLF) <~ (CRLF?)
def record: Parser[List[String]] = repsep(field, COMMA)
def field: Parser[String] = escaped|nonescaped
def escaped: Parser[String] = {
((SPACES?)~>DQUOTE~>((TXT|COMMA|CRLF|DQUOTE2)*)<~DQUOTE<~(SPACES?)) ^^ {
case ls => ls.mkString("")
}
}
def nonescaped: Parser[String] = (TXT*) ^^ { case ls => ls.mkString("") }
def parse(s: String) = parseAll(file, s) match {
case Success(res, _) => res
case e => throw new Exception(e.toString)
}
}
What you missed is whitespace. I threw in a couple bonus improvements.
With Scala Parser Combinators library out of the Scala standard library starting from 2.11 there is no good reason not to use the much more performant Parboiled2 library. Here is a version of the CSV parser in Parboiled2's DSL:
The default whitespace for
RegexParsers
parsers is\s+
, which includes new lines. SoCR
,LF
andCRLF
never get a chance to be processed, as it is automatically skipped by the parser.