I'm trying to make a Bison parser to handle UTF-8 characters. I don't want the parser to actually interpret the Unicode character values, but I want it to parse the UTF-8 string as a sequence of bytes.
Right now, Bison generates the following code which is problematic:
if (yychar <= YYEOF)
{
yychar = yytoken = YYEOF;
YYDPRINTF ((stderr, "Now at end of input.\n"));
}
The problem is that many bytes of the UTF-8 string will have a negative value, and Bison interprets negative values as an EOF, and stops.
Is there a way around this?
bison yes, flex no. The one time I needed a bison parser to work with UTF-8 encoded files I ended up writing my own yylex function.
edit: To help, I used a lot of the Unicode operations available in glib (there's a gunicode type and some file/string manipulation functions that I found useful).
flex being the issue here, you might want to take a look at zlex.
This is an question from 4 years ago, but I'm facing the same issues and I'd like to share my ideas.
The problem is that in UTF-8 you don't know how many bytes to read. As suggested above you can use your own lexer, and have it either read whole lines, or have it read 4 bytes every time. Then extract the UTF-8 character from that, and read more bytes to complete again to 4 bytes.
