Can Bison parse UTF-8 characters?

Question

I'm trying to make a Bison parser to handle UTF-8 characters. I don't want the parser to actually interpret the Unicode character values, but I want it to parse the UTF-8 string as a sequence of bytes.

Right now, Bison generates the following code which is problematic:

  if (yychar <= YYEOF)
    {
      yychar = yytoken = YYEOF;
      YYDPRINTF ((stderr, "Now at end of input.\n"));
    }

The problem is that many bytes of the UTF-8 string will have a negative value, and Bison interprets negative values as an EOF, and stops.

Is there a way around this?

Answer 1

This is an question from 4 years ago, but I'm facing the same issues and I'd like to share my ideas.

The problem is that in UTF-8 you don't know how many bytes to read. As suggested above you can use your own lexer, and have it either read whole lines, or have it read 4 bytes every time. Then extract the UTF-8 character from that, and read more bytes to complete again to 4 bytes.

Answer 2

flex being the issue here, you might want to take a look at zlex.

Answer 3

bison yes, flex no. The one time I needed a bison parser to work with UTF-8 encoded files I ended up writing my own yylex function.

edit: To help, I used a lot of the Unicode operations available in glib (there's a gunicode type and some file/string manipulation functions that I found useful).