Problems with reentrant Flex and Bison

Question

I'm learning how to use reentrant Bison and Flex together. I already got a simple calculator working without the reentrant capability. However when I activated the reentrant feature and made the necessary modifications, I couldn't get this to work.

Here is the code:

scanner.l

%{
#include <stdio.h>
#include "parser.tab.h"
%}

%option 8bit reentrant bison-bridge
%option warn noyywrap nodefault
%option header-file="lex.yy.h"

DIGIT [0-9]

%%

"+"    { return ADD; }
"-"    { return SUB; }
"*"    { return MUL; }
"/"    { return DIV; }
{DIGIT}+ { *yylval = atof(yytext); return NUM; }
\n     { return EOL; }
[ \t]  {  }
.      { printf("What is this: %s.\n", yytext); }
%%

parser.y

%{
#include <stdio.h>
#include "lex.yy.h"

void yyerror(yyscan_t scanner, char const *msg);
%}

%define api.value.type {double}
%define parse.error verbose
%define api.pure 
%lex-param {yyscan_t scanner}
%parse-param {yyscan_t scanner}

%token NUM EOL                  
%left ADD SUB
%left MUL DIV

%%

input: %empty
| input line
;

line: EOL { printf("|> ");}
| exp EOL { printf("|R> %.4lf\n", $exp); }
;

exp: NUM { $$ = $1; }
| exp ADD exp { $$ = $1 + $3; }
| exp SUB exp { $$ = $1 - $3; }
| exp MUL exp { $$ = $1 * $3; }
| exp DIV exp { $$ = $1 / $3; }
;

%%

void yyerror(yyscan_t scanner, char const *msg) {
    fprintf(stderr, "Error: %s\n", msg);
}

main.c

#include <stdio.h>
#include "parser.tab.h"
#include "lex.yy.h"

int main(void) {

  yyscan_t scanner;

  yylex_init(&scanner);
  yyset_in(stdin, scanner);

  yyparse(scanner);

  yylex_destroy(scanner);

  return 0;
}

and this is the Makefile I'm using:

all: calc.x

parser.tab.c parser.tab.h: parser.y
    bison -d parser.y

lex.yy.c lex.yy.h: scanner.l parser.tab.h
    flex scanner.l

calc.x: lex.yy.c lex.yy.h parser.tab.c parser.tab.h
    gcc main.c parser.tab.c lex.yy.c -o calc.x

clean:
    rm calc.x lex.yy.c lex.yy.h parser.tab.c parser.tab.h *.o

Running make, I got the following error:

In file included from main.c:2:0:
parser.tab.h:66:14: error: unknown type name ‘yyscan_t’
 int yyparse (yyscan_t scanner);
              ^
main.c: In function ‘main’:
main.c:12:3: warning: implicit declaration of function ‘yyparse’ [-Wimplicit-function-declaration]
   yyparse(scanner);
   ^
In file included from parser.y:5:0:
lex.yy.h:282:1: error: unknown type name ‘YYSTYPE’
 YYSTYPE * yyget_lval (yyscan_t yyscanner );
 ^
lex.yy.h:284:18: error: unknown type name ‘YYSTYPE’
 void yyset_lval (YYSTYPE * yylval_param ,yyscan_t yyscanner );
                  ^
lex.yy.h:332:17: error: unknown type name ‘YYSTYPE’
                (YYSTYPE * yylval_param ,yyscan_t yyscanner);
                 ^
parser.tab.c: In function ‘yyparse’:
parser.tab.c:1130:16: warning: implicit declaration of function ‘yylex’ [-Wimplicit-function-declaration]
       yychar = yylex (&yylval, scanner);
                ^
Makefile:10: recipe for target 'calc.x' failed
make: *** [calc.x] Error 1

But I don't understand the origin of this error and warning messages, for instance:

main.c:12:3: warning: implicit declaration of function ‘yyparse’

But yyparse is already defined at parser.tab.h and it's being included in main.c. Another example:

parser.tab.h:66:14: error: unknown type name ‘yyscan_t’

And inside parser.y, I'm including the scanner header lex.yy.h.

I have found these solutions on the Internet:

• Configuring Bison and Flex without global or static variable
• Reentrant parsers with Flex and Bison
• Make a reentrant parser with Flex and Bison
• Writing re-entrant lexer with Flex
• Implementing re-entrant parsers in Bison and Flex

But none of them work, resulting in similar errors. If someone could guide me in this quest I'll be grateful.

Software version

OS: Debian (testing), Bison: 3.0.4, Flex: 2.5.39, GCC: 5.2.1, Make: 4.0.

Answer 1

I found a solution after tinkering a bit. So the problems arise from a circular dependency between flex and bison.

The parser generated call flex routine in this way:

yychar = yylex (&yylval, scanner);

So in the bison input we must include the scanner header file lex.yy.h and it's define as:

int yylex (YYSTYPE * yylval_param ,yyscan_t yyscanner);

But YYSTYPE is defined inside the parser header parser.tab.h, in my case i said to bison that my type will be double:

typedef double YYSTYPE;

Now the solution. Inside scanner.l you must include the parser headers so that flex can return correct tokens (nothing changed).

But inside the parser.y you must include both headers file, if you include only the lex.yy.h it will complain:

lex.yy.h:282:1: error: unknown type name ‘YYSTYPE‘

because YYSTYPE is defined inside parser.tab.h. And finally, for some reason, the bison parser doesn't know what yyscan_t even including the lexer header:

error: unknown type name ‘yyscan_t’

One workaround is defining it to void:

%lex-param {void *scanner}
%parse-param {void *scanner}

see yyscan_t definition: flex yyscan_t

So here is the final result:

scanner.l

%{
#include <stdio.h>
#include "parser.tab.h"
%}

%option 8bit reentrant bison-bridge
%option warn noyywrap nodefault
%option header-file="lex.yy.h"

//rest of the scanner

parser.y

%{
#include <stdio.h>
#include "parser.tab.h"
#include "lex.yy.h"

void yyerror(yyscan_t scanner, char const *msg);
%}

%define api.value.type {double}
%define parse.error verbose
%define api.pure 
%lex-param {void *scanner}
%parse-param {void *scanner}

//rest of the input

main.c

#include <stdio.h>

#include "parser.tab.h"
#include "lex.yy.h"

int main(void) {

  yyscan_t scanner;

  yylex_init(&scanner);
  yyset_in(stdin, scanner);

  yyparse(scanner);

  yylex_destroy(scanner);

  return 0;
}

Answer 2

Accepted answer by Fabricio Sanches helped me to resolve two issues:

1. error: unknown type name ‘yyscan_t’ has indeed been solved by changing to void *.

2. Conflicts related to circular dependencies dictate very strict order of imports:

Your code where Flex/Bison's yyparse is called:

#import "parser.h"
#import "lexer.h"

Flex (Lexer.lm):

%{
#import "parser.h"
%}

Bison (Parser.ym):

%{
#import "parser.h"
#import "lexer.h"
%}

---

I have written blog post about the process of creation of reentrant parser using Flex and Bison on Mac OS with example of its integration to Xcode project: Reentrant parser using Flex and Bison.