lexical analysis stops after yy_scan_string() is f

Question

I use flex to make a lexical analyzer. I want to analyse some define compiler statements which are in the form: #define identifier identifier_string. I keep a list of (identifier identifier_string) pair. So when I reach in the file a identifier that is #define list I need to switch the lexical analysis from main file to analyse the corresponding identifier_string. (I don't put the complete flex code because is too big) here's the part:

{IDENTIFIER}                    {   // search if the identifier is in list
                                    if( p = get_identifier_string(yytext) )
                                    {
                                        puts("DEFINE MATCHED");
                                        yypush_buffer_state(yy_scan_string(p));
                                    }
                                    else//if not in list just print the identifier
                                    {
                                        printf("IDENTIFIER %s\n",yytext);
                                    }
                                }
<<EOF>>                         {
                                    puts("EOF reached");
                                    yypop_buffer_state();
                                    if ( !YY_CURRENT_BUFFER )
                                    {
                                        yyterminate();
                                    }
                                    loop_detection = 0;
                                }

The analysis of the identifier_string executes just fine. Now when the EOF is reached I want to switch back at the initial buffer and resume the analysis. But it finishes just printing EOF reached.

Answer 1

Although that approach seems logical, it won't work because yy_scan_string replaces the current buffer, and that happens before the call to yypush_buffer_state. Consequently, the original buffer is lost, and when yypop_buffer_state is called, the restored buffer state is the (now terminated) string buffer.

So you need a little hack: first, duplicate the current buffer state onto the stack, and then switch to the new string buffer:

/* Was: yypush_buffer_state(yy_scan_string(p)); */
yypush_buffer_state(YY_CURRENT_BUFFER);
yy_scan_string(p);