Matching Lua's “Long bracket” string syntax

Question

I'm writing a jFlex lexer for Lua, and I'm having problems designing a regular expression to match one particular part of the language specification:

Literal strings can also be defined using a long format enclosed by long brackets. We define an opening long bracket of level n as an opening square bracket followed by n equal signs followed by another opening square bracket. So, an opening long bracket of level 0 is written as [[, an opening long bracket of level 1 is written as [=[, and so on. A closing long bracket is defined similarly; for instance, a closing long bracket of level 4 is written as ]====]. A long string starts with an opening long bracket of any level and ends at the first closing long bracket of the same level. Literals in this bracketed form can run for several lines, do not interpret any escape sequences, and ignore long brackets of any other level. They can contain anything except a closing bracket of the proper level.

In a nutshell, I am trying to design a regular expression that will match an opening long bracket, the string contents in between, and the closing long bracket. A match should only occur when the opening long bracket and closing long bracket have the same number of equal signs, which can be zero or more.

Answer 1

Well, I'm afraid tokenizing with regular expressions isn't good enough for this task. Regular expressions just aren't powerful enough.

There's no way to compare the number of '=' marks using plain regular expressions in jFlex. Perl would have a hack for that ( \1 as suggested above), but we're not talking about programming Perl, but jFlex lexer.

The solution is to go with \[=*\[ for the left bracket token, \]=*\] for the right bracket token, and then in the layer above (a parser) compare if they match in length.

Anyway, you can look at read_long_string() in the lua source code in llex.c and see how they did it without using regular expressions at all.

Answer 2

\[(=*)\[.*?\]\1\]

the \1 captures the first ().

Answer 3

\[(=*)\[.*?\]\1\]