Parsing T-SQL statements to tokens

Question

You can see here how nicely C#code statement is parse to tokens. For example, the following code:

namespace MyNamespace
{
    class MyClass
    {
        public void MyFunction(int arg1)
        {
            int var1 = arg1;
        }
    }
}

is parsed to this:

I want to do something like this but with T-SQL statement instead. For example, if I have the following T-SQL statement:

IIF(COALESCE([Col001], [Col002], [Col003]) > [Col004], [Col005] * [Col006] + ISNULL([Col007], [Col008]), CONCAT(SUBSTRING([Col009], 0, 3), 'sample text', [Col010]))

will give me something like this:

IIF, COALESCE, ISNULL, CONCAT, SUBSTRING    - functions 
[Col001], [Col002], ... , [Col010]          - columns 
0, 3, 'sample text'                         - variables

or in case I have:

ISNULL([Col001], [Col002], [Col003])

structure with errors:

[The isnull function requires 2 argument(s).] - error

There are not any free or paid up-to-date solutions and it seems to use the Microsoft parser is the best solution here. As I have read I need to use the Microsoft.SqlServer.Management.SqlParser.Parser namespace, but there are not any examples and I was not able to split the T-SQL statement the way I like. Also, it seems to work only with complete statements (you need SELECT clause for example, and I need to use it for code fragments only).

Can I do this using this namespace or it's better to start writing C# class for my needs instead?

Answer 1

I had to add the reference manually in the csproj

Microsoft.SqlServer.Management.SqlParser, Version=12.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91

Like

<Reference Include="Microsoft.SqlServer.Management.SqlParser, Version=12.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91" />

A simple example:

string sql = "IIF(COALESCE([Col001], [Col002], [Col003]) > [Col004], [Col005] * [Col006] + ISNULL([Col007], [Col008]), CONCAT(SUBSTRING([Col009], 0, 3), 'sample text', [Col010]))";

var po = new ParseOptions { };
var scanner = new Scanner(po);
scanner.SetSource(sql, 0);

Tokens token;
int state = 0;
int start;
int end;
bool isPairMatch;
bool isExecAutoParamHelp;

while ((token = (Tokens)scanner.GetNext(ref state, out start, out end, out isPairMatch, out isExecAutoParamHelp)) != Tokens.EOF)
{
    string str = sql.Substring(start, end - start + 1);
    Console.WriteLine("{0}: {1}", token, str);
}

Taken from http://www.sqlservercentral.com/blogs/dave_ballantynes_blog/2012/03/13/parsing-t-sql-the-easy-way/

Note that this parser recognizes a certain number of functions (like IIF, COALESCE, ...). Unrecognized functions are simply marked as TOKEN_ID, like column names.