How to tokenize grammars, like in monaco.editor.to

Question

I have used the tokenizer in monaco but I do not see that it is accessible in vscode. This would be helpful for completion/signature help providers, how can I tokenize a grammar?

Answer 1

It doesn't seem like there's an official way of doing this right now. There is an open feature request for adding the ability to retrieve tmLanguage scopes at a position here: #580

There is one potential workaround, which requires adding a dependency to the scope-info extension. This extension exposes an API of its own that other extension can use. Here's a code example posted by the author in the linked issue:

import * as api from 'scope-info'
async function example(doc : vscode.TextDocument, pos: vscode.Position) {
    const siExt = vscode.extensions.getExtension<api.ScopeInfoAPI>('siegebell.scope-info');
    const si = await siExt.activate();
    const t1 : api.Token = si.getScopeAt(doc, pos);
}