Automatically Testing VSCode Syntax Highlighting

When I'm developing or debugging something, I often find an interactive, visual experience helps me.

Not when I'm testing for regressions.

Apart from the fact that it's inefficient and something that a machine can generally do better, I am really bad at spottng that either things have changed or that they are not how I expect them to me. My "internal reality" is stronger that the "observed reality", if you will.

So things like syntax highlighting present a significant challenge: it appears to be inherently visual and, more to the point, quite subtle (not to mention dependent on things like themes). There has to be a better way.

This being the case, you would think that the MS VSCode documentation was plastered with "look here for the automated testing solution" but not a bit of it. I picked up a couple of hints here and there which were so vague (and so long ago) that I won't even bother linking to them.

Eventually, however, by going to the source (literally!), I found this. Yup, I'll quote the whole paragraph:

For grammar authors


See vscode-tmgrammar-test that can help you write unit tests against your grammar.

Again, yup: this points you to a repository that isn't part of the core VSCode repository or even from Microsoft: it's by somebody called PanAeon. The whole "unit tests" thing looks gnarly, but further down, there is a reference to snapshot tests, which I believe are the same as wgat I call "golden tests" - but possibly coming from a more visual background: given any source input, once it looks the way you want it, run the "snap" and save the output file. Then when you have changed the code, come back and see if the current version produces an identical snap.

Hopefully, if this is as simple as it sounds, and works, we should be out of here in a few minutes and I can get out and enjoy this sunny Sunday morning in Manchester.

Working up an Example

So I am going to try and put two examples together and check that I understand what is going on. I am going to take the basic grammar from the VSCode Syntax Highlighting documentation, along with their example, and then embed the tmgrammar tool and see if we can build snapshots that match what the Microsoft documentation says should happen.

All Clear? Then let's begin.

The first thing we need to do, of course, is to put together a plugin that works in VSCode in "manual" mode. You might (reasonably) think this is just a question of patching everything they've given you together, and to a certain extent that's true, but there's other stuff besides. I ended up with these "source" files:

.gitignore
license.txt
package.json
samples/foo.abc
src/syntest.ts
syntaxes/abc.tmGrammar.json
tsconfig.json

The tsconfig.json is some typescript configuration file I got from somewhere, license.txt is just text, and .gitignore is just a list of output files not to check in. samples/foo.abc and syntaxes/abc.tmGrammar.json were copied directly from the Microsoft web page. So that leaves package.json and src/syntest.ts, both of which I would describe as "boilerplate" for a VSCode extension, but there are a few points that I want to make, so I'll show both of them here:

{
    "name": "syntest",
    "displayName": "SynTest",
    "description": "Syntax Testing",
    "version": "0.0.1",
    "publisher": "GarethPowell",
    "repository": {
        "type": "git",
        "url": "https://github.com/gmmapowell/ignorance-may-be-strength.git"
    },
    "engines": {
        "vscode": "^1.97.0"
    },
    "activationEvents": [
        "onLanguage:abc"
    ],
    "main": "out/syntest.js",
    "files": ["package.json", "out", "syntaxes", "license.txt"],
    "categories": [
        "Programming Languages"
    ],
    "contributes": {
        "languages": [
            {
                "id": "abc",
                "extensions": [
                    ".abc"
                ]
            }
        ],
        "grammars": [
            {
                "language": "abc",
                "scopeName": "source.abc",
                "path": "./syntaxes/abc.tmGrammar.json"
            }
        ]
    },
    "scripts": {
        "compile": "tsc -p ./",
        "package": "node_modules/@vscode/vsce/vsce package",
        "test": "vscode-tmgrammar-snap 'tests/snap/**/*.abc'",
        "update": "vscode-tmgrammar-snap --updateSnapshot 'tests/snap/**/*.abc'"
    },
    "dependencies": {},
    "devDependencies": {
        "@vscode/vsce": "^3.2.2",
        "vscode-languageclient": "^9.0.1",
        "@types/vscode": "^1.97.0",
        "@types/node": "^22.13.9",
        "typescript": "^5.8.2"
    }
}

SH_TEST_ABC:syntax-testing/package.json

I've highlighted three things that I had to deal with specifically.

The activationEvents field is what "kicks off" your plugin. I'm still not entirely sure how this works, but we register (below) the "language" abc, and this says if we open a file (or possibly a workspace with a file) which matches that language, then the plugin will be activated.

The files field limits what goes into your plugin. If you don't specify this, unnecessary files are included. It would seem that the parts of node_modules which are from dependencies are included, but devDependencies are not. But all sorts of other things (such as the typescript files) will be included if you don't specifically exclude them. Of course, you now need to be careful to keep this list up to date or you won't have the files you think you have.

The contributes section is copied directly from the Microsoft website, but since it's key to what we're doing here, it seems worth going over it again.

It has two parts. The languages section says that there is a "new" (?) language called abc and it is to be applied to all files with the .abc extension. The grammars section says that for the language abc, all tokens should be considered in the scope source.abc and then there is a JSON file where the grammar definition can be found. Be careful to make sure your paths match up: I didn't, and it was a long time before I found the relevant error in the Output>Window view.

I'm not entirely sure what is needed in the syntest.ts file, but in general you want an activate function the configures the LSP client and server. This is about the minimum I found I could get away with:

import { ExtensionContext } from 'vscode';
import {
    LanguageClient,
    LanguageClientOptions,
    ServerOptions
} from 'vscode-languageclient/node';

export function activate(context: ExtensionContext) {
    // Options to control the language client
    let clientOptions: LanguageClientOptions = {
        // Register for our languages
        documentSelector: [
            { scheme: 'file', language: 'abc' }
        ]
    };
    let serverOptions: ServerOptions = { command: "" };

    // Create the language client and start the client.
    var client = new LanguageClient(
        'Abc',
        'Abc',
        serverOptions,
        clientOptions
    );
}

SH_TEST_ABC:syntax-testing/src/syntest.ts

Here, in what appears to be duplication of the configuration in package.json we repeat that for files in the abc language, we want to be the client.

It's now possible to set everything up and install the package:

npm i
npm run compile
npm run package

(Note that this requires version 22 of node in order to work. Yeah, me too.)

It should then be possible to install the plugin from the command line but for me at least - this morning at least - on my Mac at least - I could not get it to work.

So open VSCode, go to the extensions "tab" (down the left hand side) and from the three-dot menu select "install from VSIX" and select the VSIX file that was just created.

Now, if you open samples/foo.abc, it should have highlighting. You should also be able to run the command (CMD-SHIFT-P) "Developer: Inspect Editor Tokens and Scopes". As you click around the buffer, it should give you information about the "selected" token such as "language" and "textmate scopes" as well as the colors applied by the theme.

OK, now we're ready to play the game.

Creating the Test Snapshot

Obviously the first thing we need to do is to install the tool:

$ npm i --save-dev vscode-tmgrammar-test
npm warn deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm warn deprecated glob@7.2.3: Glob versions prior to v9 are no longer supported


added 10 packages, and audited 234 packages in 3s

This is annoying. It would seem that in addition to not being an official part of VSCode, it also isn't kept up to date. Maybe I'll need to look into that if it does, at least, work.

We can now update the targets in package.json to reflect what we really want:

    "scripts": {
        "compile": "tsc -p ./",
        "package": "node_modules/@vscode/vsce/vsce package",
        "test": "node node_modules/vscode-tmgrammar-test/dist/snapshot.js samples/*.abc",
        "update": "node node_modules/vscode-tmgrammar-test/dist/snapshot.js --updateSnapshot samples/*.abc"
    },

SH_TEST_SNAP:syntax-testing/package.json

and then we can run these, updating first, of course:

$ npm run update


> syntest@0.0.1 update
> node node_modules/vscode-tmgrammar-test/dist/snapshot.js --updateSnapshot samples/*.abc


Updating snapshot for samples/foo.abc.snap

This, as it says, creates an output "snap" file foo.abc.snap.

$ npm run test


> syntest@0.0.1 test
> node node_modules/vscode-tmgrammar-test/dist/snapshot.js samples/*.abc


✓ samples/foo.abc run successfully.

And the checkmark tells you all you need to know. Success!

I, personally, don't like the pattern of having the "snap" files in the same directory as your input files, but I'm not sure if there's an option for that. But, fundamentally, this worked. The snap files are ugly, but do clearly match the interpretation given by the scope inspector tool.

Conclusion

We have been able to successfully install this tool and get it working. There are a couple of issues, but I think I can use it with a good conscience for now and then maybe come back to see what I can fix, possibly forking it for my own benefit.