First cut at LSP

Building on my previous research, I'm ready to try and do something with VSCode and LSP.

The place to start is by seeing if we can get some existing sample code to build and run. Let's try that.

Doing something

Based on my previous research, I decided to start by checking out Adam Voss' LanguageServer over Java example:

git clone https://github.com/adamvoss/vscode-languageserver-java-example.git

Given that this was based on a Microsoft sample, I checked this out too:

git clone https://github.com/microsoft/vscode-languageserver-node-example.git

But then, reading the README, it transpires that in the meantime, Microsoft have deprecated this (actually, quite a long time ago). Looking at the replacements, it looks to me not so much that the technology has changed as that Microsoft have moved from their samples from separate repositories to a "mono repo". Anyway, I went ahead and checked that out, too.

git clone https://github.com/Microsoft/vscode-extension-samples

This seems to have a number of samples within it, so which one should we pick? It seems that the most basic one is helloworld-sample, so let's start there. There appears to be some level of "getting started" documentation available, although it doesn't start with checking out this repository but using some complicated mechanism to generate your own, and seems to overly complicate matters. But based on all of this, including the instructions in the README, which also seemed a little confusing, this worked for me.

• In a terminal, cd into the helloworld-sample directory
  
• Run npm i
  
• In VSCode, select "File > New Window" and then "Open Folder…" and then select the "helloworld-sample" directory
  
• From here, push F5 or select "Run > Start Debugging"
  
• A new window opens
  

Now, I haven't (and probably won't) dig into what this extension does, because at the moment I'm not interested, but it offers a new command you can run by doing CMD-SHIFT-P and typing Hello World. This pops up a message in the bottom right corner of the window. OK, very good, we have something working. Close both the VSCode windows (not saving whatever it was we hadn't changed) and we are back to where we started.

Working with LSP

It seems like the sample closest to what I had been looking at before is the lsp-sample. So let's repeat the above process to see if we can make this sample work as well.

• In a terminal, cd into the lsp-sample directory
  
• Run npm i
  
• In VSCode, select "File > New Window" and then "Open Folder…" and then select the "lsp-sample" directory
  
• From here, push F5 or select "Run > Start Debugging"
  
• A new window opens
  

The documentation says that you need to open a file in the "'plain text' language mode" but I have no idea how to open a file in a particular language mode. So I guessed that what it means is open a file that doesn't have an extension that it recognizes. Fortunately, I have a fair few of those, so I tried one. The important thing seems to be to find a file which does not already have a built in language server in VSCode (e.g. .js or .ts files are a bad choice).

The sample appears to have three features: if you type a 'J' or a 'T' it will offer the completions "JavaScript" and "TypeScript" respectively; meanwhile, if you type an "identifier" that starts with at least two uppercase letters, it flags an error.

OK, that seems to do what I want, except it does it entirely inside node, whereas I want to be delegating all the hard work to Java. There doesn't seem to be an official sample that does that, so I think I'm back to rolling my own based on the model I have from Adam Voss.

Syntax Highlighting

At this point, it seems worth going on a diversion to look into syntax highlighting. As I noted last time, syntax highlighting is handled differently to the language features that depend on LSP and is handled using regular expressions entirely on the client side.

Strangely, there doesn't seem to be a sample that specifically addresses this; I'm not sure why. There are three samples, however, which do include syntax highlighting of which contentprovider-sample seems to be the simplest. This is actually a sample about how you can generate documents and display them in an editor window; the syntax highlighting is just an adjunct that is there in order to make the generated document "look good" (and possibly to define regions to which affordances can be added).

Syntax highlighting is covered thoroughly in the documentation but the thrust is that you need to add a "contribution point" into the package.json file that defines the grammars for the languages you want to define (you also need to define a "languages" contribution point which identifies the languages based on file extensions, but I think that is fairly much a given here). The basic idea is that you identify regions of text and associate each of them with a "scope" (the word scope implies, I believe accurately, that the regions can nest within other regions). In the contribution point, you specify a list of bindings, each of which identifies a language, the path to a JSON grammar file (relative to the extension directory) and the root scope for the language.

The JSON grammar file defines an object representing the grammar of the language. Within this, there appears to be duplication of the scopeName. Since we are referencing the grammar file by name, I don't see how these two can be different, yet both seem to be needed, which is my definition of duplication. The other two values are patterns which is a list of pattern names which can appear at the top level, and repository which amounts to the complete set of productions for the grammar. These can, of course, be recursive, and the upshot is that each identified pattern in the grammar has a chain of matching rules (or scope) which enables VSCode to apply the appropriate themes.

The themes can likewise be introduced as extensions, and the theme-sample shows how this can be done. It would seem that Microsoft for some reason have adopted much of this from TextMate and largely expect you to have existing TextMate themes that you wish to reuse. There does appear to be an alternative scheme for defining a new color scheme which would be compatible with the grammar.

These grammar JSON files do not look easy to define. Since I already have a machine-readable formal grammar for my language, I'm sure I will just end up writing one more tool to convert that into a set of regular expressions for syntax highlighting. It would appear that somebody at or connected with Microsoft has done something similar to generate some of their examples.

Conclusion

Well, I still haven't written any code. But I am starting to get closer to knowing what it is that I need to do, so next time out I'm going to start by copying across parts of these various samples and trying to get something which does an amalgam of them: register a language, syntax highlighting, themes and start a Java LSP server.