Wednesday, April 26, 2023

Populating a Side View with a Notification

Following on from my previous post, it has come to my attention that it is also possible to populate the view by using a notification from the back end to the front end.

The purpose of notifications is to provide updates without being asked. In our case, it makes sense when we have updated the repository (after parsing one or more files) to send such a notification. Otherwise, the front end needs to "guess" when the information may have changed and request it again.

So the task here is to try and turn the process around, and instead of sending a request for the information we requested, for the back end to automatically send it when it is ready.

Reading through the specification, it seems that this is possible, but that it requires us to "extend" the protocol. But I'm not entirely sure how to do that. It turns out that while it's possible, it doesn't seem to me that it's "genuinely" supported at all.

On the server side, we need to define a new interface which extends LanguageClient and specify the new notification methods that we want, tagged with the @JsonNotification attribute, which specifies the command string that will be used in sending the message.
package ignorance;

import org.eclipse.lsp4j.jsonrpc.services.JsonNotification;
import org.eclipse.lsp4j.services.LanguageClient;

public interface ExtendedLanguageClient extends LanguageClient {
  @JsonNotification("ignorance/tokens")
  void sendTokens(Object object);

}
It should then just be a case of telling the LSPLauncher that this is the protocol we want to use, but none of the factory methods takes an interface - they all assume that you want to use LanguageClient. But fortunately, there is nothing magic about any of these methods, so we can extract it locally and make the change we need.
    Launcher<ExtendedLanguageClient> launcher = createServerLauncher(server, in, out);
  private static Launcher<ExtendedLanguageClient> createServerLauncher(IgnorantLanguageServer server, InputStream in, OutputStream out) {
    return new LSPLauncher.Builder<ExtendedLanguageClient>()
      .setLocalService(server)
      .setRemoteInterface(ExtendedLanguageClient.class)
      .setInput(in)
      .setOutput(out)
      .create();
  }
Likewise, on the client side, it should be just as simple as calling the onNotification method of the client, but this can only be done once the client is ready, and thus after communication has already started up - meaning that some messages might be sent (and lost) before the handler is installed. I tried a number of ways of working around this (there appears to be a notion of Features which can install handlers) but nothing fixed the fundamental problem, so I ended up having to add my own synchronization around this. But for now, here's the code that sets up the handler in the onReady method.
  client.onReady().then(() => {
    client.onNotification("ignorance/tokens", () => {
      console.log("received token notification");
    });
So to handle the synchronization, we add another command with the message ignorance/readyForTokens. In the onReady method we call this AFTER configuring the notification handler.
    client.onNotification("ignorance/tokens", () => {
      console.log("received token notification");
    });
    client.sendRequest(ExecuteCommandRequest.type, {
      command: 'ignorance/readyForTokens',
      arguments: [ ]
    });
      
    const tokensProvider = new TokensProvider();
On the server side, we can then start sending messages when appropriate.
      public CompletableFuture<Object> executeCommand(ExecuteCommandParams params) {
        switch (params.getCommand()) {
        case "java.lsp.requestTokens": {
          System.err.println("requestTokens command called for " + params.getArguments().get(0));
          return repo.allTokensFor(URI.create(((JsonPrimitive) params.getArguments().get(0)).getAsString()));
        }
        case "ignorance/readyForTokens": {
          System.err.println("readyForTokens");
          amReady = true;
          return CompletableFuture.completedFuture(null);
        }
        default: {
          System.err.println("cannot handle command " + params.getCommand());
          return CompletableFuture.completedFuture(null);
        }
        }
      }

Which just leaves the process of actually sending a message, which we want to do once we have finished parsing. There is a certain amount of off-camera wiring to make this happen reliably.
  private void parseAllFiles() {
    File file = new File(workspaceRoot.getPath());
    parseDir(file);
    client.sendTokens("hello, world");
  }
All the complex wiring is once again left as an exercise for the reader (see VSCODE_NOTIFICATIONS_COMPLEX_WIRING).

Conclusion

It is certainly possible to configure both client and server to handle custom notifications, but there are enough tricky issues involved that I don't think that it can honestly be described as "supported". But if you're an event-driven freak like me, you just have to bite it off.

Tuesday, April 25, 2023

Adding a side view to LSP



It seems like a while since I last worked on my LSP server (*checks git history*: it's a very long while, OK).

But right now I find myself wanting to add a new view with some meta data to my LSP server in "the real world". And so I come back here to figure out how to support that.

I was originally thinking I could add a window at the bottom (akin to Problems, or Output) but it doesn't seem possible to add something there. I considered adding a virtual document but then I realized that was a lot of hard work for not much gain and what I think I probably want is a view.

So here goes …

The basic setup is that we want to add a view which is capable of presenting some hierarchical information as a tree. The thought would be that the server has some way of deducing "global" information about a project which it can then feed back to you (a class explorer, for example).

Looking back at what I did before, I can see that for each project in the workspace, I collected together a list of tokens (called Names) which have a file (in the form of a URI), and a location. For the purposes of testing what I want to, I am going to use this a source of information and build a tree which has a list of Names, and then for each Name, it will return a list of URIs, and within each URI a list of locations. If I understand what I did previously, each name will only have one URI and location, but that is not all that interesting: it's the fact that it's presented as a list which I care about.

But before we get to that, we need to start on the front end, and add a new contribution to our existing package.json.
    "views": {
      "explorer": [
        {
          "id": "ignorantTokens",
          "name": "Ignorant Tokens"
        }
      ]
    }
And run the extension using F5. Well, yes, that was easy. The "Ignorant Tokens" view appears at the bottom of the left hand column. Click on it and it expands to show … nothing.

So now we need to put some content there. For now, we're not going to do the hard work of getting the content from the server, we're just going to hack something in, say a list of items with lists in them. We'll use multiple items here to make sure our code works, even if that isn't something the Java backend is going to offer.

Start small, Gareth, start small. So let's just start with a list of the currently active folders in the workspace.

The key to providing the content is to call the window.registerTreeDataProvider method and to provide it with an object which can provide all the data that you need. That, in itself, is not that hard.
  const tokensProvider = new TokensProvider();
  window.registerTreeDataProvider('ignorantTokens', tokensProvider);
Now we can move on to actually implementing the provider, along with the data model representing the tree. The provider has to be an instance of the interface vscode.TreeDataProvider, and this has basically two methods: one to get the children of an item and one to get the "tree item data" itself. The Microsoft example I'm working from assumed it was, in fact, possible for the item to be an instance of the TreeItem class, and I have followed suit. This is my implementation of ProjectTokens, which basically just says that it's not hard to represent a workspace folder as a tree item by providing the super constructor with the folder name and saying that it should initially be in the "collapsed" state (click to show children).
import { TreeItem, TreeItemCollapsibleState, WorkspaceFolder } from "vscode";

export class ProjectTokens extends TreeItem {
  constructor(wf : WorkspaceFolder) {
    super(wf.name, TreeItemCollapsibleState.Collapsed);
  }
}
The provider has a constructor which looks at all the folders in the workspace and creates an item for them. It's not actually clear to me whether it is better to put this code in the constructor, or to put it in the code which returns the top-level list of items. I have put it in the constructor here to provide for a clearer separation of responsibilities, but it is not clear to me how well this will adapt to changes.

On the subject of which, I think this is the case which is supposed to be handled by the Event onDidChangeTreeData, but I have to confess that I am ignorant about this and I could not (at this juncture) be bothered to go and check it out, nor am I sure who would do the checking to generate this event (a later juncture occurred while tidying up at the end; see below for how this works and is used).

The first method that VSCode calls when trying to generate the tree view is, oddly, getChildren. But it does so passing in undefined or null or something which gives you the clue that it doesn't know whose children it wants: and the children of nobody must be the top level items, which we have to hand and can return.

Then for each of these items that you pass back, it turns around and asks you to hand it a TreeItem. As noted above, we chose to implement our ProjectTokens as TreeItems, so we can just return the very thing we are given. In other situations, you could return a member of the object, or presumably create one on the fly.
import { Event, ProviderResult, TreeDataProvider, TreeItem } from "vscode";
import { ProjectTokens } from "./tokenlocation";
import * as vscode from 'vscode';

export class TokensProvider implements TreeDataProvider<ProjectTokens> {
  locations: ProjectTokens[];
  constructor() {
    this.locations = [];
    if (vscode.workspace.workspaceFolders == null)
      return;
    for (var wf=0;wf<vscode.workspace.workspaceFolders.length;wf++) {
      this.locations.push(new ProjectTokens(vscode.workspace.workspaceFolders[wf]));
    }
  }

  onDidChangeTreeData?: Event<ProjectTokens | null | undefined> | undefined;
  getChildren(element?: ProjectTokens | undefined): ProviderResult<ProjectTokens[]> {
    if (!element) { // it wants the top list
      if (!this.locations) {
        return Promise.resolve([]);
      } else {
        return Promise.resolve(this.locations);
      }
    }
  }
  getTreeItem(element: ProjectTokens): TreeItem {
    return element;
  }
  getParent?(element: ProjectTokens): ProviderResult<ProjectTokens> {
    throw new Error("Method not implemented.");
  }
}
To support multiple levels of nesting, we now need to provide two more data classes (implementing TreeItem): Token and TokenLocation. The idea here is that Token gives you the name of the token that has been defined and TokenLocation gives you the location where it can be found.
import { TreeItem, TreeItemCollapsibleState, WorkspaceFolder } from "vscode";

export class ProjectTokens extends TreeItem {
  tokens: Token[];
  constructor(wf : WorkspaceFolder, tokens : Token[]) {
    super(wf.name, TreeItemCollapsibleState.Collapsed);
    this.tokens = tokens;
  }

  children() : Promise<Token[]> {
    return Promise.resolve(this.tokens);
  }
}

export class Token extends TreeItem {
  locations: TokenLocation[];
  constructor(name: string, locations: TokenLocation[]) {
    super(name, TreeItemCollapsibleState.Collapsed);
    this.locations = locations;
  }

  children() : Promise<TokenLocation[]> {
    return Promise.resolve(this.locations);
  }
};

export class TokenLocation extends TreeItem {
  constructor(where: string) {
    super(where, TreeItemCollapsibleState.None);
  }

  children() : Promise<Token[]> {
    return Promise.resolve([]);
  }
};
In the provider, we need to make three sets of changes. First, we need to change the signatures of the provider and all the methods to enable all three TreeItem types to be returned. The second, and most important, change is that in the getChildren method, when an element is provided, we must ask that element for its children. And finally, because we are hacking things together, we must update the constructor to provide the appropriate tokens.
import { Event, ProviderResult, TreeDataProvider, TreeItem } from "vscode";
import { ProjectTokens, Token, TokenLocation } from "./tokenlocation";
import * as vscode from 'vscode';

export class TokensProvider implements TreeDataProvider<ProjectTokens | Token | TokenLocation> {
  locations: ProjectTokens[];
  constructor() {
    this.locations = [];
    if (vscode.workspace.workspaceFolders == null)
      return;
    const tmp = [
      new Token("List", [
        new TokenLocation("46.2"),
        new TokenLocation("53.1")
      ]),
      new Token("Map", [
        new TokenLocation("15.7"),
        new TokenLocation("28.9")
      ])
    ];
    for (var wf=0;wf<vscode.workspace.workspaceFolders.length;wf++) {
      this.locations.push(new ProjectTokens(vscode.workspace.workspaceFolders[wf], tmp));
    }
  }

  onDidChangeTreeData?: Event<ProjectTokens | Token | TokenLocation | null | undefined> | undefined;
  getChildren(element?: ProjectTokens | Token | TokenLocation | undefined): ProviderResult<ProjectTokens[] | Token[] | TokenLocation[]> {
    if (!element) { // it wants the top list
      if (!this.locations) {
        return Promise.resolve([]);
      } else {
        return Promise.resolve(this.locations);
      }
    } else {
      return element.children();
    }
  }
  getTreeItem(element: ProjectTokens | Token | TokenLocation): TreeItem {
    return element;
  }
  getParent?(element: ProjectTokens | Token | TokenLocation): ProviderResult<ProjectTokens | Token | TokenLocation> {
    throw new Error("Method not implemented.");
  }
}

The back end

Now it's time to turn our attention to the back end and how we can generate this data on the server side and pass it across. You may recall (or, if you're more like me, may not - I had to go and look at the code) that the communication between client and server is handled by a language client (called client in extension.ts).

Now, the key to doing this is to be able to send a command across from the client to the server when we would otherwise generate the data ourselves - i.e. in the constructor of the data provider. But what message should we send? The documentation doesn't seem wildly clear on this, but it would seem that we can repurpose executeCommand for this purpose. The documentation states that "usually" this will return an applyEdit message, but it seems entirely possible that we could return something else - maybe "hello, world"? Let's give it a try, shall we?

First things first: we need to pass the client to the token provider, and that in turn means that we need to move the declaration down to the bottom. There are a couple of other caveats here: because we are going to call a method in client, we need to wait to make sure that it has initialized, so we attach this code to its onReady method. And because this call is asynchronous, we will want to call await, so it cannot be located in the constructor, so we need to move it out into a separate method. Note that because of this, the constructor could be outside the onReady callback if you needed to store it somewhere; I don't, and it looks neater to me all grouped together.
  client.onReady().then(() => {
    const tokensProvider = new TokensProvider();
    tokensProvider.loadTokens(client);
    window.registerTreeDataProvider('ignorantTokens', tokensProvider);
  });
Inside the tokensprovider, we need to call sendRequest specifying that we want an ExecuteCommandRequest and providing a unique string as the command identifier and then any arguments we might want (in this case the URI of the workspace folder). When we get the response back, we need to process it but this is left as an exercise for the reader.
  async loadTokens(client: LanguageClient) : Promise<undefined> {
    if (vscode.workspace.workspaceFolders == null)
      return;
    for (var wf=0;wf<vscode.workspace.workspaceFolders.length;wf++) {
      let uri = vscode.workspace.workspaceFolders[wf].uri.toString();
      const result = await client.sendRequest(ExecuteCommandRequest.type, {
        command: 'java.lsp.requestTokens',
        arguments: [ uri ]
      })
      // this.locations.push(new ProjectTokens(vscode.workspace.workspaceFolders[wf], result));
    }
  }

On the back end, we need to make two changes. First, we need to say that we now support executeCommand and to list the unique command identifiers we support. Then, in the WorkspaceService, we need to provide a trivial implementation of the command execution - in this case, returning "hello, world".
        ServerCapabilities capabilities = new ServerCapabilities();
        ExecuteCommandOptions requestTokens = new ExecuteCommandOptions(Arrays.asList("java.lsp.requestTokens"));
    capabilities.setExecuteCommandProvider(requestTokens);
        return new WorkspaceService() {
          @Override
          public CompletableFuture<Object> executeCommand(ExecuteCommandParams params) {
            System.err.println("execute command called for " + params.getArguments().get(0));
            return CompletableFuture.completedFuture("hello, world");
          }
I thought that was all I needed to do, but it turns out that it's important to take the final step of updating the list, because we need to make sure that we send notifications when we do.

I had commented earlier about the Event object that was part of the interface and I wasn't quite sure what to do about it. The answer would appear to be that it is an idiom that you define another, similar object called an EventEmitter and then wire up the Event to be the event field of this, and then to call the fire() method on the EventEmitter when you want to the tree to be updated.
      this.locations.push(new ProjectTokens(vscode.workspace.workspaceFolders[wf], result));
    }
    this._onDidChangeTreeData.fire();
  }

  private _onDidChangeTreeData: vscode.EventEmitter<ProjectTokens | Token | TokenLocation | null | undefined> = new vscode.EventEmitter<ProjectTokens | Token | TokenLocation | null | undefined>();
  readonly onDidChangeTreeData: vscode.Event<ProjectTokens | Token | TokenLocation | null | undefined> = this._onDidChangeTreeData.event;
In order to support this, I had to change "hello, world" to be an empty JSON array:
          public CompletableFuture<Object> executeCommand(ExecuteCommandParams params) {
            System.err.println("execute command called for " + params.getArguments().get(0));
            return CompletableFuture.completedFuture(new JsonArray());
          }

Exercise

As I always go back and read my own work, I also have to follow through on the exercise of wiring up the dots. I'm not going to bore you with the details here, but you can check it out in the repository (tag VSCODE_SIDEVIEW_EXERCISE).

Conclusion

It is possible to create side views in VSCode and populate them from a repository of information that can be found on the server side.

Late in the game, I also came across ​​https://code.visualstudio.com/api/extension-guides/tree-view which contains explanations for some of the things I found confusing while doing this work. It might be worth a read.