Syntax Highlighting in VSCode

So, time to start writing some code. Or, at least copying it.

I created a new directory in the ignorance repository, called vscode-java. This is where I'm going to put the VSCode half of the language server - the client if you will. As trailed in the last post, my starting point is going to be copying the contentprovider sample and simplifying it. So that's the code that I copied.

And then I went through "simplifying" it - i.e. deleting most of the actual code so that I was just left with the syntax highlighting portion. I then copied in a couple of sample text files from my own repository. And obviously I had to run npm install and open it in VSCode.

How Syntax Highlighting Works

The instructions on  configuring syntax highlighting in the Microsoft documentation are actually quite clear, but not very exhaustive. Mainly it seems to defer most of the details to "it's the same as TextMate" without referencing anything.

The official word on TextMate grammars appears to be  this document, but it's not very detailed itself. I haven't managed (yet) to find any more introductory work.

The key thing seems to be to configure language and grammar contributions in package.json, so I did this:

"contributes": {
  "languages": [
    {
      "id": "flas",
      "extensions": [ ".fl" ]
    },
    {
      "id": "flas-st",
      "extensions": [ ".st" ]
    }
  ],
  "grammars": [
    {
      "language": "flas",
      "path": "./syntax/flas.json",
      "scopeName": "source.flas"
    },
    {
      "language": "flas-st",
      "path": "./syntax/flas-st.json",
      "scopeName": "source.flas-st"
    }
  ]
}

Here I appear to be defining two languages, but that's just because I have two types of file for my language: the main files have extension .fl and the system tests have .st. Each of these has its own grammar. The grammars are placed in files under the syntax directory and each has its own scope name for theming purposes.

Defining the Grammars

The grammars are defined in JSON approximately in line with the description of "TextMate grammars" insofar as I understand it (not a lot as yet). I'm sure it will become clearer as I dig in more. Sadly, the syntax is sufficiently opaque as to discourage you from learning by example.

However, this is an excerpt of one of the grammars I defined.

{
  "name": "flas",
  "scopeName": "source.flas",
  "patterns": [
    { "include": "#comment" },
    { "include": "#contract-intro"}
  ],
  "repository" : {
    "comment": {
      "name": "comment.line.file",
      "match": "^[^ \t].*$"
    },
    "contract-intro": {
      "begin": "\tcontract\\b",
      "beginCaptures": {
        "0": { "name": "keyword.intro" }
      },
      "end": "(//.*$|$)",
      "name": "statement.contract",
      "patterns": [{"include":"#typename"}]
    }
  }
}

The name and scopeName match the language and scopeName from the package.json. Failure to match both invalidates the grammar and it will not be used for syntax highlighting. The patterns array defines a set of productions or rules that can occur at the top level. In spite of being defined by regular expressions, there is an element of grammar productions to this, and it is certainly NOT the case that each regular expression just matches what it feels like.

The repository allows you to define more complex, (possibly recursively) nested rules. The #include syntax says that instead of specifying a specific pattern it is possible to delegate to a rule in the repository. The reference to the rule name must begin with a #, while the rule name itself does not; I'm not sure why. The patterns array both at the top level and within a repository definition is an array, any of which of the patterns may match some or all of the portion of the inner text, but it is not possible for them to match overlapping text.

It's also important to realize that the begin and end patterns are not part of the body of the rule and so are not included in the sub-matching of the patterns array but rather have separate logic to style them (beginCaptures and endCaptures).

Debugging

If reading (and writing) these grammars is hard, figuring out what is going on - and wrong - is insanely hard. First off, every time you make a change to any of the grammar files, you need to restart the Extension instance. This is done by using Sh-F5 to stop the current instance and then F5 to start a new instance.

It is then possible to see the consequences of your actions. If you're fortunate, you will see visual effects on the screen. If not, or if you just want clarity about what happened, it's possible to bring up the Token Inspector to see what happens. In the extension window, type M-Sh-P to bring up the command window and then type some portion of Developer: Inspect Editor Tokens and Scopes. Selecting this pops up a window which shows which rules were applied at the current location. To choose a different location, simply click there (unless it's under the popup window, in which case you may need to resort to trickery such as clicking elsewhere first or using the keyboard). To dismiss the window, press ESC.

On the upside, every time you restart, VSCode picks up from where it left off, so you don't need to go through the steps of re-opening the relevant windows. It also learns fairly quickly that you want to use the Token Inspector and suggests it sooner. And of course, if you are desperate, you can bind it to a simple keyboard shortcut.

Conclusion

Actually wiring up syntax highlighting was surprisingly easy. Getting the patterns to work was not. The complete lack of any tooling (at least in VSCode) that points out errors or failings was really annoying. Some things that particularly caught me out were forgetting the # when referencing a rule in the repository; not doubling the backslash characters before special characters (such as \b) in regular expressions (but note that this is not wanted for \t, which is a tab character); the begin and end syntax along with the fact that they are not included in the inner patterns; and the fact that regular expressions do not overlap.

I need to spend considerably more time looking into the syntax and trying to figure out how to use it to reasonably describe a quite complex "context sensitive" grammar using this weird mixture of regular expressions and production rules. But that is more for the "real world" than it is for this blog (although I may come back here if I have any wisdom to distill), and in the meantime it is time to move on to the task of integrating with a Java back end.

First cut at LSP

Building on my previous research, I'm ready to try and do something with VSCode and LSP.

The place to start is by seeing if we can get some existing sample code to build and run. Let's try that.

Doing something

Based on my previous research, I decided to start by checking out Adam Voss' LanguageServer over Java example:

git clone https://github.com/adamvoss/vscode-languageserver-java-example.git

Given that this was based on a Microsoft sample, I checked this out too:

git clone https://github.com/microsoft/vscode-languageserver-node-example.git

But then, reading the README, it transpires that in the meantime, Microsoft have deprecated this (actually, quite a long time ago). Looking at the replacements, it looks to me not so much that the technology has changed as that Microsoft have moved from their samples from separate repositories to a "mono repo". Anyway, I went ahead and checked that out, too.

git clone https://github.com/Microsoft/vscode-extension-samples

This seems to have a number of samples within it, so which one should we pick? It seems that the most basic one is helloworld-sample, so let's start there. There appears to be some level of "getting started" documentation available, although it doesn't start with checking out this repository but using some complicated mechanism to generate your own, and seems to overly complicate matters. But based on all of this, including the instructions in the README, which also seemed a little confusing, this worked for me.

• In a terminal, cd into the helloworld-sample directory
  
• Run npm i
  
• In VSCode, select "File > New Window" and then "Open Folder…" and then select the "helloworld-sample" directory
  
• From here, push F5 or select "Run > Start Debugging"
  
• A new window opens
  

Now, I haven't (and probably won't) dig into what this extension does, because at the moment I'm not interested, but it offers a new command you can run by doing CMD-SHIFT-P and typing Hello World. This pops up a message in the bottom right corner of the window. OK, very good, we have something working. Close both the VSCode windows (not saving whatever it was we hadn't changed) and we are back to where we started.

Working with LSP

It seems like the sample closest to what I had been looking at before is the lsp-sample. So let's repeat the above process to see if we can make this sample work as well.

• In a terminal, cd into the lsp-sample directory
  
• Run npm i
  
• In VSCode, select "File > New Window" and then "Open Folder…" and then select the "lsp-sample" directory
  
• From here, push F5 or select "Run > Start Debugging"
  
• A new window opens
  

The documentation says that you need to open a file in the "'plain text' language mode" but I have no idea how to open a file in a particular language mode. So I guessed that what it means is open a file that doesn't have an extension that it recognizes. Fortunately, I have a fair few of those, so I tried one. The important thing seems to be to find a file which does not already have a built in language server in VSCode (e.g. .js or .ts files are a bad choice).

The sample appears to have three features: if you type a 'J' or a 'T' it will offer the completions "JavaScript" and "TypeScript" respectively; meanwhile, if you type an "identifier" that starts with at least two uppercase letters, it flags an error.

OK, that seems to do what I want, except it does it entirely inside node, whereas I want to be delegating all the hard work to Java. There doesn't seem to be an official sample that does that, so I think I'm back to rolling my own based on the model I have from Adam Voss.

Syntax Highlighting

At this point, it seems worth going on a diversion to look into syntax highlighting. As I noted last time, syntax highlighting is handled differently to the language features that depend on LSP and is handled using regular expressions entirely on the client side.

Strangely, there doesn't seem to be a sample that specifically addresses this; I'm not sure why. There are three samples, however, which do include syntax highlighting of which contentprovider-sample seems to be the simplest. This is actually a sample about how you can generate documents and display them in an editor window; the syntax highlighting is just an adjunct that is there in order to make the generated document "look good" (and possibly to define regions to which affordances can be added).

Syntax highlighting is covered thoroughly in the documentation but the thrust is that you need to add a "contribution point" into the package.json file that defines the grammars for the languages you want to define (you also need to define a "languages" contribution point which identifies the languages based on file extensions, but I think that is fairly much a given here). The basic idea is that you identify regions of text and associate each of them with a "scope" (the word scope implies, I believe accurately, that the regions can nest within other regions). In the contribution point, you specify a list of bindings, each of which identifies a language, the path to a JSON grammar file (relative to the extension directory) and the root scope for the language.

The JSON grammar file defines an object representing the grammar of the language. Within this, there appears to be duplication of the scopeName. Since we are referencing the grammar file by name, I don't see how these two can be different, yet both seem to be needed, which is my definition of duplication. The other two values are patterns which is a list of pattern names which can appear at the top level, and repository which amounts to the complete set of productions for the grammar. These can, of course, be recursive, and the upshot is that each identified pattern in the grammar has a chain of matching rules (or scope) which enables VSCode to apply the appropriate themes.

The themes can likewise be introduced as extensions, and the theme-sample shows how this can be done. It would seem that Microsoft for some reason have adopted much of this from TextMate and largely expect you to have existing TextMate themes that you wish to reuse. There does appear to be an alternative scheme for defining a new color scheme which would be compatible with the grammar.

These grammar JSON files do not look easy to define. Since I already have a machine-readable formal grammar for my language, I'm sure I will just end up writing one more tool to convert that into a set of regular expressions for syntax highlighting. It would appear that somebody at or connected with Microsoft has done something similar to generate some of their examples.

Conclusion

Well, I still haven't written any code. But I am starting to get closer to knowing what it is that I need to do, so next time out I'm going to start by copying across parts of these various samples and trying to get something which does an amalgam of them: register a language, syntax highlighting, themes and start a Java LSP server.

Integrating with VisualStudio using Language Server Protocol

In my daily life, I do a lot of work with compilers and programming languages. In the course of doing that, I want to provide a quality editing experience; in the modern era that means things like syntax highlighting, auto completion and rapid feedback on errors. But I don't want to write tools, I want to integrate with them. The question then arises as to which editors to integrate with. As a Java programmer, in the past I have generally used Eclipse, but it is not an easy architecture to plug into.

Recently, I have started using Microsoft's VSCode to do front-end JavaScript, HTML and CSS development: in spite of being a Microsoft product, it actually seems to be quite stable, reliable and sane. Its approach to embedding editing features for new languages is not necessarily to strictly embed them, but to allow for a connection to an external server which delivers the relevant features. This appeals to me because it enables me to write most of the integration code in Java - with which I am familiar and which is the implementation language of the compiler itself - thus making the job easier.

As an added benefit, the Language Server Protocol is supported by a wide array of languages and tools which means that doing this work once provides easier access to a range of tools, although it is still necessary to implement a relatively thin "client" experience for each tool.

So what is the Language Server Protocol?

Basically, the language server protocol is a communication protocol between editing tools and "compilers" which abstracts away the language details and allows the two ends to communicate in terms of the kinds of abstract operations that tools want to perform on language elements - look up definitions, search for usages, complete symbols, etc.

The protocol is a version of JSON-RPC over a lightweight HTTP protocol.

Building a Server in Java

Nobody wants to actually go to all the effort of writing the code to read and write JSON-RPC over HTTP. Fortunately, people have been there before us and done that. For example, in Java there is the lsp4j library which makes it possible to write to interfaces and then have a main method that wires up a server.

Most of the work is involved in implementing the LanguageServer interface and implementing all the methods. The main method then instantiates this and creates a "server" by wrapping this using the LSPLauncher.createServerLauncher() method. In addition to the server instance, this method requires an input stream and an output stream. Where do these come from?

This is where the genuine connection to the client comes in. You need a physical transport layer - most likely a socket connection - from which you can extract a stream in each direction. In passing these to the "server" you enable it to read requests and write responses.

Finally, there is a little bit of magic in wiring up the "remote client" interface (by which the server communicates back with the actual client) with the server code by having the server implement the LanguageClientAware interface.

Embedding a Connector in VSCode

Integrating with VSCode is not quite as simple as merely implementing a server. For a variety of reasons, VSCode requires a "beachhead" on the client side to handle the communication with the server.

The process is described in the VSCode documentation. Apart from anything else, this defines the capabilities of the language server and provides the implementation of the connection layer including starting up the "remote" (from the VSCode's point of view) server.

Additionally, some of the functionality associated with a language (such as syntax highlighting) is not implemented over the server protocol at all. Syntax highlighting, for example, is implemented strictly on the client side using regular expression matching. I can't say as I like it - parsing is so much more sane - but it sadly common to most tools.

A Simple Example

For this post I am only going to offer a little code and, breaking with tradition, it is not even my code. Microsoft offers an example of how to connect to another node.js based language server but the late Adam Voss ported this to use a Java server.

Reversing the order of presentation from the sections above, I am going to start with the client side (i.e. the code embedded in VSCode). Obviously, this needs to be node.js compatible and, since we are in Microsoft-land, that means typescript (although I believe it is possible to use JavaScript if you insist).

The client code

Each client needs to have a manifest associated with it, in package.json. Most of this is, of course, vanilla node.js/npm configuration: setting up dependencies and the like.

The key elements seem to be engines, activationEvents and configuration. These are described in some detail in the developer guide and I don't think I have very much to add to that at this point. Obviously, the engines describes the versions of VSCode with which the plugin is compatible; activationEvents describes the portion of the protocol that the plugin implements; and configuration covers the rest of the concerns, including (it would seem) allowing the plugin to introduce settings which the user can then configure.

What is not configured - and is therefore presumably implicit - is how the client is configured. It would seem that the module needs to export a single function called activate which receives an ExtensionContext and is responsible for creating a new LanguageClient object (defined by a Microsoft library) and, after configuring it as appropriate, calling start on it. The cunning, of course, is all in the options parameters that are passed to it.

Moving on to the Java example, we can look at the equivalent extension.ts file in this repository.

Starting towards the bottom (line 76), the language client is created and started (all on one line). The client options look fairly vanilla, but in lieu of the server options, there is a function name. For full disclosure, I haven't so much as cloned this repository yet, so for all I know it doesn't even work, but at the same time I know that JavaScript - and typescript, presumably - will accept a function as a parameter and then call it when it needs the value. I'm assuming that is what is going to happen here. It is worth noting, on the other hand, that this repository is a couple of years out of date, so it is also possible that it is using a no-longer-supported feature.

Anyway, assuming that it is right, it is passing in the function which takes up most of the module (lines 17-60). Again, confusingly, it returns a Promise of a StreamInfo, not the LanguageServerOptions I was expecting. But no matter.

First, it creates a socketpair which, on completion, resolves the promise by providing the reader and the writer. It also listens for the socket to be closed and reports that to the console (it doesn't actually close anything, which seems surprising, but it is possible that somebody else catches that). It connects the listen event to a handler which starts a java process (the server), telling it the port number which has been opened.

I have to admit that there are a number of things going on here which don't seem exactly right to me - but that is probably because I don't understand enough about how the node.js net.server abstraction works.

The server side

The server is a fairly simple and brain-dead Java application. On the flip side, it doesn't do very much.

The main code is in App.java. Basically, this reads the port from the arguments, creates a client around it, and then does the work to set up an LSP server using the streams and an ExampleLanguageServer.

This implements the minimal number of methods to implement a TextDocumentService, although for reasons I don't understand, the actual implementation is split between a class FullTextDocumentService and an inner class inside the language server class.

The server has methods to initialize, connect, shutdown and exit the server, as well as to return the implementation of the FullTextDocumentService. It also provides an implementation of the WorkspaceService which appears to be responsible for handling user configuration changes.

Conclusion

I've learned a lot about VSCode and the Language Server Protocol that I didn't previously know and having saved away the links, I am hoping this will be of use to me when I return to actually try and implement something.

The next step is obviously to clone these various repositories, bring everything up to date, get it to work as is and understand it a little better. After that, I will need to try and understand the breadth of the protocol before trying to connect an actual compiler.

Expect to hear more.

Chaos

This weekend I was re-reading John Gleick's excellent book Chaos and was reminded of my youth when a team of researchers and Bristol University were working with the Mandelbrot set as an example of a "highly parallel" problem. The focus of their research was the Inmos Transputer and the Occam programming language. In those days, this was a compute-intensive problem.

The modern-day inheritor of that kind of parallel-programming architecture is the GPU. It occurred to me to try and use a mandelbrot generator as a way of experimenting with low-level GPU programming, but in the end I was more interested in how much more powerful my current Macbook Pro is than all that hi-tech architecture of the late 1980s. TL;DR: you can't believe how much.

Implementation

So I set about building a quick-and-dirty implementation of a chaos field generator. I wanted to go a little bit beyond just the Mandelbrot Set and consider other examples too - such as the attractors for the various solutions to n³ -1 = 0 using Newton's Method.

"Obviously" I chose to do all of this in JavaScript. The main reason being that all the code runs in the browser (using a 2d canvas), and that makes it easy to distribute. Moreover, JavaScript has the ability to plug-and-play with different functions, and I used that to control what you see.

The code can be found in the usual place under the directory chaos, and the "results" are up on my website at http://www.gmmapowell.com/chaos/.

Conclusion

You may be able to, but I cannot believe how much more powerful our computers are today than they were 30 years ago. Things that took forever on the latest, most expensive equipment can now be done in an interpreted language on a regular laptop (or even a phone!).

A More Functional Database

I have recently been using DynamoDB and, to be frank, its API (at least in Java) is, IMHO, awful. So awful, in fact, that in wrapping the API as described below, I was forced to write a mini-wrapper to call from within the wrapper.

Obviously, I was not going to put up with this. But rather than launching in and doing "something", I sat back and considered what I really wanted. If you know me or read this blog frequently, you would probably fairly rapidly come up with a list something like this:

• Functional or functional-leaning;
  
• Tell-Don't-Ask;
  
• Transactional;
  
• Asynchronous.
  

Now, as it happens, these things are not really in conflict in this case, although I ended up with three APIs:

• A mini-wrapper I had to create just to tidy up the $DynamoDB$ interface;
  
• A low-level "tell-don't-ask" API; and
  
• A more transactional API with a functional feel.
  

What I really want to talk about here is the last of these; the other two are really just stepping stones on the way, although, for full disclosure, it was only when I had used the TDA API for a while and experienced the pain of doing it, that the third API occurred to me.

While I’m fully disclosing, this code didn't start here as an experiment, but grew organically out of one of my many “real life” complex and messy projects across a swathe of production code intended to support multiple databases as well as other integrated systems; for simplicity I’ve rewritten history a little and pulled out all the relevant classes for DynamoDB and assembled an example test case that steps through the basic CRUD flow using the functional API.

What am I thinking?

First things first. What do I even mean by a "functional" database? Almost by definition, databases are the opposite of functional. The very way we think about - and describe - databases is CRUD: a sequence of read/update operations with occasional creation and deletion to keep life interesting.

Given my long background in functional programming, a lot of things have been percolating in my brain over the decades and recently have experimented a little with (the somewhat-functional) FaunaDB (which I will eventually get around to writing up here) and in doing that I noticed something of a similarity between functional programs and how I might ideally write database logic.

Here is some functional code:

top = repeat x '*'
x = 5
repeat 0 c = []
repeat n c = c:repeat (n-1) c

Nothing particularly special here. But there is a rhythm to functional programs, which can be described as "define something; use it; repeat". I've often described writing functional programs as "do a little bit of the work now and push the rest off onto another function".

On the other hand, what drives a functional program is the fact that somebody somewhere wants to know a result and expresses this somehow. Generally, this either comes from a "main" method or from some kind of console or REPL. The "top level" expression is broken down into sub-expressions which are evaluated in turn until the basic blocks are reached.

(As an aside, TDA almost exactly inverts this logic: it takes all the basic blocks and says that the results should be sent to a consolidator that combines them and then promotes them to the next level until the top level handler is reached.)

So the question is: how does this relate to databases?

A "pull" model for databases

The normal way of dealing with databases is as an imperative begin-read-write-commit loop. But this transaction can also be viewed as a single operation in a REPL - the model used by functional programs. In this model the transaction becomes:

• Decide what you want to do
  
• Do all the reads and transformations
  
• Do all the writes in a single step
  

How is this different? Most importantly, an important topic in standard database theory isread your writeswhich basically says that there is a choice when you have done a write in a transaction about whether when you use "the value" again - particularly if you choose to read it back from the database - whether you see the version you wrote or the one that existed before your transaction started. By deferring all the writes until the end of the transaction, we avoid this conundrum, which is exactly what you would expect from a functional model, in which values do not change over the course of a function's evaluation.

The other major change is in the way in which we describe the steps of the logic. As with a functional program, we name each value that we read in; as we name it, it becomes available for other steps in the logic. For example, a transaction which reads two values and then writes the result might ordinarily look like this:

begin tx
x = get 'A'
y = get 'B'
z = x + y
put 'C' z
end tx

In a functional notation, we might write something like this:

tx = [Store 'C' z]
z = x + y
x = get db 'A'
y = get db 'B'

Now, this is neither purely functional or really executable; but the idea is that we are describing the transaction rather than actually running it.

Handling the impedance mismatch with imperative languages

Now, while I want to use a "more functional" approach to databases, I in fact still want to do this from within Java - an imperative language. How do I handle that?

It's actually not all that hard. Java has functions, and I can say that each of my "steps" can map to a function. I have an entry point, which kicks off the initial GET operations and identifies the logic operation (z) that I want to be invoked when both of the return values are available. I then define z and annotate its parameter arguments to say where they come from. Basically, the transaction becomes a state that the functions can access through the parameter annotations. Once a value is read or calculated, it is available and any method that depends on that can now be executed.

Handling asynchronicity

It's obviously very important that the database access be asynchronous - the alternatives are to waste threads blocking or just to kill performance - which aren't really alternatives at all.

This is done by having all the functions call into an asynchronous layer in the database and provide a central place to call back. When they do, the current state is updated with the new value and the list of pending logic calls is examined. Any that are now "ready" - i.e. those that were just waiting for this value to arrive - can now execute, possibly launching more GET or LOGIC requests. Any of these that are ready to execute can be run the moment that the current method returns; others will be deferred until all their arguments are present. Every logic method can also add to the pool of "pending writes".

Eventually, all of the logic is complete and it is possible to do the writes - unless an error occurred in the transaction, in which case none of the writes will be done.

An implementation

The implementation (available in the git repository) is called GLS for get-logic-set/subscribe, describing this alternative loop.

Start with the simple test case (SimpleGLSUsage). The fundamental concept here is the UnitOfWork, which is created in the test initializer initTest. This corresponds to the conventional notion of a "transaction" (I have shied away from the word transaction in part because it is overloaded and in part because the semantics of the underlying database are unspecified; in the case of DynamoDB it is not transactional).

Within a unit of work, it is possible to create multiple parallel Relations. A relation represents a thread of work going on with its own namespace - a scope, if you will, within an outer definition in a functional program. This allows multiple lines of logic to coexist while still being part of the same "unit" - the same values are only read once, have the same value across all relations, and the unit succeeds or fails together.

This simple test case is not a series of unit tests. Rather, it is a "script" for executing tests. To make this work, JUnit 5 is used along with its OrderAnnotations. The first test ensures that we can create a trivial object. The enact method on the unit of work says that we have configured everything, and it is ready to go; waitForResult blocks until the unit has completed - success or failure.

The second test simply attempts to read this object back in.

The third test reads the object back in and then prints it. Because Java does not treat functions as "first-class" objects, the method printHello is referenced as a string in the call to logic. This is the name of a method in the RelationHandler class, which in this case is defined to be this, which is the current class.

The fourth to sixth tests check that the greeting is what we would expect it to be, update it, and check that it has changed. The final test cleans up the record by deleting the record (although if anything goes wrong, the whole thing will be deleted on restart).

It is obviously possible to run these tests, but some setup is required: you obviously need an AWS account with a DynamoDB instance configured; you need to create a table and pass its name to the test in the test.table.name property.

Conclusion

The normal database paradigm fits well with imperative languages, but has the normal drawbacks of those languages - most specifically, very slow, synchronous behaviour. It's hard in imperative languages to break out of that because of the complexity of dealing with all the asynchronous state. Changing the metaphor - and making a central agent responsible for the state management - simplifies the code and improves reliability.

Interestingly, in writing this, I can see that my actual implementation does not map perfectly onto my mental model - my implementation has turned out to be more imperative than my mental model. My entry point is fairly close to the "bottom" of the execution stack - as it would be with a TDA implementation. To match the mental model more closely, the "get" operations should not be invoked directly from the entry point function, but should be their own functions, and each of the functions should be named to match the "value" it produces. Maybe I should try again and report on that experiment.

A Guest Post By Nelson

I have been very privileged to have had Nelson pairing with me for the past few weeks during lockdown. Here he gives an account of his experiences pairing with me.

I wish I could share Gareth's appreciation for the opportunity to work with me. But the fact is that on the whole, I find Gareth (or "the idiot" as I tend to think of him) very difficult to work with. Apart from just being generally cranky, cantankerous and egotistical, he seems to want to be in control all the time and not very interested in what I can contribute. I sometimes wonder why I'm here at all.

Leaving aside his continual desire to "drive" and control the keyboard (I prefer to think of myself as the more cerebral, anyway), he seems to blame me for everything that goes wrong - even though most of the time it's his fault - and then, whenever we do finally manage to get something working, he takes all the credit - because he was the one doing the typing! As if he's had a single original thought in his life!

More than anything, it's his interaction style that I find difficult to deal with. As I said, I am more a thinker, an observer of reality. He's always chomping at the bit to do something without really stopping to consider if it is a good thing. And then when I try to interrupt his flow, he doesn't even want to listen to me.

Often, I'll find myself contemplating some aspect of the code which seems perplexing or hacked together when suddenly, without warning, he will change the window and flit off onto some completely different task. Then, just when I'm about to point out some obvious flaw, he'll start the debugger and tell me that he's "busy". Eventually, of course, he comes back to the same point I'd originally wanted to let him know about. If only he slowed down for a moment to talk to me.

And then there's his language and communication style. Leaving aside the continual shouting and swearing (well, when you make that many mistakes can you blame him?), whenever he does talk to me it tends to be in a challenging and angry way. Why can't he recognize that I have something to contribute? And even when he is in a good mood - finally having figured out some trivial bug - he says "high five, Nelson", even though he knows I have flippers. Talk about species-ist.

And, finally, since I'm here, I'd like to point out how much nagging and gnashing of teeth it has taken in order to get him to just let me write this little piece on his blog. You'd think I wanted to do a hit job on him or something.

About Nelson

Our guest blogger today was Nelson, a plush shark from Ikea, who sits next to me on a daily basis as my pair partner when I am working from home. He appears to have deep insights into my psyche as a pair partner, not to mention being able to provide truly honest feedback.

Configuring a Custom Domain for API Gateway

Yesterday I was trying to configure a custom domain for one of my sites working in API Gateway. At the end of the day, everything you would expect to have to be there was there, but I found the process very hard to comprehend. While I'm not going to directly recount my experiences - that would be too painful - this is hopefully a somewhat simplified, annotated guide to what I did.

Route 53

Most of the Amazon documentation about this will tell you how to do this, that and the other in Route 53 - Amazon's DNS service. I'm sure that it all works and although I think they say that you don't have to use Route 53 I went down a deep bunnyhole trying to establish the right records on Route 53 that appears to have been completely unnecessary (on a practical level).

It was, however, while I was doing this that I understood the vital piece of information that I was missing. But, no spoilers …

Assumptions

First off, I'm assuming that just by reading this you know what AWS is, have an account and basically know your way around the console. If you don't, then you probably don't want to do any of this …

I'm also assuming that you know a little bit about DNS and have some provider (whether AWS or otherwise) which registers and manages domains for you and will enable you to play with their DNS records, in particular, to create an Alias or CNAME record. If you don't know what I'm talking about, you probably don't want to do this; if it sounded plausible until you reached the CNAME bit, you probably want to look at the help pages or forums for your provider. This is often under the set of "Advanced" or "Do Not Touch" features.

Assuming you are still with me, I'm going to assume that you have configured an API Gateway already and that you know the (so-called 'custom') domain that you want to serve it on. I'm assuming that you have some request that "works" with the default domain. If not, I suggest you get that to work first.

Because API Gateway is secure, you need to provide a certificate through ACM (the Amazon Certificate Manager). This (obviously) needs to be a certificate which names the domain that you wish to use. This process is a little tricky in itself and I should have probably blogged about that when I did it a couple of weeks ago.

Finally (and possibly a bit more tricky), I'm going to assume that you have considered the ramifications of "edge-optimised" vs "regional" API gateways and that you have gone with regional. This is all a bit complicated (too complicated for me?) but basically if you have "edge-optimised" gateways, you will need to set cloudfront up across them in order to do this. I didn't try that, so the rest of this article may or may not be relevant. If you aren't sure, the "Endpoint type" column on the API Gateway screen says if an endpoint is Regional or Edge.

Creating a Custom Domain Name

The first step is to create a "Custom Domain Name" in API Gateway. This does not work in at all the way that I would expect or match any of my mental models, so I will try and explain it from my point of view. I would expect that the option would be called "bind to domain name" or "map domain name to stage" and would be an option on a specific stage of a specific gateway. The functionality is exactly what I would expect, but you reach it through a completely different route.

Note the ID of the gateway you want to bind (that's the string of 10 or so random alphanumerics, not the more user-friendly "Name") but do not click on it. Instead click on the sidebar menu item "Custom domain names".

This will take you to a simple screen which has a search box and a create button. Click the create button to open up a detail window.

In the "Domain name" box, put the domain name you want to bind.

In the "Configuration" section, you will (probably) want to select "Regional" and "TLS 1.2". You then need to choose the appropriate certificate from the ACM.

Add any tags you want and click "Create".

You could easily be forgiven for thinking that you are done at this point but you are not (I have to forgive you since this is where I went wrong).

You should now be on a screen with a label "Domain name details". It all looks pretty reasonable and like what you just specified. But scroll down. Keep on scrolling. Yes, there. The section that says "API mappings". This is what you need to configure in order to bind this domain name to your API Gateway.

Configure API Mappings

Click the button that says "Configure API Mappings". On the screen that appears, click "Add new mapping". You now need to select the correct gateway (API) by comparing the id to the one you remembered earlier (while the name is shown it is not guaranteed to be unique) and the correct stage. It is also possible to bind individual domains to sub-paths of API Gateway, although I personally don't have a use for that.

Click "Save". You should be returned to the "Domain name details" screen.

Configuring DNS

Internally, what has happened here is that API Gateway has cunningly created a new, forwarding web server with a new DNS name. If you look down the list of configuration options you will see an "API Gateway domain name" that looks like nothing you've seen before. All of mine start with "d-" and then have a 10-character name that is most assuredly not one I've seen before.

What's cunning about this server is that if you make an HTTP request to it telling it that you want it as the 'Host', it will return that certificate. If you tell it that the 'Host' name is your custom name, it will return that certificate (when we've finished, you can check that using curl -v, or otherwise).

So now all you need to do is to point your DNS of record at this DNS name. Exactly how you do that depends on your DNS provider. AWS will continually push you towards using Route 53, and it seems that there instructions in that regard are correct, but given that I don't use Route 53 on a daily basis I tried to avoid that and it all worked fine.

Simply set up a CNAME record (AWS, and possibly others, refer to this as an Alias). The value of the CNAME record needs to be the API Gateway domain name quoted above with a final trailing '.'. The purpose of this final dot is to indicate that it is an absolute DNS name. Depending on your DNS provider, it may or may not be necessary for you to actually enter it.

Testing

Given that you have a working API gateway, you should now be able to substitute the API Gateway domain name quoted above in place of the default domain name AND the stage (together with any prefix path you may have chosen to specify in your mapping). This should work immediately - even before you have configured your DNS.

I'm not really sure what to tell you if it doesn't - the whole reason for this blog is that I spent a good hour or more thrashing yesterday because I couldn't figure out the problem (which was that I hadn't set up a mapping from my custom domain name).

Assuming that works, try using your "custom" domain name in place of the API Gateway domain name.

There's a very good probability that this doesn't work. DNS takes a while to propagate for all kinds of reasons. If your DNS provider also offers hosting, try from one of those machines - they are "closer" to the source of the change and will see it sooner. Otherwise your best bet is to wait for a while.

You can use the tool nslookup (where available) to check whether your DNS is reporting the existence of the domain and you can use ping to see if it responds. Or you can just try using curl or your browser until it gives in. But seriously, it can take literal hours for DNS changes to propagate around the world so don't panic.

If you see that the DNS has propagated, and it still doesn't work, then you have probably configured the CNAME incorrectly. The most likely options are that you mistyped (or mis-copied) it and that you either didn't - or did, depending on your provider - provide the final dot.

Conclusion

Configuring a custom domain for AWS API Gateway is actually quite simple but, IMHO, the interface is poorly designed and then poorly explained. If you don't make any mistakes, the process is straightforward, but if you do any one of the steps wrong, you receive very little feedback about where the problem might be.