PWA Notifications

Notifications are both more interesting and more complex than adding to the home screen.

"Push notifications" are, in fact, more a browser technology than they are a web app technology. This makes them even more browser specific than adding to the home screen. I generally develop with Chrome, and for my personal projects that's all I consider, so that's as far as I'm going to go in this article: Chrome uses a cloud module called "Firebase Cloud Messaging". However, it's my understanding that on the desktop, Firefox support for push notifications uses a similar cloud service called "autopush". It is my understanding that Edge and Safari on the desktop do not support push notifications at all. Mobile browsers are a different story again.

On the plus side, I believe that wherever they are available, these technologies are all compatible at the API level - at least as far as I describe it here. Your mileage may vary …

The basic paradigm

The basic paradigm for push notifications is that there is "special, magic infrastructure" that can deliver messages from a server to a client through the cloud and the browser. For Chrome, this is called "Firebase Cloud Messaging". As far as I can tell, it is only part of "Firebase" in a branding sense: you don't have to create an account and a project to use the cloud messaging service.

I have attempted to draw out what I think the basic architecture is here:



The browser loads the application from the server and enters into the "usual" two way interaction with it, possibly including Ajax or Websocket interactions. During this interaction, a "magic" key (called a VAPID key) is generated on the server and passed to the client. It uses this key to register its interest in push notifications with the browser. The browser in turn, magically and under the covers, notifies the cloud messaging service of this registration. The client receives a unique and somewhat persistent handle which it can pass to the server as it sees fit - the server needs this in order to be able to send messages to the client.

When the web server wants to send a push notification to the client, it contacts the cloud messaging service, passing it the appropriate request signed with the private portion of the VAPID key. The messaging server then looks up the corresponding registration(s) and delivers the messages to the corresponding browser(s). These, in turn, deliver the message to the service worker thread(s) of the appropriate web application(s), waking them up if necessary and displaying some notification to the user.

Security and Permissions

Everything to do with "the modern web" seems wrapped up in security and permissions. Sadly, with the number of bad actors out there, this is just a fact of life. Three separate processes go on in this context: firstly, as indicated above, there is a public/private keypair generated to ensure that the server sending the message corresponds to the clients wishing to receive messages; secondly, before any messages can be sent, the client must obtain permission from the user of the browser for push notifications to happen; and thirdly, when messages are transmitted with content (or "payload") the content must be encrypted from end to end which is handled by generating a shared key with the subscription.

The guidelines suggest that users should be encouraged to opt in to push notifications by taking a concrete action to enable them. Based on the number of websites I visit where the first thing you see is the message saying that the site would like to send push notifications, this does not seem to be widely adhered to. We will, of course.

Sending Messages

Sending messages theoretically requires a server, but because we are trying to just do this using a static website, we are going to take advantage of a command line tool to send notifications. There are tools of this kind for most languages it would seem, but I have chosen to install a Node.js version.

This can be installed by running

npm  install  -g  web-push

This installs the command globally which, if you have npm correctly configured means you should now be able to run the web-push command from your command-line.

$  web-push
Usage:  …

But before we can send any messages, we need to create a keypair. This is the basic security mechanism to ensure that all messages are sent by an approved party. In principle, the server generates the keypair and retains the private key, shipping the public key to the application somehow.

The private key in this context is a signing key: that is, it is used to provide a signature for the message that it sends and the public key can then be used to check that the signature is valid. Invalid messages are rejected.

This is done using the web-push command with the generate-vapid-keys subcommand:

$  web-push  generate-vapid-keys
Public  Key:
BNX8bG8mNTIJmXai9k35J5CKB2Wyc8kZoJS9Y31qkfUSfiQr7q22vDe5CHCxUclvpl1gEVAewVoINOvFlFFl4
Private  Key:
7XXms7NXIM-FuCrxVzoQqlLYz3kuYpdftzL5Dz_LI

We can now send a message using the send-notification subcommand:

web-push  send-notification  --vapid-pubkey="BNX…"  --vapid-pvtkey="7XXX...LI"  --vapid-subject="mailto:ignorant@blogspot.com"  --payload='hello,  world'
Usage: 
    web-push  send-notification  --endpoint=<url>  [--key=<browser  key>]  [--auth=<auth  secret>]  [--payload=<message>]  [--ttl=<seconds>]  [--encoding=<encoding  type>]  [--vapid-subject=<vapid  subject>]  [--vapid-pubkey=<public  key  url  base64>]  [--vapid-pvtkey=<private  key  url  base64>]  [--gcm-api-key=<api  key>]

The problem is that we haven't specified an --endpoint - where to send the message. In order to send a payload, --key and --auth are also required. Fortunately, we can obtain all three at once by the simple expedient of subscribing in our web app.

Subscribing to Push Messages

In order to get the endpoint - and the end-to-end encryption keys in order to send a payload - we need to subscribe on the client. As I said in the introduction, the "polite" way of doing this is to add a button to your web page to enable the user to "ask" for notifications.

Since this depends on having a registered service worker, by default this button should be invisible and only be displayed when the service worker has been registered. In the registration callback, it can then be displayed until such time as it is clicked (or otherwise dismissed). Of course, as with everything else, this doesn't have to be a button per se but can be any kind of user affordance which indicates a deliberate intent to subscribe.

The button then needs an event listener which actually does the subscription like so:

var  options  =  {
    userVisibleOnly:  true,
    applicationServerKey:  applicationServerKey
};
registration.pushManager.subscribe(options)
.then(function(sub)  {
    console.log("subscribed  to",  sub.endpoint);
    var  simple  =  JSON.parse(JSON.stringify(sub));
    console.log("auth",  simple.keys.auth);
    console.log("key",  simple.keys.p256dh);
});

When you click the button, it turns around and asks the registration object obtained from registering the service worker to subscribe using a set of options. The applicationServerKey is the public VAPID key used by the server. This is defined at the top of start.js; if you want to run this example, you will need to replace that value with the one you generated above. The userVisibleOnly flag is one that says that when we send a message we will alert the user that we have done so. Our current code in fact does not do this; instead, the browser will (sometimes?) display an automatic notification on our behalf to say that messages have been received.

The result of calling subscribe is a subscription object, returned through a promise, which contains a new endpoint describing this application in this browser on this machine. Note that this subscription is somewhat persistent: if you run this code multiple times, you will get the same value over and over. Obviously on a different browser or a different device, you will get a different code.

The endpoint also automatically contains everything you need to know to send messages - as a URI, it has within it the server that is capable of sending messages to this browser.

The auth and key values are the values we need to use to encrypt the payload for end-to-end transmission.

Receiving Messages

Turning to the service worker, we need to handle messages when they arrive. This is done by listening for the message event.

self.addEventListener('push',  function(ev)  {
      console.log("received  push  event",  ev.data.text());
});

This is where most of your code will need to be placed, but for now this is enough to show something working end to end. Add the extra parameters to web-push send-notification and you should see messages come out in the console.

web-push  send-notification  --vapid-pubkey="BNX…"  --vapid-pvtkey="7XXX...LI"  --vapid-subject="mailto:ignorant@blogspot.com"  --payload='hello,  world'  --endpoint="..."  --auth="..."  --key="..."

Displaying Notifications

As noted above, we are expected to display a user notification when these messages arrive. More than that, it is obviously useful to attract the user's attention, especially since the notification can be displayed when the application (and even the browser, they say) is not running.

It is easy to create a notification in the callback:

self.addEventListener('push',  function(ev)  {
    console.log("received  push  event",  ev.data.text());
    self.registration.showNotification('New  Message',  {
        body:  ev.data.text()
    });
});

Now, what if we want to have the user able to do something when this happens?

Handling Notifications

There is a notificationclick event that the service worker can handle. In this case, it is possible to take actions based on the notification arriving. This handler simply closes the notification and shows a message:

self.addEventListener("notificationclick",  function(ev)  {
      const  notify  =  ev.notification;
      notify.close();
      var  longOp  =  new  Promise(function(resolve,  reject)  {
              console.log('notification  was  clicked');
              resolve();
      });
      ev.waitUntil(longOp);
});

although, for full disclosure, I deliberately showed the message inside a promise to show how that can be wired up to the notification handling mechanism.

It's possible to do much more than this and, in particular, it's possible to make sure that our whole app is woken up. There are examples of how to do this on the Google Developers' blog.

A Pattern for Using Notifications

For me at least, there is something of a mental model dissonance in using this push technology. I grew up on socket-based client-server architectures and then moved to event-driven computing with an event bus at TIBCO. From this perspective, the web always seems backward from this point of view; the closest web technology is the WebSocket.

I think the right way to think about push notifications is to only use them when the server already feels ignored and has no other way of communicating with the app. When the user is actively working on the client, the server should just interact with the app directly and the user should see that.

The problem, of course, is knowing when the user is interacting with the client.

There may be a better way of knowing, but I think the simplest thing to do is just to use WebSockets for communication "when the app is active" and then allow that to time out (this generally seems to happen after about ten minutes) or deliberately close it after a few minutes of inactivity. Then when the user next interacts with the app restart the websocket connection; if the server wants to bring something new to the user's attention when the websocket connection is down, it sends a "push notification" which is displayed and can get the user interacting with the client again (or not, if they so choose).

A Note on Terminology

I have found in this area that a lot of the terminology seems to be used loosely and inaccurately, but I'm not really sure what "accurate" would look like. Consequently, I've followed the herd and been loose and inaccurate. But here are my thoughts on how the terms seem to be used.

"Push" is a very vague, general term that seems to mean something along the lines of "initiated by the server". The idea seems to be akin to "unsolicited". For me, of course, the key concept is the idea that events happen "in the real world" and you want to be able to react to them. If the server sees the event, it is only reasonable that it lets the client know - and the client lets the user know.

"Messages" is a word I use a lot that means what it says. A message has been passed from somewhere to somewhere else. It's encoded in some way (separately from encryption) that has been agreed by both parties making it a sensible communication. Many people seem to use the phrase "message" or "push message" in the current context to mean the process of sending a message from the web server to the app via the cloud (the very name "Firebase Cloud Messaging" is such a usage).

"Notifications", I think, technically refer to just the final step of the journey: showing something to the user. This is common parlance in the Android world. "Push Notifications" seems to blur the meaning somewhat. Yes, it should end in a notification - as we saw, the API wants you to commit to user visibility - but it encompasses the full lifecycle of the message's travel.

"Subscription" describes the way in which the app connects to the cloud messaging service for a particular web app. This word is used in many different ways in other fields (especially within the publish/subscribe paradigm) but here it has a very specific meaning of a single web app on a single device in a particular browser.

Push from an Actual Web Server

We have used the command line to generate our messages which is obviously not realistic. From within a real web server it is possible to do exactly the same thing - the command line tool that we used simply spun up a node.js instance and used the server-side library.

As far as I can see, the github "user" web-push-libs supports libraries for Node, Java, PHP, Python, C and C#. If you need something else, it is possible to work more directly with the REST API and talk directly to the endpoint.

Likewise, we have copied and pasted various items from the console to glue all of this together. A real application would need to use a technology such as AJAX or WebSockets to connect everything together.

All of that is left as an exercise to the reader.

Firebase

Given that this uses "Firebase Cloud Messaging" on Chrome, it may seem like a good idea to use Firebase. This may in fact be a good idea. But it seems to me that it adds a lot of complexity and moving parts - and I am unclear on the benefits.

Conclusion

Notifications are definitely harder than most of the other web technologies I've used. There are more moving parts than usual and connected together in different ways. But it is certainly possible to get something working in an hour or two if you know what you're doing.

I think I do now, and hopefully you do too!

Adding to the Home Screen in PWA

Moving on from "just being a website", there are two things that most Progressive Web Apps want to do: be added to the home screen, and to deliver notifications.

These are relatively easy to achieve, but possibly more arcane than I would like, not to mention being inconsistent: the treatment on different platforms is platform- and browser-specific. It would seem to me that Chrome on Android is the "gold standard" of what's supported, and everything else is either "inadequate", "on the way there" or "unsupported" depending on your perspective.

Tidying up from before

While everything seemed to work before, it nevertheless remained the case that Chrome felt a bit "picky" about what we'd done. There are a couple of warnings that tell you that something is up, but not really what.

Scope

Message: Site cannot be installed. No matching service worker detected.

The service worker runs in the background and has the ability to 'intercept' requests (see the next section). But it can be limited in the number of requests that it can intercept by specifying a "scope". Either way, the scope is constrained by the directory from which the service worker file is loaded. This is annoying, since it stops you properly arranging your code, but only the service worker file itself needs to be at the top level.

I moved service_worker.js up to the top level.

Intercepting Requests

One of the key functions of a PWA is its ability to continue to function even when the device is not connected to the internet (this is also one of the main reasons that support is so much better on mobile devices than desktop devices).

In order to make this work, it is necessary to be able to provide all of the resources from local storage rather than from the internet, which means that you need to know where to find them locally.

The browser delegates this task to the service worker through the "fetch" mechanism.

fetch is an event which the service worker must register for. Registering for events in the service worker is just like doing so in a regular javascript application except that the "magic" variable is not document but self. self is a global variable in the context of the service worker which resolves to the current instance of type ServiceWorkerGlobalScope.

Thus we have something like this in service_worker.js:

self.addEventListener('fetch',  function(ev)  {
  ...
});

Here ev is a FetchEvent. The key thing that it supports is a respondWith method which enables the service worker to return a cached copy of a file.

It seems to me that most of this method is "boilerplate" code in that it seems to be there to connect requests from the browser to a builtin "caching" mechanism. Of course, it also gives you the opportunity to decide that some URIs cannot be cached, or to pull them from another data source - such as a database - but it seems overly reliant on user code rather than allowing more of a "filter" approach.

The "cache" is not in fact a single cache but a set of caches. In order to store the results of a fetch it is necessary to open an individual cache, but the matching process - by which we test if we have a cached file - operates across all the caches.

The implementation of the fetch logic can be found in the git repo with the tag PWA_BASIC_FETCH.

Background Color

For some reason, before you can add to the home screen, you need to set a background_color. This is mentioned in the Mozilla Documentation

Adding to Home Screen

This feature comes for free on Android the moment you load a PWA that Google Chrome recognizes. On the desktop however, more work is still required.

There is an event on the window called beforeinstallprompt which is triggered by the browser when all the conditions for being installed locally are met. In short these are:

• the manifest can be found and is configured correctly;
• the service worker is installed and has a correctly configured fetch event;
• the app is being served securely (either by HTTPS or from localhost);
• it hasn't already been installed.

We can now add the appropriate handler in our start.js script (because this is on the window, we need to do this in the client portion of the browser, not the service worker).

This does not immediately add the app to the home screen; rather it needs to provide an affordance - usually a button - to enable the user to do so. It is a requirement of the specification that the user must actively choose to enable this feature.

However, the action to actually add the app to the home screen is triggered by calling a method on the event passed to your beforeinstallprompt handler. Thus we need to squirrel away a copy of this event when we are given it for later usage.

Because this is a two-step process (first receive the beforeinstallprompt event, then click on a button), we need to make our button initially invisible and then make it visible when the event arrives. We do this by attaching a CSS class to the button with an initial setting of display: none and then specifically overriding the style of the element when the beforeinstallprompt event arrives. (I realize that there are many other ways of doing this; I'm just saying what I chose).

This is sufficiently tricky that I've checked it in before moving on - the tag is PWA_BASIC_BEFORE_INSTALL_PROMPT.

If you refresh, the button should appear. Note that the system can be a little picky about this, so you may need to hard refresh to get it to happen reliably.

Finally, we need to wire up the event handler. This does three things:

• makes the button vanish again by setting its display value back to 'none';
• prompts the user to check that they want to add to the home screen;
• if they agree, does any subsidiary processing.

Note that we don't actually do any subsidiary processing, but the code is there for completeness and reference.

In Chrome on a Mac (my environment), once this is complete the browser window "pops out" and becomes its own application. It is also added to the "Chrome Apps" folder which is then opened in Finder.

This is all checked in with the tag PWA_BASIC_A2HS.

Removing the App from the Home Screen

Again, I can't speak for all platforms, but on the Mac it is possible to uninstall the app by selecting the three dots in the right hand corner of the window and choosing "Uninstall …".

This is obviously useful - and important - for developing the install flow without reinstalling Chrome.

Getting Started with Progressive Web Apps

If you're reading this, I'm going to assume that you know what a web app is. You've probably written one or more. If you don't, think amazon.com.

If you're alive in the 21st century, you have to know what a mobile phone app is. You may well have written - or considered writing - one of those too.

If you have, in fact, written both web apps and mobile phone apps, you may have asked, "can I just do this once?" - particularly if you have written mobile phone apps for multiple platforms.

The answer is yes.

The technology is continually changing, but all of it essentially depends on you writing a completely client-side app - that is, you need to write your application almost entirely in JavaScript, and just communicate with the server to load data using APIs - "AJAX" as they often still confusingly call it.

The latest iteration of the technology is to say "just write a client-side web app that works when connected to your server, and then progressively add features to enrich it".

Whence progressive web app.

What's the minimum I can do?

Technically, the minimum you can do is to create a web app. In keeping with tradition, we could serve up hello world from helloworld.html. It's not my job to write HTTP servers for you, so I'm going to assume you either have one you like or can easily obtain node.js or python's SimpleHTTPServer. If you can't, you probably want to stop reading about now.

However, for my money, this doesn't actually qualify as a progressive web app because the amount of progress it has made is about zero. So the minimum I would consider is a web app with a manifest.

Adding a manifest

Any number of programming languages have had "manifests" for as long as I can remember. The idea comes from a "shipping manifest" (you know, one of those labels on the boxes FedEx drop off at your house): basically, it tells you what's in the box and where to go look for it.

I have to say, they always seem like a bit of a hack to me. If these things need to be known, why are they not required somewhere in the code? Anyway, progressive web apps require a manifest, the manifest must be a JSON document your server can serve, and your HTML page needs to tell the browser that it's there. In doing so, it notifies the browser that your web app is not just a web app, but a progressive web app and that it should trawl through it until it finds the things it needs.

You might think (well, anyway, I might think) that if this thing is JSON, the minimum it could be is an empty object, so:

{
}

In order to include this in your application, you need to add a link tag in the <head> section of your index.html.

<link  rel='manifest'  href='/json/manifest.json'>

Now we can load the website using our trusty friend, python's SimpleHTTPServer (or otherwise):

cd  pwa/html
python  -m  SimpleHTTPServer

And then the website should be visible on http://localhost:8000/.

So far, so good. You can check this out from github as  PWA_MINIMAL_ERRORS.

Uh, it doesn't look any different

Well, no.

That is what the "progressive" bit is all about. It starts off as a completely generic website and then when you have reached a certain threshold of adding stuff, you can start doing fancy things such as working offline, adding it to the homepage and sending notifications.

But we can check up on our work, though. Assuming you are using Google Chrome, open up the developer tools and go to Application. The top thing there, which you have probably normally ignored on your way to Cookies is a tab called Manifest. Click on that. You'll see a number of things there.



First off, there is a link to the source: always useful to check that it has downloaded the most recent version of your manifest (which can be a problem as discussed later). After that come a list of errors that Chrome helpfully describes as "instabilities". We're going to fix those before we do anything else.

Naming It

The second message here complains that there is no "name" or "short name". Drawing on  Mozilla's Documentation

Note that for reasons best known to themselves, manifest JSON files separate words with underscores.

{
      "name":  "First  Ignorant  PWA",
      "short_name":  "PWA1"
}

We can reload and the error now goes away. Scrolling down, we can now see that these fields have been added to the Identity section of the manifest information.

Where to Start?

Going back to the first error, most websites "know" that the home page of the website is called something like index.html. This is just a convention, of course, and you can configure a web server to point to any page on your website. Much the same convention applies with progressive web apps, but Chrome considers it an error to take advantage of the convention and instead wants it to be made explicit, hency the warning. By adding a start_url field to the manifest, we can identify that index.html is where we want the application to start when it is restarted locally rather than downloaded.

What Can I Show You?

The display field says how the application wants to present itself. If you are thinking of a PWA as a webpage that just has some fancy features, you will want to go with the simplest level standalone. But PWAs also offer the opportunity to try and become full-fledged apps on your phone or tablet. In that case, there are additional levels of control, each providing less browser UI and depending on you to do all the work until you reach the fullscreen level of control - where your app takes up the whole screen.

According to the Mozilla docs, there is also a browser option that it claims is the default. Chrome says this is an error. I am going to the reality that Chrome provides rather than "the way it should be". Your mileage may vary.

We, of course, are going to go for standalone, since we're not doing anything fancy at all.

Icons

Icons are the bane of my life. Mainly because I'm not at all artistic. But also because there are all sorts of arcane rules about what is and is not allowed - sizes, formats, etc - all of which vary between platforms.

For this project, I used an  online icon generatorto generate a package of icons. As it happens, it also generates a minimal manifest which would not be an unreasonable place to start. But instead, I just used the icons and copied and edited the relevant sections of the manifest into my manifest.

That completes a minimal manifest. This is tagged  PWA_MINIMAL_MANIFEST.

May I Be of Service?

The final thing that Chrome complains about is the absence of a service worker. What is one of these? Well, it is the thing that is key to making a web app progressive, and it's really what we've been building up to.

Web applications, like other UI applications, have a "main thread" or "UI thread" that is responsible for dealing with user interactions and handling display. If you have done any amount of JavaScript (or other UI development), you will know that it is important to keep things simple and short on the main flow in order to keep response snappy. If you have done quite a bit of JavaScript, you will know that because there is only one thread, it can be difficult.

Workers solve this problem by offering other execution environments in which it is possible to do work that does not interfere with the main flow. And it really is separate and does not interfere: nothing is shared between these two environments except for a message passing mechanism. In a sense, they are like iframes without any visual component.

In order to handle all the things that Progressive Web Apps need to handle without interfering with the main application rendering cycle, it is necessary for them to have at least one service worker and to identify this in the manifest.

Oddly, the service worker is not declared in the manifest but rather must be created from main thread JavaScript code. (To be honest, there is nothing odd about this at all. It used to be declared in the manifest, but the declaration became inadequate to cover all the registration cases so has been deprecated.)

So at this point we need to create two JavaScript files, which I'm calling start.js and service-worker.js. Note, however, that only the start.js finds its way into a script tag in index.html. This is because service_worker.js is not loaded into the main body of the page but into a background "page".

Starting with the service worker (because it's simpler for now), all we want to do to begin with is identify that we have in fact been loaded.

console.log("hello  from  the  service  worker");

All the hard work (for now) is getting that to load.

Loading the service worker

First off, not all browsers support service workers (really? You know this is 2020?). Well, possibly they all do now, but you can't be sure, so first test that the functionality we are going to use - the serviceWorker property of navigator - has been defined. If it has, then add a callback when the document is loaded to try and load the service worker. Note that while it's not strictly necessary to wait until the whole page is loaded, it makes sense because you will probably want to have things happen and you don't want the page to not have loaded the elements that you need. You may also want to add other setup and configuration to this callback before you register, just to make sure that everything happens in the right order.

if  ('serviceWorker'  in  navigator)  {
    window.addEventListener('load',  ()  =>  {
        ...
    });
}

The final thing we need to do in there is call the register method on the serviceWorker property of the navigator. This needs the path to the JavaScript file to use (service-worker.js) and returns a Promise than will contain the registration if successful - or an error if not.

navigator.serviceWorker.register('/js/service-worker.js')
    .then((registration)  =>  {
        console.log("have  registered  service  worker",  registration);
        registration.update().catch(ex  =>  console.log(ex.message));
    }).catch(ex  =>  {
    console.log("failed  to  register  service  worker",  ex.message);
});

While not exactly idempotent, the register method can be called regardless of whether there is already a service worker installed. This may seem bizarre, but remember that PWAs can be stored locally and thus can be rerun in different scenarios. Anyway, go ahead and call it and it will do the right thing.

But what it doesn't generally do is to check whether the JavaScript is up to date. It will generally look at the local cache and accept whatever version is there. This is absolutely fine if you are not connected to the internet (offline working is the first benefit of using a PWA) and is OK if you are a casual user of an application. But it is not at all good if you are actively developing. To bypass the cache, the returned registration has an update method that says "if you can, go and check if there is a more up-to-date version out there".

If all goes well, you should now see messages come out in your console.



This is all available as  PWA_MINIMAL_WORKER.

Conclusion

That is pretty much as minimal as you can make a Progressive Web App. Obviously, I have no intention of stopping there, so read some of the other posts in this thread.

Understanding TextMate Grammars

When I was working through the example VSCode syntax highlighter, I observed that the grammar was specified using "TextMate grammars" and noted that I had never heard of these. Research was called for.

My particular challenge was to figure out how to match the syntax of a language which leans heavily on indentation. It seems hard to find the end of a block using the definition that "it doesn't have more tabs than this one".

Excuse my ignorance (as it happens, this is a rule of reading this blog - I am exposing my ignorance precisely to help others) but it turns out somebody has already come up with a solution to this problem. It is apparently called  negative lookahead. As explained in the link, the idea of "lookahead" regular expressions is that they can check whether or not a pattern occurs but do not consume the matched characters. Thus, providing a negative lookahead pattern for the end pattern of a rule which matches one level more of nesting will cause the block to end on the previous line. I think.

I gained this insight from a  separate web discussion which simply went ahead and  used these expressions to achieve this objective without really explaining what they did.

Breaking Down the Example

So what does it do? Well, first off, I ran it to check that it did actually work in the way I was expecting. However, it doesn't really do what I want.

This is the specification of the rule in some cson format that I don't entirely understand but appears to be some form of simplified JSON:

'indent-block': {
  name: 'indent-block.test-grammar'
  begin: '^(\\s+)(?=\\S)'
  end: '^(?!\\1\\s+)(?!\s*$)'
  patterns: [
    {include: '$$self'}
  ]
}

So this says that a block starts on a line that has a specific amount of leading whitespace and ends at the first line which starts with no more whitespace and has exactly one token on it. This is good for matching blocks such as

  while true
     …
  done

because the begin rule will match while true and then end rule will match done.

Unpicking the patterns, the begin pattern wants to match text which:

• starts at the beginning of the line;
  
• has at least one whitespace (\\s) character;
  
• is followed by at least one non-whitespace character which is not included in the pattern.
  

This allows the block to contain arbitrary matches for things like while as well as inner blocks.

The end pattern wants to match text which:

• starts at the beginning of the line;
  
• does not begin with more whitespace than the begin line (\\1 represents the pattern which matched the inside of the parenthesis in the begin pattern);
  
• contains arbitrarily many non-whitespace characters until the end of the line.
  

None of this (except the beginning of the line) is actually part of the pattern, leaving the remaining text to belong to an outer block (although I'm not sure why this is desired).

I did not come to a positive conclusion, but it seemed to me several times during my debugging sessions that if the beginning of line anchor is matched, it cannot be used again. That is, it seems that VSCode treats the beginning of line anchor as it would any other character and once it is matched it cannot be matched by a subsequent pattern.

Making it Work for Me

But this is not the pattern I want to match.

First off, I want a block to include all the lines after this one that are indented more than this one. And I want the block to end the moment it sees a line that is indented the same amount as this one. But I want it to end before that line starts.

To add to the complexity, I'm happy for blank lines - or lines with non-whitespace text in column 1 - to appear within a block, so any line which is blank or doesn't start with whitespace is not an end marker.

An additional challenge for me personally is a mental model shift: I'm used to thinking outside-in: breaking up the document into large chunks, then breaking those up into smaller chunks and so on. But this mechanism for matching regular expressions approaches the problem in a more inside-out way: the outer blocks "do not get a look in" until the innermost block has reached its end.

So a block is a sequence of lines which starts with a line with at least one leading tab character and continues until it finds a line which has:

• at least one leading tab character;
  
• no more leading tab characters than the begin line had;
  
• is not blank (consisting entirely of whitespace characters).
  

Finally, it does not actually consume any of this, leaving it all for the next start line.

So the begin pattern is fairly easy to write:

^(\t+)(\\S+)

This only matches lines with at least one leading tab and at least one non-whitespace character after the leading tabs. The second capture group is simply there to capture the "keyword" for highlighting to make it clear where the blocks begin.

The end pattern is considerably more complex:

(?=^\\1\\S)|(?=\t(?!\\1\\S))

This is an OR of two positive lookahead conditions.

The left hand condition says that the block will end if the putative end line starts with exactly the same number of tab characters that were in the block opener, followed by a non-whitespace character. The right hand condition says it will also come to an end if the line begins with a tab and does not then contain the full block opening.

Put another way, the left hand condition will end the block when a line appears with the same indentation; the right hand condition will end the block when a line appears with less indentation (but at least one tab, thus ignoring blank lines and literate comments).

I have put this rule in the grammar for .st files and you can experiment with that.

Once again, I want to point out how very hard debugging all this is: in particular, there are some things you can do with lookahead patterns (I'm not sure exactly what) that cause the syntax highlighter to just "give up" and when you try and open the inspector, it just sits there saying "loading". Your only option at this point is to quit the editor and try a different pattern.

It also really annoys me, both from a development and a documentation standpoint, that you cannot add comments to a JSON file.

But what I Really Want is …

Having put all that hard work in, I discovered that it wasn't what I really wanted.

The reason, of course, is my mental model is wrong. As I noted above, I tend to think "outside-in", so I feel that if I can break the code up into blocks, then I can tackle one block at a time. This is not the right way to think about the problem.

Instead, what I need to do is to match each type of block that I have, and then provide it with a set of included patterns. Since my grammar does, in fact, distinguish quite significantly between blocks, this is actually fine.

But all the effort was not wasted, because pretty much every pattern is going to be a variant of what we saw above. One simplification that arises is that for many of the blocks, I know exactly what the acceptable indentation levels are, so I can just use that directly rather than the \\1 matching. On the other hand, I've made it more complex by trying to catch explicitly some versions of invalid indentation.

Another Challenge - Nesting

Another challenge I am facing is: "how do rules nest"? That is, if I specify multiple sub-patterns for a rule, in what order are they selected?

Again, a lot of this just doesn't matter. We are not fully parsing the code, trying to determine its structure and meaning, and rejecting invalid programs; we are just trying to highlight things according to their category. So, except for going deep into blocks, it is generally enough just to say "ah, this is a type" or "ah, this is a variable".

A couple of things do seem to matter, though. One is that I want end-of-line comments (starting with //) to be selected in preference to anything else. Another is that I want to handle blank and literate comments as a priority. And so, by trial and error, I determined that the rules seem to be processed in the order in which they are presented in the patterns block for a given rule. And, as noted above, once a rule has been selected, no other rule will be selected until it has finished (except for those nested within it).

Creating a grammar

Taking all these points together, I looked at the sample .fl file and created a grammar which seemed to cover the subset of rules that I had relied on.

This is now installed in flas.json and can be seen in the repository.

Making it real

My original plan was to generate a grammar file from my formal grammar, but having experimented in this way, I've realized two things:

• the impedance mismatch between a formal grammar and the regular expressions used by VSCode is vast and an automatic conversion between the two would not be easy;
  
• having sunk the investment that I have into figuring it out, it actually isn't that hard to define rules in this way, and I might either do it by hand or invest in a quick conversion tool that automates all of the hard work of JSON formatting and repetition and checks my work.
  

Conclusion

It's not really news, but  doing anything with regular expressions is a pain.

That is particularly true when you are bending them out of shape to do something of the job of an actual parser.

But provided you can bend your mind around it all, it is perfectly possible to come up with an acceptable set of regular expressions for an indentation-based language.