Saturday, July 22, 2023

Importing ICS calendars

ICS is a standard format for specifying calendar information over the web. It is possible to store and download such files using HTTP and, by extension, AJAX. We can use that to pull information from the internet and display it on the calendar in the appropriate place.

There is a lot of complexity here that I can't be bothered to document, but I think it is a little bit interesting to do the absolute minimum: pull a calendar from somewhere and extract the basic "start date" and "summary" information to show a time and summary in the appropriate box.

So, first off, we need to add an input box to provide the URL where the calendar is to be found, and a button to do the loading. When this is triggered, we want to fire an AJAX request to that location and load the calendar. I think this is all fairly straightforward.

Because my life revolves around the Oldham Athletic home schedule, I am using their 2023/2024 season calendar as my example here, and I've checked it into the repository under data/Oldham_Athletic.ics. So, if you've set up the server as described above, you can now access this as http://localhost:8080/data/Oldham_Athletic.ics. Note that you can't use AJAX if you are looking at the page directly from the file system.

So now we need to process this file and turn it into a sequence of calendar events. I'm not going to claim that I understand the ICS format fully - or suggest that you should. But it's a fairly simple blocked, key-value text format that you can figure out if you read it. Each block consists of a pair of BEGIN:type and END:type. The outermost block type is VCALENDAR, and then within that are a sequence of EVENTs. Within that are more fields and more blocks, particularly VALARM blocks. We are going to parse this into a structure which has fields and blocks, with each inner block being the same. The outermost structure corresponds to the VCALENDAR. We can then do later processing based on this.

I have four observations as I write this code. The first is that, unlike most of the code we've written so far, this code really feels like it could use TDD in its development. Most of it is text based and involves pushing and popping stacks and any number of possible bunnyholes and error cases (particularly in the shape of malformed data). I'm going to resist that temptation for simplicity of description of nothing else, and hope for the best.

The second has to do with those error cases themselves. Given that so much can go wrong (not least the AJAX call itself), we need a method of providing error feedback to the user. Because we haven't needed it so far, we don't have it, and that calls for careful thought and design and probably testing. I'm going to admit I'm lazy and probably come back and do that later when it starts biting me (and I know what kind of feedback I'd like) and when I start actively styling the controls pane.

The third observation is that this code is all kind of generic. Most obviously, making an ajax call, for which I ripped off an implementation (I hesitate to call it a library) which I wrote for another project several years ago and now just pull out whenever I want some AJAX stuff doing. It's not jquery, but it's not that big either. But all the rest of the code is just processing what is basically a fairly simple hierarchical key/value file format.

Which leads to the fourth observation that this code is so boring I can't be bothered to describe it. But I will include it here in all its glory (although not the AJAX implementation - you can look at ajax.js for that). This is all tagged PLANNING_CALENDAR_PARSE_ICAL.
function loadICS() {
  var urlEntry = document.getElementById("ics-url");
  var url = urlEntry.value;
  ajax(url, handleICS);
}

function handleICS(status, response) {
  console.log("ajax returned status", status);
  if (status / 100 == 2) {
    var lines = joinLongLines(response.split(/\r\n/));
    var blocked = makeHierarchical(lines);
  } else {
    // TODO: handle status error cases somehow (e.g. 401, 404, 500)
    console.log("error status", status);
    console.log(response);
  }
}

function joinLongLines(lines) {
  var i=0;
  var ret = [];
  while (i < lines.length) {
    var s = lines[i];
    while (++i < lines.length && lines[i].startsWith(" ")) {
      s += lines[i].substring(1);
    }
    ret.push(s);
  }
  return ret;
}

function makeHierarchical(lines) {
  var ret = { type: "VCALENDAR", blocks: [], fields: {} };
  if (lines[0] != "BEGIN:VCALENDAR") {
    console.log("not a calendar"); // errors
    return null;
  }
  var curr = ret;
  var stack = [ ];
  for (var i=1;i<lines.length && lines[i] != "END:VCALENDAR";i++) {
    var kv = lines[i].split(/:/);
    if (kv[0] == "BEGIN") {
      var next = { type: kv[1], blocks: [], fields: {} };
      curr.blocks.push(next);
      stack.push(curr);
      curr = next;
    } else if (kv[0] == "END") {
      if (kv[1] != curr.type) {
        console.log("line",i,"mismatched BEGIN",curr.type,"and END",kv[1]);
        return null;
      }
      curr = stack.pop();
    } else {
      curr.fields[kv[0]] = kv[1];
    }
  }
  return ret;
}
Now we have a "calendar" that probably contains a set of "events". But it's not really a calendar as such, because we expect a calendar to be a set of dates. The date is contained in each event (in the DTSTART field) and a brief description of the event is in the SUMMARY field. What we want to do is extract these and build a table which is indexed by date, where every date has a list of events that happen on that date.

This is, again, very generic code and I don't feel the need to comment on it other than to point out (for those at the back) that this is code which is about calendars and dates and times, and thus issues with timezones arise. I'm not convinced that there is a "correct" solution to this, but the best solution would be to allow our calendar app to have an interface in which you could choose which timezone should be used for each day (if you don't use your calendars to plan trips across and between continents, your life is easy in more ways than one), or at least per calendar, but I am just going with the tried-and-true use the timezone you have set right now. One of the implications of this is that events might end up on a different date to the one in which they are written (e.g. an event at 19:45 in England could start at 06:45 the next morning in New Zealand). For this reason, we read the dates (which should be in GMT according to the spec) and translate them into Date objects, and then format that as the date string we will use as the key.

This code is checked in as PLANNING_CALENDAR_TABULATE_ICS and the tabulation function is shown below. The toStandard method converts the "standard ISO" date format used by ICAL to the "standard ISO" format expected by Javascript - adding in - and : characters between elements.
function makeTabular(blocks) {
  // This assumes that blocks is a VCALENDAR containing VEVENT objects
  // It will ignore other events

  var ret = {};
  for (var i=0;i<blocks.blocks.length;i++) {
    var b = blocks.blocks[i];
    if (b.type != "VEVENT")
      continue;
    var starts = new Date(Date.parse(toStandard(b.fields["DTSTART"])));
    var date = starts.getFullYear() + "-" + (starts.getMonth()+1).toString().padStart(2, '0') + "-" + starts.getDate().toString().padStart(2, '0');
    var time = starts.getHours().toString().padStart(2, '0') + starts.getMinutes().toString().padStart(2, '0');
    if (!ret[date]) {
      ret[date] = [];
    }
    var entry = { time, summary: b.fields.SUMMARY, fields: b.fields };
    for (var j=0;j<ret[date].length;j++) {
      if (ret[date][j].time < entry.time) {
        ret[date].splice(j, 0, entry);
        entry = null;
        break;
      }
    }
    if (entry) { // was not spliced in, append
      ret[date].push(entry);
    }
  }
  return ret;
}

function toStandard(date) {
  return date.substring(0, 4) + "-" + date.substring(4,6) + "-" + date.substring(6,11) + ":" + date.substring(11,13) + ":" + date.substring(13);
}
Now all we have to do is to make sure that these entries appear in the calendar. This consists of four relatively simple steps:
  • First, maintain a central registry of all the calendars we have loaded; this is almost as simple as it sounds, but we need to be sure to replace rather than duplicate calendars if the same URL is loaded twice;
  • Secondly, we need to remember to call redraw() when this is done;
  • Thirdly, redraw() needs to consider for each date cell whether or not there are any events for that day and, if so, to add divs for them within the box for the relevant day;
  • Fourthly, we need to add styles for these, which may involve dynamic metrics.
The main questions that occur with the first couple of items are "who has the responsibility for storing this registry?" and "who calls redraw?". We don't actually have any classes to take responsibility for anything, so the first is basically, "where do I put the code?" and I don't have a good answer for that either, but in controls.js seems the place where it is closest to most of the things it is associated with. On the second point, I much prefer to have calls to methods like redraw() as close to the event handler as possible. In this case, that is at the end of handleICS.

Again, most of this code is uninteresting, but I might as well rattle through it.

When we load the calendar, we call the registration function, and then redraw():
function loadICS() {
  var urlEntry = document.getElementById("ics-url");
  var url = urlEntry.value;
  ajax(url, (status, response) => handleICS(url, status, response));
}

function handleICS(url, status, response) {
  console.log("ajax returned status", status);
  if (status / 100 == 2) {
    var lines = joinLongLines(response.split(/\r\n/));
    var blocked = makeHierarchical(lines);
    if (blocked) {
      var table = makeTabular(blocked);
      addCalendar(url, table);
      redraw();
The registration function itself is very simple and stores all the calendars in a global object calendars:
function addCalendar(url, cal) {
  calendars[url] = cal;
}
redraw() first compiles a list of events on each day it displays:
      var calDate = cellDate.getFullYear() + "-" + (cellDate.getMonth()+1).toString().padStart(2, '0') + "-" + cellDate.getDate().toString().padStart(2, '0');

      var toShow = [];
      for (var url in calendars) {
        var cal = calendars[url];
        var today = cal[calDate];
        if (!today)
          continue;
        for (var j=0;j<today.length;j++) {
          var next = today[j];
          for (var k=0;k<toShow.length;k++) {
            if (next.time < toShow[k].time) {
              toShow.splice(k, 0, next);
              next = null;
              break;
            }
          }
          if (next != null)
            toShow.push(next);
        }
      }
      console.log("calDate is", calDate, "have", toShow);
And then adds divs for these events:
      if (toShow.length > 0) {
        var events = document.createElement("div");
        events.className = "body-day-events-container";
        day.appendChild(events);

        for (var j=0;j<toShow.length;j++) {
          var event = document.createElement("div");
          event.className = "body-day-event";
          events.appendChild(event);
          var timeText = document.createTextNode(toShow[j].time);
          var timeSpan = document.createElement("span");
          timeSpan.className = "body-day-event-time";
          timeSpan.appendChild(timeText);
          event.appendChild(timeSpan);
          var eventText = document.createTextNode(" " + toShow[j].summary);
          var eventSpan = document.createElement("span");
          eventSpan.className = "body-day-event-text";
          eventSpan.appendChild(eventText);
          event.appendChild(eventSpan);
        }
The formatting is slightly more tricky. As seen above, we divide the text for each event into two spans (one for the time, one for the summary) and give each of these and the enclosing div a class name. Each of these can be separately styled. To begin with, we make sure the div has absolute positioning, and we make the time bold. With this, all that is left is the sizing and positioning.
.body-day-events-container {
  position: absolute;
}

.body-day-event {
  border: none;
}

.body-day-event-time {
  font-weight: bold;
}

.body-day-event-text {
  font-style: normal;
}
The sizing and positioning just builds upon what we have already done to show the date box. All the events for the day are contained in a single box so that is the only thing we need to position and can contain a font-size declaration which will then cascade to inner boxes. So, when we are building our custom stylesheet, we can simply add a rule for this box.
  var eventsContainerY = 2 * ypos;
  sheet.insertRule(".body-day-events-container { top: " + eventsContainerY + pageSize.unitIn + "; font-size: " + dateSize + pageSize.unitIn + " }");
  sheet.insertRule(".body-day-events-container { top: " + eventsContainerY + pageSize.unitIn + "; font-size: " + dateSize + pageSize.unitIn + " }");
And that's it for calendar and events handling. It's all checked in and tagged as PLANNING_CALENDAR_SIZE_EVENTS.

No comments:

Post a Comment