Lambda Snapstart is Harder than I Thought

Apologies, but kind of in violation of my rules, I don't have any actual example working code. That's because this is more complex than that and the number of moving parts were just too vast. But if you want any help or conversation about this, drop me a line and I'll do/share whatever I can.

When I saw that AWS had released a "snapstart" feature for Lambda, I was ecstatic. I have taken to using Lambda as a way of delivering servers with minimum fuss, but I have somewhat abused the technology by basically moving my existing server-based code into a lambda, along with its long start time.

I grew up in a world where the logic has always been that what you want to optimize is the time spent doing frequent operations; initialization time is effectively "free" (up to the point where it becomes minutes or hours: at which point you really have to do something about it). Lambda, on the other hand, says you have 10s: if you fail to complete in this time, you will be rejected and the whole thing starts again. When you are trying to configure multiple things with multiple services … it doesn't quite make it.

I finally went over the edge when I found that in order to support the latest version of JavaFX, I needed to copy a whole bunch of ".so" files from S3 to the "local disk". This was taking pretty much all the 10s … As I started to consider my options (I'd got as far as panicking, and decided that was not very productive), AWS announced the "SnapStart" feature: initialize once, use repeatedly. Excited, I turned it on for my functions (so excitedly, I just did it in the console, rather than using CloudFormation; more on that later).

And nothing happened. Or, at least, I continued to have problems. But why?

It only works on published functions

Uh, boss, it's not as simple as that. On the upside, it is very clear that the 10s timeout does not apply to SnapStart functions - you have up to 900s to be used when the function is published. On the downside, this implies and, indeed, it is actively stated that you need to publish your functions in order to take advantage of this. Except my test environment does not bother with publishing and versioning lambdas, so I still want to bring that initialization time down.

The Runtime Hooks

Before doing that, I decided to at least try and integrate the runtime hooks and issue tracing messages. My thought process was firstly to see if these were called when the lambda wasn't published, and even if they weren't, I would then at least know when I had successfully published the lambda, because my tracing would come out.

In order to turn on the runtime hooks, it is necessary to download the CRAC library and attach this to your project. The AWSHandler then needs to implement the CRAC Resource interface, and register with the CRAC global context. So I added code like this:

    public  AWSHandler()  {
        Core.getGlobalContext().register(this);
    }

and then it's a simple matter of implementing the callbacks provided in the Resource interface:

    public  void  beforeCheckpoint(org.crac.Context<?  extends  Resource>  arg0)  {
        logger.info("before  checkpoint");
    }
    public  void  afterRestore(org.crac.Context<?  extends  Resource>  arg0)  {
        logger.info("after  restore");
    }

When It Works …

I added the relevant code to publish and alias my lambdas, and also to ensure that the code inside APIGateway called the alias (and thus the published version), rather than the "$LATEST" version, and, lo and behold, it all worked.

During the publication, this happens:

INIT_START  Runtime  Version:  java:11.v15                Runtime  Version  ARN:  arn:aws:lambda:us-east-1::runtime:0a25e3e7a1cc9ce404bc435eeb2ad358d8fa64338e618d0c224fe509403583ca
Picked  up  JAVATOOLOPTIONS:  -Dui4j.headless=true
-Dglass.platform=Monocle
-Dmonocle.platform=Headless
-Dprism.order=sw
-Djavafx.cachedir=/tmp/solibs
20230104-13:06:48.640                    tdaserver/Thread-0  INFO:  In  config

The key thing here being the "INIT_START" rather than just "START". Interestingly, it doesn't seem to issue any message when the initialization is done, it just stops issuing messages.

And then, when the lambda is called during API Gateway access, I see this:

RESTORE_START  Runtime  Version:  java:11.v15                Runtime  Version  ARN:  arn:aws:lambda:us-east-1::runtime:0a25e3e7a1cc9ce404bc435eeb2ad358d8fa64338e618d0c224fe509403583ca
RESTORE_REPORT  Restore  Duration:  383.49  ms
START  RequestId:  c55dfaca-70b4-4a7e-8e4e-b6fc920269aa  Version:  1
…
END  RequestId:  c55dfaca-70b4-4a7e-8e4e-b6fc920269aa
REPORT  RequestId:  c55dfaca-70b4-4a7e-8e4e-b6fc920269aa                Duration:  785.60  ms                Billed  Duration:  1044  ms                Memory  Size:  1024  MB                Max  Memory  Used:  401  MB                Restore  Duration:  383.49  ms               

Here the RESTORE_START and RESTORE_REPORT make it clear that a SnapStart image is being used, and how much time has been used to literally start the lambda (383ms may seem a lot, until you realize that it was over 15000ms to actually do the initialization).

After that, the lambda proceeds in the normal way.

No CRAC Output

Interestingly, up to this point, I have not seen any of the tracing output I would expect from my CRAC callback. I don't know whether I didn't succeed in registering correctly, or whether my tracing is simply not coming out. In the fullness of time, I will need to sort this out because it is necessary to check that all the initialization that has been done up to this point is up to date.

Configuring from CloudFormation

As a bleeding-edge adopter, when I first tried to use SnapStart, there wasn't any active CloudFormation documentation on using it. For all I know, it wasn't supported in CloudFormation. However, now, a couple of months later, all the relevant documentation is there.

As you'd expect, you configure this by adding a SnapStart property to your Lambda Function configuration which is quite simple so that it basically amounts to adding:

"SnapStart":  {
    "ApplyOn":  "PublishedVersions"
}

to your existing function declarations.

Conclusion

For a long time, Lambda on AWS with Java has been plagued by painfully slow startup times. It does seem that SnapStart makes major strides towards addressing these and, provided you are publishing your lambdas, is relatively easy to set up.

On the other hand, it seems somewhat opaque to use.

A More Functional Database

I have recently been using DynamoDB and, to be frank, its API (at least in Java) is, IMHO, awful. So awful, in fact, that in wrapping the API as described below, I was forced to write a mini-wrapper to call from within the wrapper.

Obviously, I was not going to put up with this. But rather than launching in and doing "something", I sat back and considered what I really wanted. If you know me or read this blog frequently, you would probably fairly rapidly come up with a list something like this:

• Functional or functional-leaning;
  
• Tell-Don't-Ask;
  
• Transactional;
  
• Asynchronous.
  

Now, as it happens, these things are not really in conflict in this case, although I ended up with three APIs:

• A mini-wrapper I had to create just to tidy up the $DynamoDB$ interface;
  
• A low-level "tell-don't-ask" API; and
  
• A more transactional API with a functional feel.
  

What I really want to talk about here is the last of these; the other two are really just stepping stones on the way, although, for full disclosure, it was only when I had used the TDA API for a while and experienced the pain of doing it, that the third API occurred to me.

While I’m fully disclosing, this code didn't start here as an experiment, but grew organically out of one of my many “real life” complex and messy projects across a swathe of production code intended to support multiple databases as well as other integrated systems; for simplicity I’ve rewritten history a little and pulled out all the relevant classes for DynamoDB and assembled an example test case that steps through the basic CRUD flow using the functional API.

What am I thinking?

First things first. What do I even mean by a "functional" database? Almost by definition, databases are the opposite of functional. The very way we think about - and describe - databases is CRUD: a sequence of read/update operations with occasional creation and deletion to keep life interesting.

Given my long background in functional programming, a lot of things have been percolating in my brain over the decades and recently have experimented a little with (the somewhat-functional) FaunaDB (which I will eventually get around to writing up here) and in doing that I noticed something of a similarity between functional programs and how I might ideally write database logic.

Here is some functional code:

top = repeat x '*'
x = 5
repeat 0 c = []
repeat n c = c:repeat (n-1) c

Nothing particularly special here. But there is a rhythm to functional programs, which can be described as "define something; use it; repeat". I've often described writing functional programs as "do a little bit of the work now and push the rest off onto another function".

On the other hand, what drives a functional program is the fact that somebody somewhere wants to know a result and expresses this somehow. Generally, this either comes from a "main" method or from some kind of console or REPL. The "top level" expression is broken down into sub-expressions which are evaluated in turn until the basic blocks are reached.

(As an aside, TDA almost exactly inverts this logic: it takes all the basic blocks and says that the results should be sent to a consolidator that combines them and then promotes them to the next level until the top level handler is reached.)

So the question is: how does this relate to databases?

A "pull" model for databases

The normal way of dealing with databases is as an imperative begin-read-write-commit loop. But this transaction can also be viewed as a single operation in a REPL - the model used by functional programs. In this model the transaction becomes:

• Decide what you want to do
  
• Do all the reads and transformations
  
• Do all the writes in a single step
  

How is this different? Most importantly, an important topic in standard database theory isread your writeswhich basically says that there is a choice when you have done a write in a transaction about whether when you use "the value" again - particularly if you choose to read it back from the database - whether you see the version you wrote or the one that existed before your transaction started. By deferring all the writes until the end of the transaction, we avoid this conundrum, which is exactly what you would expect from a functional model, in which values do not change over the course of a function's evaluation.

The other major change is in the way in which we describe the steps of the logic. As with a functional program, we name each value that we read in; as we name it, it becomes available for other steps in the logic. For example, a transaction which reads two values and then writes the result might ordinarily look like this:

begin tx
x = get 'A'
y = get 'B'
z = x + y
put 'C' z
end tx

In a functional notation, we might write something like this:

tx = [Store 'C' z]
z = x + y
x = get db 'A'
y = get db 'B'

Now, this is neither purely functional or really executable; but the idea is that we are describing the transaction rather than actually running it.

Handling the impedance mismatch with imperative languages

Now, while I want to use a "more functional" approach to databases, I in fact still want to do this from within Java - an imperative language. How do I handle that?

It's actually not all that hard. Java has functions, and I can say that each of my "steps" can map to a function. I have an entry point, which kicks off the initial GET operations and identifies the logic operation (z) that I want to be invoked when both of the return values are available. I then define z and annotate its parameter arguments to say where they come from. Basically, the transaction becomes a state that the functions can access through the parameter annotations. Once a value is read or calculated, it is available and any method that depends on that can now be executed.

Handling asynchronicity

It's obviously very important that the database access be asynchronous - the alternatives are to waste threads blocking or just to kill performance - which aren't really alternatives at all.

This is done by having all the functions call into an asynchronous layer in the database and provide a central place to call back. When they do, the current state is updated with the new value and the list of pending logic calls is examined. Any that are now "ready" - i.e. those that were just waiting for this value to arrive - can now execute, possibly launching more GET or LOGIC requests. Any of these that are ready to execute can be run the moment that the current method returns; others will be deferred until all their arguments are present. Every logic method can also add to the pool of "pending writes".

Eventually, all of the logic is complete and it is possible to do the writes - unless an error occurred in the transaction, in which case none of the writes will be done.

An implementation

The implementation (available in the git repository) is called GLS for get-logic-set/subscribe, describing this alternative loop.

Start with the simple test case (SimpleGLSUsage). The fundamental concept here is the UnitOfWork, which is created in the test initializer initTest. This corresponds to the conventional notion of a "transaction" (I have shied away from the word transaction in part because it is overloaded and in part because the semantics of the underlying database are unspecified; in the case of DynamoDB it is not transactional).

Within a unit of work, it is possible to create multiple parallel Relations. A relation represents a thread of work going on with its own namespace - a scope, if you will, within an outer definition in a functional program. This allows multiple lines of logic to coexist while still being part of the same "unit" - the same values are only read once, have the same value across all relations, and the unit succeeds or fails together.

This simple test case is not a series of unit tests. Rather, it is a "script" for executing tests. To make this work, JUnit 5 is used along with its OrderAnnotations. The first test ensures that we can create a trivial object. The enact method on the unit of work says that we have configured everything, and it is ready to go; waitForResult blocks until the unit has completed - success or failure.

The second test simply attempts to read this object back in.

The third test reads the object back in and then prints it. Because Java does not treat functions as "first-class" objects, the method printHello is referenced as a string in the call to logic. This is the name of a method in the RelationHandler class, which in this case is defined to be this, which is the current class.

The fourth to sixth tests check that the greeting is what we would expect it to be, update it, and check that it has changed. The final test cleans up the record by deleting the record (although if anything goes wrong, the whole thing will be deleted on restart).

It is obviously possible to run these tests, but some setup is required: you obviously need an AWS account with a DynamoDB instance configured; you need to create a table and pass its name to the test in the test.table.name property.

Conclusion

The normal database paradigm fits well with imperative languages, but has the normal drawbacks of those languages - most specifically, very slow, synchronous behaviour. It's hard in imperative languages to break out of that because of the complexity of dealing with all the asynchronous state. Changing the metaphor - and making a central agent responsible for the state management - simplifies the code and improves reliability.

Interestingly, in writing this, I can see that my actual implementation does not map perfectly onto my mental model - my implementation has turned out to be more imperative than my mental model. My entry point is fairly close to the "bottom" of the execution stack - as it would be with a TDA implementation. To match the mental model more closely, the "get" operations should not be invoked directly from the entry point function, but should be their own functions, and each of the functions should be named to match the "value" it produces. Maybe I should try again and report on that experiment.