Deploying Infrastructure

I first thought about using Neptune several months ago, but I ran into an obstacle I'd had with any number of other things. It looked painful to deploy.

I started trying to write a cloud deployer about five years ago, but ended up with a tangled mess of spaghetti because I started "coding things" and thought I could back out into a language-driven abstraction. I couldn't. There is a school of thought that the only valid abstractions are ones that are wrung out of example code. I still find that incredibly difficult.

So, given I had a number of things at the same time all waiting for the ability to deploy AWS infrastructure seamlessly, I decided to hold off on this project until I had made reasonable progress on my deployer. Now that I have, I'm in a position to start playing with Neptune.

The deployer now has its own website, but it is still very much in the alpha state. I'm working on that, and it may be in a lot better shape by the time you are reading this.

In the meantime, I have this configuration that I want to deploy today. Note that is undoubtedly incomplete - indeed, woefully incomplete if you consider that there is nothing in here about web infrastructure. Most importantly, though, I think a Neptune Cluster is insufficient on its own - I think we will need at least an Instance, and possibly an Endpoint. We will come back to those as and when.

I'm optimistic that I have commented the file well enough that I don't need to comment further.

The purpose of this project is to test out how easy it is to "link" DynamoDB tables
together using Neptune, and then to see how easily we can navigate the resulting graph.

This obviously requires a lot of AWS infrastructure.

This is the target to put up (and pull down) all the infrastructure.

    target infrastructure

The sample application calls for two core concepts: "stocks" and "users".  Each of these
requires a table in DynamoDB.

Because DynamoDB tables are relatively simple, we can just use the standard verb "ensure"
along with the concept of a "DynamoDB.Table".  This one is going to be called "stocks".

        ensure aws.DynamoDB.Table "Stocks"
            @teardown delete

DynamoDB has a weird thing about AttributeDefinitions and KeySchema which is that there is
considerable overlap, and only "key" attributes should appear in the AttributeDefinitions.
This smacks of duplication to me, but the consequence is that any fields (such as price here)
that are not _key_ fields are not actually used.

            Fields <= aws.DynamoFields
                Symbol string
                    @Key hash
                Price number

And we have a table of users, which (for now) is just a username.

        ensure aws.DynamoDB.Table "Users"
            @teardown delete
            Fields <= aws.DynamoFields
                Username string
                    @Key hash

Neptune is more complicated.  I expect that when I know more about all the moving parts I will
add a reductive "neptune" verb that makes sure you create all the necessary parts or gives you
an error.  But for now, I'm just going to create a cluster.

Neptune has to run "inside" a VPC.  In order to do this, it requires you to create a special
"SubnetGroup" (specific to Neptune; this is not a VPC thing).  This is quite hard to create,
so I did it in the UI and I'm just finding it here and assigning it to the variable "subnet".

        find aws.Neptune.SubnetGroup "neptunetest" => subnet

So now we can create the Cluster.  It would seem that you can create a cluster with little more
than a name and a SubnetGroup.

        ensure aws.Neptune.Cluster "user-stocks"
            @teardown delete
            SubnetGroupName <- subnet

That "appears" to be enough to be getting on with, but I suspect we will be back here soon to
create more elements.

NEPTUNE_DPLY:neptune/dply/infrastructure.dply

In order to use AWS, it's necessary to provide the $AWS_PROFILE environment variable. Deployer handes these kinds of things seamlessly by allowing you to store collections of environment variables in files and then reference those files on the command line. So I have this:

AWS_PROFILE=ziniki-admin

NEPTUNE_DPLY:neptune/dply/gareth.envs

And this, of course, references the relevant (secret) information in my ~/.aws/credentials file.

And this can all be deployed by running the command:

$ $DEPLOYER/driver/cmd/deployer/deployer -m $DEPLOYER/coremod/plugin/coremod.so -m $DEPLOYER/../deployer-module-aws/plugin/deployermoduleaws.so -e gareth -d dply infrastructure

So, yes, that's ugly. The thing is, the deployer is built to be very modular - so you need to specify which modules to include. I am thinking of adding a modules file to the directory that is read and loads in the specified modules. But in the meantime I have just put this in a script:

#!/bin/bash

$DEPLOYER/driver/cmd/deployer/deployer -m $DEPLOYER/coremod/plugin/coremod.so -m $DEPLOYER/../deployer-module-aws/plugin/deployer_module_aws.so -e gareth -d dply "$@" infrastructure

NEPTUNE_DPLY:neptune/scripts/deploy

And when we run this, we see the following:

$scripts/deploy
2025/07/10 12:15:24 loading scripts from dply
2025/07/10 12:15:24 reading envs from gareth.envs
2025/07/10 12:15:24 setting AWS_PROFILE to ziniki-admin
2025/07/10 12:15:24 want to find dynamo table stocks
2025/07/10 12:15:25 no dynamo table found called stocks
2025/07/10 12:15:25 want to find dynamo table users
2025/07/10 12:15:25 no dynamo table found called users
2025/07/10 12:15:25 want to find neptune cluster neptunetest
2025/07/10 12:15:25 subnets = 0xc000351880
2025/07/10 12:15:25 subnet found for neptunetest
2025/07/10 12:15:25 want to find neptune cluster user-stocks
2025/07/10 12:15:26 clusters = 0x0
2025/07/10 12:15:26 cluster user-stocks not found
2025/07/10 12:15:26 have desired neptune config for user-stocks
2025/07/10 12:15:26 asked to create table for stocks: arn:aws:dynamodb:us-east-1:331358773365:table/stocks
2025/07/10 12:15:27 created table arn:aws:dynamodb:us-east-1:331358773365:table/stocks for stocks
2025/07/10 12:15:27 asked to create table for users: arn:aws:dynamodb:us-east-1:331358773365:table/users
2025/07/10 12:15:28 created table arn:aws:dynamodb:us-east-1:331358773365:table/users for users
2025/07/10 12:15:29 initiated request to create cluster user-stocks: creating arn:aws:rds:us-east-1:331358773365:cluster:user-stocks
2025/07/10 12:15:30 status was creating, not available
2025/07/10 12:15:30 waiting another 2s
2025/07/10 12:15:33 status was creating, not available
2025/07/10 12:15:33 waiting another 4s
2025/07/10 12:15:37 status was creating, not available
2025/07/10 12:15:37 waiting another 8s
2025/07/10 12:15:45 status was creating, not available
2025/07/10 12:15:45 waiting another 15s
2025/07/10 12:16:00 status was creating, not available
2025/07/10 12:16:00 waiting another 30s
2025/07/10 12:16:31 created neptune cluster user-stocks arn:aws:rds:us-east-1:331358773365:cluster:user-stocks

This output may change as the deployer evolves. I'm still trying to find the right balance between "this is what's going on" and "this is what happened". At the moment, while I'm working on it, I'm leaning towards the former. In the fulness of time, I expect to move towards the latter with a "verbose" mode to get back to more detailed information.

Conclusion

This episode was a boring but necessary first step to have (some of?) the infrastructure we need available.

We didn't come here to discuss the deployer. So next time, let's get started writing some code.