Ignorance may be Strength : Testing at Scale

I'm not always a fan of "test first" (or even unit testing in general), but one of the things I like to do is "test early", particularly around things that are often left undone or left way too late. Specifically, end-to-end testing and performance testing.

This project being in Go, and goroutines making concurrency so simple, I am going to start my test harness as an all-in-one program, with each node and client running in its own goroutine - and with the clients and servers communicating over HTTP within the same process. When we start deploying nodes to cloud infrastructure, we will revisit this topic and see how we can refactor this into a tool to handle that as well. Knowing that is where I'm going will influence many of the decisions I make today.

A Simple End to End Test

I complained earlier about the pain I always feel having to start a node, then run the client, then stop the node ... and I said life would be easier when we started running tests at the unit level, rather than running the applications. It certainly did, and by the end of the last episode, I'd almost forgotten all the steps I had to follow in order to run the node and client.

Now what I want to do is to put the whole process of running (multiple) nodes and clients in a single "test". We start a node, then launch two clients, each submitting the same transaction but signed as different users: once that has happened, we expect to see the journaller report that it has a transaction stored. For now, that will be success. The actual "testing" of our results will come later (much later).

Enough waffle, let's start building something. We know that:

We want something configurable, so we can test all kinds of different things;
We want to build it out in an agile fashion, so we starting small is absolutely fine;
It needs to be able to run one or more nodes;
It needs to be able to run one or more clients;
We need to have the concept of "users";
The clients need to be configurable to do work, and will need to coordinate so that users can collaborate to create a resolved transaction;
We will want to be able to measure performance.

So building an outline main() for the harness is not hard. It starts by "reading" the configuration; starting up all the nodes; telling the clients to do any preparatory work they need (i.e. loading anything they need in advance, connecting to nodes, etc); start a timing clock; tell the clients to get on with it; wait for them to finish; stop the timing clock; report the performance.

I suspect that there will be other things we need to do (clean up?) but I'm not going to speculate at this point.

In Go, that looks like:

package main

import (
    "log"
    "time"

    "github.com/gmmapowell/ChainLedger/internal/harness"
)

func main() {
    log.Println("starting harness")

    config := harness.ReadConfig()
    harness.StartNodes(config)
    clients := harness.PrepareClients(config)

    startedAt := time.Now().UnixMilli()

    for _, c := range clients {
        c.Begin()
    }

    for _, c := range clients {
        c.WaitFor()
    }

    endedAt := time.Now().UnixMilli()
    log.Printf("elapsed time = %d", endedAt-startedAt)

    log.Println("harness complete")
}

TEST_HARNESS_OUTLINE:cmd/harness/main.go

You may wonder what the various things in harness look like. Well, at the moment, the least we can get away with and still have it compile. Something like this:

package harness

type Config interface {
}

type Client interface {
    Begin()
    WaitFor()
}

func ReadConfig() *Config {
    return nil
}

func StartNodes(c *Config) {

}

func PrepareClients(c *Config) []Client {
    return make([]Client, 0)
}

TEST_HARNESS_OUTLINE:internal/harness/api.go

Not surprisingly, this is blindingly fast. On the other hand, it doesn't do anything. So let's change that. I'm going to start with StartNodes because I don't feel ReadConfig can do anything useful until it knows what somebody expects from Config, and we can't do anything with clients until we have a node.

Starting Nodes

For now (i.e. for this simple test), we are just going to have one node, so StartNodes can just create a node and we are on our way. I seem to remember I have already written this code - oh, yes, it's in cmd/chainledger/main.go. Now, I also seem to remember saying that that should only have a very high-level view of the code and delegate everything else. It seems we failed in that mission.

Let's review:

func main() {
    log.Println("starting chainledger")
    config, err := config.ReadNodeConfig()
    if err != nil {
        fmt.Printf("error reading config: %s\n", err)
        return
    }
    pending := storage.NewMemoryPendingStorage()
    resolver := clienthandler.NewResolver(&helpers.ClockLive{}, config.NodeKey, pending)
    journaller := storage.NewJournaller()
    storeRecord := clienthandler.NewRecordStorage(resolver, journaller)
    cliapi := http.NewServeMux()
    cliapi.Handle("/store", storeRecord)
    err = http.ListenAndServe(":5001", cliapi)
    if err != nil && !errors.Is(err, http.ErrServerClosed) {
        fmt.Printf("error starting server: %s\n", err)
    }
}

TEST_HARNESS_OUTLINE:cmd/chainledger/main.go

And yes, it's a bundle of things that are specific to main() and things that are about a node. I'm going to try and tease them apart and put as much as I can (for now, I'm going to go overboard) into a new struct called ListenerNode which I'm going to hide behind an interface called Node. For now, this code is going to be found in clienthandler.

package clienthandler

import (
    "errors"
    "fmt"
    "log"
    "net/http"

    "github.com/gmmapowell/ChainLedger/internal/config"
    "github.com/gmmapowell/ChainLedger/internal/helpers"
    "github.com/gmmapowell/ChainLedger/internal/storage"
)

type Node interface {
    Start()
}

type ListenerNode struct {
    addr string
}

func (node *ListenerNode) Start() {
    log.Println("starting chainledger node")
    config, err := config.ReadNodeConfig()
    if err != nil {
        fmt.Printf("error reading config: %s\n", err)
        return
    }
    pending := storage.NewMemoryPendingStorage()
    resolver := NewResolver(&helpers.ClockLive{}, config.NodeKey, pending)
    journaller := storage.NewJournaller()
    storeRecord := NewRecordStorage(resolver, journaller)
    cliapi := http.NewServeMux()
    cliapi.Handle("/store", storeRecord)
    err = http.ListenAndServe(node.addr, cliapi)
    if err != nil && !errors.Is(err, http.ErrServerClosed) {
        fmt.Printf("error starting server: %s\n", err)
    }
}

func NewListenerNode(addr string) Node {
    return &ListenerNode{addr}
}

CHAINLEDGER_NODE:internal/clienthandler/node.go

And the main code has come down to this:

func main() {
node := clienthandler.NewListenerNode(":5001")
node.Start()
}

CHAINLEDGER_NODE:cmd/chainledger/main.go

Now I can flesh out the harness code:

func StartNodes(c *Config) {
node := clienthandler.NewListenerNode(":5001")
go node.Start()
}

HARNESS_START_NODE:internal/harness/api.go

What is really important here (and the only difference from the main code) is that we specify that we should go the starting of the node. This is how Go introduces a goroutine (or, if you will, a thread or background execution).

node.Start() calls http.ListenAndServe which, as the name implies, listens for connections on the given port and serves any requests. In doing so, it blocks forever. This is just what we want in main(): we want a process that starts and doesn't stop. But in the harness context, we want to be able to move on to the next step in our test, so the node itself needs to run in the background, which is easily accomplished with go.

Goroutines are like daemon threads in Java: they all stop immediately when the process terminates. I'm not entirely sure what happens about cleaning up resources, but it certainly seems that they release the sockets that they are listening on. Because of this, the harness still runs very quickly: especially since this code isn't even in the timed part of the loop.

Preparing Clients

Now we need to repeat the process with the client. Here is the current code in ledgerclient/main.go:

func main() {
    repo, e1 := client.MakeMemoryRepo()
    if e1 != nil {
        panic(e1)
    }
    uid := "https://user1.com/"
    uu, e2 := url.Parse(uid)
    if e2 != nil {
        panic(e2)
    }
    pk, e3 := repo.PrivateKey(uu)
    if e3 != nil {
        panic(e3)
    }
    cli, err := client.NewSubmitter("http://localhost:5001", uid, pk)
    if err != nil {
        log.Fatal(err)
        return
    }

    var hasher maphash.Hash
    hasher.WriteString("hello, world")
    h := hasher.Sum(nil)

    tx, err := api.NewTransaction("http://tx.info/msg1", h)
    if err != nil {
        log.Fatal(err)
        return
    }
    err = tx.SignerId("https://user2.com")
    if err != nil {
        log.Fatal(err)
        return
    }
    err = cli.Submit(tx)
    if err != nil {
        log.Fatal(err)
        return
    }
    fmt.Printf("submitted transaction: %v", tx)
}

HARNESS_START_NODE:cmd/ledgerclient/main.go

This is going to take a little more effort than the node, because it is basically in three parts to it: setting up the client; creating a message; and submitting the message. And boilerplate code is mixed in with sample data throughout.

First, let's pull out the creation of the client. The first part of main thus becomes:

func main() {
    repo, e1 := client.MakeMemoryRepo()
    if e1 != nil {
        panic(e1)
    }
    cli, err := repo.SubmitterFor("http://localhost:5001", "https://user1.com/")
    if err != nil {
        panic(err)
    }

HARNESS_PREPARE_CLIENTS:cmd/ledgerclient/main.go

The ClientRepository interface needs to have the new member SubmitterFor:

type ClientRepository interface {
PrivateKey(user *url.URL) (*rsa.PrivateKey, error)

SubmitterFor(nodeId string, userId string) (*Submitter, error)
}

HARNESS_PREPARE_CLIENTS:internal/client/repo.go

And MemoryClientRepository needs to implement that:

func (cr MemoryClientRepository) SubmitterFor(nodeId string, userId string) (*Submitter, error) {
    if uu, err := url.Parse(userId); err != nil {
        return nil, err
    } else if pk, err := cr.PrivateKey(uu); err != nil {
        return nil, err
    } else {
        return NewSubmitter(nodeId, userId, pk)
    }
}

HARNESS_PREPARE_CLIENTS:internal/client/repo.go

Turning our attention to the harness code, we need a struct to hold our Client. Since we want it to be configurable, we will call it ConfigClient. And it will ultimately need to implement the methods Begin and WaitFor:

type ConfigClient struct {
submitter *client.Submitter
}

func (cli *ConfigClient) Begin() {

}

func (cli *ConfigClient) WaitFor() {

}

HARNESS_PREPARE_CLIENTS:internal/harness/api.go

And finally, we can do what we came here to do and implement PrepareClients:

func PrepareClients(c *Config) []Client {
    repo, err := client.MakeMemoryRepo()
    if err != nil {
        panic(err)
    }
    ret := make([]Client, 1)
    if s, err := repo.SubmitterFor("http://localhost:5001", "https://user1.com/"); err != nil {
        panic(err)
    } else {
        ret[0] = &ConfigClient{submitter: s}
    }
    return ret
}

HARNESS_PREPARE_CLIENTS:internal/harness/api.go

For now, this just prepares the one client, and it isn't at all configurable, but it's the right code in the right place.

Running the Clients

The idea with the configured client is that it represents a particular user connected to a particular node. When we say Begin, we expect it to generate a whole bunch of messages that will be cosigned by other users. It will submit each of these in turn. All of this will happen in a background thread; once all the clients are launched, the main test function will call WaitFor on each of them in turn to wait for them all to finish before declaring the test over.

For now, I'm just going to copy-and-paste the rest of the code from ledgerclient into Begin, wrapping it in an anonymous function and then putting all of that into the background using the go keyword.

func (cli *ConfigClient) Begin() {
    go func() {
        var hasher maphash.Hash
        hasher.WriteString("hello, world")
        h := hasher.Sum(nil)

        tx, err := api.NewTransaction("http://tx.info/msg1", h)
        if err != nil {
            log.Fatal(err)
            return
        }
        err = tx.SignerId("https://user2.com")
        if err != nil {
            log.Fatal(err)
            return
        }
        err = cli.submitter.Submit(tx)
        if err != nil {
            log.Fatal(err)
            return
        }
        fmt.Printf("submitted transaction: %v", tx)
    }()
}

HARNESS_CLIENT_BEGIN:internal/harness/api.go

So this does the basic job we need of sending a transaction to the node. In the fullness of time, we will need to extend this to be a loop, and then we will want to extract the internals of the function into a proper function - probably in fact a struct - in its own right.

So how do we implement WaitFor? Well, we need some kind of "flag" that we can signal at the end of the goroutine in Begin to say that it is done. How do we do that? Well, the "easiest" thing to do in Go, is to use an unbuffered channel.

The concept of channels comes from CAR Hoare's 1978 paper, Communicating Sequential Processes which he expanded into the canonical book in 1985. Both are well worth a read, and seemed seminal at the time, but looking back over the paper now, it seems somewhat "quaint" in retrospect.

Any book (or probably article) on Go will cover channels, and if you are unfamiliar with them, I think you are best going and reading up on them rather than trying to learn by example, but an unbuffered channel, while it can be used for communicating data, is mainly a mechanism for synchronization, much like a signal on a railway, and is similar to how notify and wait work in Java.

So, in our case, each of our clients is going to be created with a unbuffered channel in it. Because channels can transmit data, they a created with a type, but since we don't want to send any actual data, we will use the "empty" type struct{}. This is a struct with no elements and thus essentially no size. The declaration consists of the field name, the keyword chan and the type of the messages sent down the channel. I have opted to call the channel done because that reflects the intention that a message is sent to notify that the client is done.

type ConfigClient struct {
submitter *client.Submitter
done chan struct{}
}

HARNESS_CLIENT_WAITFOR:internal/harness/api.go

And we can initialize this when we prepare the client. Go uses the make function to create a channel using the keyword chan and indicating the type.

func PrepareClients(c *Config) []Client {
    repo, err := client.MakeMemoryRepo()
    if err != nil {
        panic(err)
    }
    ret := make([]Client, 1)
    if s, err := repo.SubmitterFor("http://localhost:5001", "https://user1.com/"); err != nil {
        panic(err)
    } else {
        ret[0] = &ConfigClient{submitter: s, done: make(chan struct{})}
    }
    return ret
}

HARNESS_CLIENT_WAITFOR:internal/harness/api.go

We send an empty message (struct{}{}) down the channel at the end of Begin() to indicate that it has finished work. The syntax struct{}{} creates an instance of the anonymous type struct{}.

func (cli *ConfigClient) Begin() {
    go func() {
        var hasher maphash.Hash
        hasher.WriteString("hello, world")
        h := hasher.Sum(nil)

        tx, err := api.NewTransaction("http://tx.info/msg1", h)
        if err != nil {
            log.Fatal(err)
            return
        }
        err = tx.SignerId("https://user2.com")
        if err != nil {
            log.Fatal(err)
            return
        }
        err = cli.submitter.Submit(tx)
        if err != nil {
            log.Fatal(err)
            return
        }
        fmt.Printf("submitted transaction: %v", tx)
        cli.done <- struct{}{}
    }()
}

HARNESS_CLIENT_WAITFOR:internal/harness/api.go

The implementation of WaitFor is ludicrously simple. It justs reads a "value" from the channel. The semantics of reading a channel in Go are that if no value is present, the read will wait until a value is placed on the channel. So the code in WaitFor will pause until Begin finishes and sends a message on the channel.

func (cli *ConfigClient) WaitFor() {
<-cli.done
}

HARNESS_CLIENT_WAITFOR:internal/harness/api.go

Running this a few times, I discover there are cases where the node has not opened its port by the time that the client attempts to connect. This is annoying, but unfortunately not uncommon in HTTP applications. To solve this, I used the "usual" trick of adding another endpoint - /ping - to the node which just confirms that it is alive and then adding a loop to wait for the node to be ready. If it is not ready within 10s, the client assumes it will never start and panics.

This isn't very interesting at all, so I'll just say it's in the commit HARNESS_PING_NODE if you want to look.

Signing Other Users' Requests

Our node does not have much logic at the moment, but the one thing it does do is to wait for all the users listed in a transaction to send across the same transaction. If we're going to make this happen, we need at least one submitter for each user, and we need to make sure that they all send the same message across. In the general case, this requires coordination, and more channels, but for now we just want to send one transaction, so we can just do a bit more code duplication.

I promise you that I will soon come back and refactor this to re-establish some kind of order here, but for now, I'm just hacking more and more code in.

The client needs to know who the "other" user is, so we store this in the struct:

type ConfigClient struct {
    submitter *client.Submitter
    other     *url.URL
    done      chan struct{}
}

HARNESS_SIGN_OTHER:internal/harness/api.go

and add it when creating the clients:

func PrepareClients(c *Config) []Client {
    repo, err := client.MakeMemoryRepo()
    if err != nil {
        panic(err)
    }
    ret := make([]Client, 2)
    if s, err := repo.SubmitterFor("http://localhost:5001", "https://user1.com/"); err != nil {
        panic(err)
    } else {
        url, _ := url.Parse("https://user2.com/")
        ret[0] = &ConfigClient{submitter: s, other: url, done: make(chan struct{})}
    }
    if s, err := repo.SubmitterFor("http://localhost:5001", "https://user2.com/"); err != nil {
        panic(err)
    } else {
        url, _ := url.Parse("https://user1.com/")
        ret[1] = &ConfigClient{submitter: s, other: url, done: make(chan struct{})}
    }

    for _, s := range ret {
        s.PingNode()
    }
    return ret
}

HARNESS_SIGN_OTHER:internal/harness/api.go

and then use it in Begin in the call to Signer:

func (cli *ConfigClient) Begin() {
    go func() {
        hasher := sha512.New()
        hasher.Write([]byte("hello, world"))
        h := hasher.Sum(nil)

        tx, err := api.NewTransaction("http://tx.info/msg1", h)
        if err != nil {
            log.Fatal(err)
            return
        }
        err = tx.Signer(cli.other)
        if err != nil {
            log.Fatal(err)
            return
        }
        err = cli.submitter.Submit(tx)
        if err != nil {
            log.Fatal(err)
            return
        }
        fmt.Printf("submitted transaction: %v", tx)
        cli.done <- struct{}{}
    }()
}

HARNESS_SIGN_OTHER:internal/harness/api.go

And now, when I run through, I see a whole bunch of messages in different orders, but I always see this message somewhere:

Recording tx with id [201 121 143 71 154 197 76 196 235 155 240 205 217 149 79 107 21 65 241 143 61 134 66 8 73 198 235 111 241 160 169 204 157 144 192 238 143 87 164 66 170 46 78 180 150 81 12 49 172 129 48 44 236 22 87 81 245 75 202 206 167 35 141 99]

(Not that I always get the same ID, obviously, as it is a hash of values that include a timestamp). At some point, I need to make it so that the hash comes out as a hex or base64 string. I will probably do that at some point when I'm bored and not blog about it.

This is not a Test

The alert among you may have noticed we're not actually testing anything right now. A fair point, but not one that worries me unduly. I could argue that this is "test first", but it's really just a question of having something in place to run the code as it grows and make sure that we don't paint ourselves into a corner. The only thing I think that is worth testing is that some (or all) of the transactions we submitted are in the "provable" blockchain. And we can't do that pretty much until we're finished.

So, not it's not a test, but it does demonstrate that we have something working end to end, and reports an elapsed time (in my case 13ms).

Now Let's Scale Up!

We now have two clients and one node. Let's go beyond that and see what happens. At the moment, the nodes don't communicate with each other, so this is not really all that interesting, but it is the best that we can do at the moment, and it helps reassure us that the nodes don't trample over each other.

This is going to be a mix of more copy-and-paste and some refactoring to get rid of some of our prior copying and pasting.

The first step is to actually define a concrete class to implement this:

type Config interface {
    NodeEndpoints() []string
}

type HarnessConfig struct {
    nodeEndpoints []string
}

// NodeEndpoints implements Config.
func (c *HarnessConfig) NodeEndpoints() []string {
    return c.nodeEndpoints
}

func ReadConfig() Config {
    return &HarnessConfig{nodeEndpoints: []string{":5001", ":5002"}}
}

HARNESS_MULTIPLE_NODES:internal/harness/api.go

I should at this point say that VSCode does seem to sometimes offer a "Quick Fix", specifically when you try and create a struct and return it as an interface where the relevant methods defined in the interface do not exist in the struct, it offers to create them, which is great. But it doesn't seem to happen every time.

And then StartNodes starts them up:

func StartNodes(c Config) {
    for _, ep := range c.NodeEndpoints() {
        node := clienthandler.NewListenerNode(ep)
        go node.Start()
    }
}

HARNESS_MULTIPLE_NODES:internal/harness/api.go

The clients are more complicated, because I need to separate the creation of the "client" per se from the other that I just hacked in, which has to do with the creation of messages, not the submission process.

My first step, then, is going to be to continue hacking in the other for now, and continue with just two users. But the other will be part of the configuration.

There are a number of ways of organizing this data, but in my head the easiest thing is to group the clients I want to configure by the nodes that will process them, and store that in a map. Doing this gives us this:

type Config interface {
    NodeEndpoints() []string
    ClientsPerNode() map[string][]CliConfig
}

type HarnessConfig struct {
    nodeEndpoints []string
    clients       map[string][]CliConfig
}

// NodeEndpoints implements Config.
func (c *HarnessConfig) NodeEndpoints() []string {
    return c.nodeEndpoints
}

// ClientsPerNode implements Config.
func (c *HarnessConfig) ClientsPerNode() map[string][]CliConfig {
    return c.clients
}

type CliConfig struct {
    client string
    other  string
}

func ReadConfig() Config {
    return &HarnessConfig{nodeEndpoints: []string{":5001", ":5002"}, clients: map[string][]CliConfig{
        "http://localhost:5001": {
            CliConfig{client: "https://user1.com/", other: "https://user2.com/"},
            CliConfig{client: "https://user2.com/", other: "https://user1.com/"},
        },
        "http://localhost:5002": {
            CliConfig{client: "https://user1.com/", other: "https://user2.com/"},
            CliConfig{client: "https://user2.com/", other: "https://user1.com/"},
        },
    }}
}

HARNESS_MULTIPLE_CLIENTS:internal/harness/api.go

The CliConfig type may seem overkill at this point, but it is simpler and less hacky than the alternatives, and we are going to end up putting a lot more information on there later, so, yes, I'm just planning ahead.

Thus PrepareClients becomes:

func PrepareClients(c Config) []Client {
    repo, err := client.MakeMemoryRepo()
    if err != nil {
        panic(err)
    }
    ret := make([]Client, 0)
    m := c.ClientsPerNode()
    for node, clis := range m {
        for _, cli := range clis {
            if s, err := repo.SubmitterFor(node, cli.client); err != nil {
                panic(err)
            } else {
                url, _ := url.Parse(cli.other)
                ret = append(ret, &ConfigClient{submitter: s, other: url, done: make(chan struct{})})
            }
        }
    }

    for _, s := range ret {
        s.PingNode()
    }
    return ret
}

HARNESS_MULTIPLE_CLIENTS:internal/harness/api.go

Running this submits four messages, which ends up with two transactions being resolved. Interestingly, this only takes 14ms.

Configuring MemoryClientRepo

Way back when, I introduced MemoryClientRepo and said it was OK that I was just hacking values in, because one day I would come back and configure it. For the harness at least, that day has come.

First, I need to move the current initialization out of MemoryClientRepo:

func MakeMemoryRepo() (MemoryClientRepository, error) {
mcr := MemoryClientRepository{clients: make(map[url.URL]*ClientInfo)}
return mcr, nil
}

HARNESS_CONFIG_REPO:internal/client/repo.go

and into ledgerclient:

func main() {
    repo, e1 := client.MakeMemoryRepo()
    if e1 != nil {
        panic(e1)
    }
    repo.NewUser("https://user1.com/")
    repo.NewUser("https://user2.com/")

HARNESS_CONFIG_REPO:cmd/ledgerclient/main.go

Then in PrepareClients we need to look at all the client users we are configuring and ensure they are added to the repository (but only if they don't exist):

func PrepareClients(c Config) []Client {
    repo, err := client.MakeMemoryRepo()
    if err != nil {
        panic(err)
    }
    m := c.ClientsPerNode()
    for _, clis := range m {
        for _, cli := range clis {
            if repo.HasUser(cli.client) {
                continue
            } else if err := repo.NewUser(cli.client); err != nil {
                panic(err)
            }
        }
    }
    ret := make([]Client, 0)

HARNESS_CONFIG_REPO:internal/harness/api.go

The function HasUser is new, but has the obvious definition in repo.go; if you're interested, you can go and look at the source.

That deals with all of the creation of the clients, but we still need to generate multiple messages for each client.

Message Generation

This is a lot more tricky, for a number of reasons. First, there is just the random generation element. Then there is the issue of choosing who your counter-signatories are. We need to configure how many messages are going to be generated and put that in a loop. And then there is the trickiest element, which is to make sure all your counter-signatories do in fact sign your messages.

So, let's start by extracting the content generation into a separate function.

First, we need to import a random number library. This comes from the package math/rand/v2 which gives it the package name v2, which is not ideal. We can rename it with this syntax:

import (
    "crypto/sha512"
    "log"
    "time"

    rno "math/rand/v2"

    "github.com/gmmapowell/ChainLedger/internal/api"
    "github.com/gmmapowell/ChainLedger/internal/client"
    "github.com/gmmapowell/ChainLedger/internal/clienthandler"
)

HARNESS_GEN_MESSAGES:internal/harness/api.go

We simplify the anonymous submitting function inside Begin by moving the message creation out:

func (cli *ConfigClient) Begin() {
    go func() {
        tx, err := makeMessage(cli)
        if err != nil {
            log.Fatal(err)
            return
        }
        err = cli.submitter.Submit(tx)
        if err != nil {
            log.Fatal(err)
            return
        }
        cli.done <- struct{}{}
    }()
}

HARNESS_GEN_MESSAGES:internal/harness/api.go

and then provide makeMessage:

func makeMessage(cli *ConfigClient) (*api.Transaction, error) {
    content := "http://tx.info/" + randomPath()
    hasher := sha512.New()
    hasher.Write(randomBytes(16))
    h := hasher.Sum(nil)

    tx, err := api.NewTransaction(content, h)
    if err != nil {
        return nil, err
    }
    for _, s := range cli.repo.OtherThan(cli.user) {
        err = tx.Signer(&s)
        if err != nil {
            return nil, err
        }
    }

    return tx, nil
}

HARNESS_GEN_MESSAGES:internal/harness/api.go

which depends on a couple of trivial functions to create random things:

func randomPath() string {
    ns := 6 + rno.IntN(6)
    ret := make([]rune, ns)
    for i := 0; i < ns; i++ {
        ret[i] = alnumRune()
    }
    return string(ret)
}

func alnumRune() rune {
    r := rno.IntN(38)
    switch {
    case r == 0:
        return '-'
    case r == 1:
        return '.'
    case r >= 2 && r < 12:
        return rune('0' + r - 2)
    case r >= 12:
        return rune('a' + r - 12)
    }
    panic("this should be in the range 0-38")
}

func randomBytes(ns int) []byte {
    ret := make([]byte, ns)
    for i := 0; i < ns; i++ {
        ret[i] = byte(rno.IntN(256))
    }
    return ret
}

HARNESS_GEN_MESSAGES:internal/harness/api.go

and also on a method on the ClientRepository to find users who are not the current submitter, which is dull and requires quite a bit of rework so I've not shown it here. Again, you can check out the source if you're really interested.

Putting it in a Loop

Each client wants to publish multiple messages. This is quite easy. We simply configure the count for each client

type CliConfig struct {
    client string
    count  int
}

func ReadConfig() Config {
    return &HarnessConfig{nodeEndpoints: []string{":5001", ":5002"}, clients: map[string][]CliConfig{
        "http://localhost:5001": {
            CliConfig{client: "https://user1.com/", count: 10},
            CliConfig{client: "https://user2.com/", count: 2},
        },
        "http://localhost:5002": {
            CliConfig{client: "https://user1.com/", count: 5},
            CliConfig{client: "https://user2.com/", count: 7},
        },
    }}
}

HARNESS_PUBLISH_MANY:internal/harness/api.go

and store it in the configured client:

type ConfigClient struct {
    repo      client.ClientRepository
    submitter *client.Submitter
    user      string
    count     int
    done      chan struct{}
}

HARNESS_PUBLISH_MANY:internal/harness/api.go

copying it across in PrepareClients:

    for node, clis := range m {
        for _, cli := range clis {
            if s, err := repo.SubmitterFor(node, cli.client); err != nil {
                panic(err)
            } else {
                ret = append(ret, &ConfigClient{repo: &repo, submitter: s, user: cli.client, count: cli.count, done: make(chan struct{})})
            }
        }
    }

HARNESS_PUBLISH_MANY:internal/harness/api.go

And then put makeMessage and Submit in a loop, being very careful to keep the done notification at the end:

func (cli *ConfigClient) Begin() {
    go func() {
        for i := 0; i < cli.count; i++ {
            tx, err := makeMessage(cli)
            if err != nil {
                log.Fatal(err)
                return
            }
            err = cli.submitter.Submit(tx)
            if err != nil {
                log.Fatal(err)
                return
            }
        }
        cli.done <- struct{}{}
    }()
}

HARNESS_PUBLISH_MANY:internal/harness/api.go

Now we are publishing a good number of messages, but unfortunately because they are all random, we are never resolving any of them.

Cooperating with Counter-signatories

What we need to do, of course, is to get the other counter-signatories that we have randomly allocated to each message to also submit a copy of the same message that we have just created.

And this requires us to have more goroutines and more channels. And, sadly, even then it isn't all that simple.

The problem really comes down to termination: when do we stop? If we are going to add a goroutine which listens for other peoples' messages and then signs and submits them, how do we know to end this goroutine? The simple answer is "when they tell us", but how can they do that?

Go provides a close method which indicates when the sender has finished sending messages down the channel. We can use that, but only if there is only one sender, otherwise the first sender to finish will close the channel, ruining the party for everyone.

All my instincts scream "you can't possibly solve this by having one channel between each pair of clients, and then one goroutine reading each channel", but maybe you can. I'm told that both channels and goroutines are cheap - much cheaper than real threads or processes. So let's start there, and with relatively simple code, and then let's worry about the consequences if and when they come to bite us.

So, what we're going to do is, in Begin, for each client goroutine that we create, we are going to create a channel and a goroutine for each other registered user. Each of those goroutines will be responsible for listening on its corresponding channel, and submitting its version of the transaction when it receives a message. Meanwhile, we will create a map of user id to channel and in the "main" goroutine we will send messages to each of the channels associated with the other signers.

We are not going to send Transaction objects over the channels, but rather the ingredients needed to assemble one. This ensures we don't accidentally share any state (such as signatures) between the two goroutines.

Finally, we need to update our logic around when the done message is sent, because we need to wait for all the other channels to have closed before we can honestly say we're done. We will use a WaitGroup for that.

And, yes, I believe this is the "simple" solution. I have considered a number of other more complicated solutions and rejected them all. Take a deep breath, you may need it.

I opted to do this in two phases. The first phases puts the channels in place, adds the readers, closes the channels and makes sure the program still terminates. The second phase actually deals with the necessary changes to make the messages flow and be resolved.

Each configured client now needs two collections of channels: one ("writer channels") is a map by user name of a channel to send a request to cosign; the other ("reader channels") is a slice of channels that it is going to read from until they have all been closed.

type ConfigClient struct {
    repo      client.ClientRepository
    submitter *client.Submitter
    user      string
    count     int
    cosigners map[string]chan<- PleaseSign
    signFor   []<-chan PleaseSign
    done      chan struct{}
}