I am going to rattle through a very simple first version of the
ledgerclient and
chainledger node. The purpose of this is to get to a point where we have a client that can create and sign a message and upload it to a server. It may at first seem we have achieved a lot here; we really haven't.
The Client
I'm going to start with the client, in large part because it's a nice simple thing. I'm going to present the client files one at a time, in their entirety, with commentary among the various parts.
cmd/ledgerclient/main.go
The first section of the file is mainly boilerplate.
package main
import (
"fmt"
"hash/maphash"
"log"
"net/url"
"github.com/gmmapowell/ChainLedger/internal/api"
"github.com/gmmapowell/ChainLedger/internal/client"
)
The
package statement identifies the package that this file claims to be in. In order to become a command executable, it must be declared as being in the package
main; if you don't do this the function
main() will be flagged as not being used anywhere.
The
import statement lists all the other packages which are going to be used in this file. If a package is going to be used, it must be listed and, conversely, if it is listed it must be used.
The
fmt package is where Go places things like
Printf, which will be using to confirm that we have submitted a transaction. We will be using the
hash/maphash package to generate an acceptable hash to submit as the (alleged) hash of the contents of the document we are submitting.
log contains all the statements to do logging and
net/url contains the definition of the
URL struct which we will be using everywhere to validate our strings are valid URLs.
The main module delegates most of its work to two
internal packages (i.e. packages elsewhere in this same project),
api and
client. We will cover those when we get there.
func main() {
repo, e1 := client.MakeMemoryRepo()
if e1 != nil {
panic(e1)
}
uid := "https://user1.com/"
uu, e2 := url.Parse(uid)
if e2 != nil {
panic(e2)
}
pk, e3 := repo.PrivateKey(uu)
if e3 != nil {
panic(e3)
}
As is the case with most C-based languages,
go defines a function called
main which is the entry point for the entire program. These opening lines show the basic setup of the client. We create an internal repository for things to do with users and nodes (the client and node respositories will be similar, but will be different). For now, we are just using this to hide the details of creating and managing the initial user's private key, but in the log run will store more and more information in there. For now, we are not using any external sources of information (e.g. command line arguments) but we will come back and do that later.
One such thing that should be an argument is the submitting user id, which is hardcoded to be
https://user1.com/. I am using URLs as IDs for the simple reason that they have some structure and it makes it easy to have user ids with given relationships. In the long run, I expect that I will want this to be "real" URLs that return a JSON document that describes the user's key information, such as the public key for their signing key.
And then we ask the repository for the private key for the current user. Note that the repository is not going to have ALL the private keys for ALL the users; any given instantiation of the client repository will probably only have ONE private key (for the given user), although it is not going to be a hardwired constraint (mainly because I want to violate it in order to build a stress test). Ultimately, I expect the repository to read from files, databases or something like Amazon SSM parameters, depending on the deployment environment. In that case, you have access to whatever would be available.
Here in this code it is possible to see how much of the code is dominated by error handling in the absence of exception handling. I am optimistic that in the fullness of time I will become better at dealing with this and it will look less ugly.
One of the things that this does surface, however, is just how many things can go wrong, and it does (to a certain extent) force you to think about them. I'm not really sure what to do with any of these errors - for now, at least, nothing should go wrong here, because I am not depending on either external data or external services. I used the
panic command because I saw it and thought it probably did what I wanted here: made it clear there was a problem.
cli, err := client.NewSubmitter("http://localhost:5001", uid, pk)
if err != nil {
log.Fatal(err)
return
}
In order to connect to the server, we need a "transaction submitter". This is a module which takes the URL of a node to connect to, a the URL id of a user who is going to be submitting the transactions, and their private key. It is then ready to submit all of the transactions on their behalf.
If anything goes wrong, the error is logged and the main function returns. This is subtly different handling from the above, which is because there really are a number of things that can go wrong here, most notably that the server isn't running.
var h maphash.Hash
h.WriteString("hello, world")
tx, err := api.NewTransaction("http://tx.info/msg1", &h)
if err != nil {
log.Fatal(err)
return
}
err = tx.SignerId("https://user2.com")
if err != nil {
log.Fatal(err)
return
}
err = cli.Submit(tx)
if err != nil {
log.Fatal(err)
return
}
The rest of the code comes here in a rush, in which we create a new transaction message which is defined by the URL
http://tx.info/msg1 and for which we provide the hash of the alleged contents "hello, world". As I said in the design, it is not assumed that either the client or the server can read the URL, and indeed it may not exist, so the URL and the hash must both be provided.
Before the message can be hashed and signed, the transaction must be given a list of
all the users who will sign it. Now, obviously, it is not possible for any one user to sign for all the parties, so the transaction will only have one signature - for the submitting party - and, given that the current party
is submitting it, the submitting user does not need to explicitly provide their user id.
Finally, by calling
Submit, on the
cli reference (the submitter), the transaction is signed and uploaded to the server.
All of these methods can reasonably fail, but even so, all I'm doing is logging the errors. In the fullness of time, I probably want to revisit at least the most common errors.
fmt.Printf("submitted transaction: %v", tx)
}
Finally, if all went well, we report to the user that the transaction was (successfully) submitted.
internal/client/submit.go
The usual boilerplate starts us off:
package client
import (
"crypto/rsa"
"net/http"
"net/url"
"github.com/gmmapowell/ChainLedger/internal/api"
)
Now, I'm new to Go, so I'm going to cause offence and say we are going to declare a
class. It's not a class, it's a struct, and you are (I am) going to hobble yourself if you thing that classes and
structs are the same thing (in either direction). But anyway, we are going to declare a class to submit transactions to the server.
type Submitter struct {
node *url.URL
iam *url.URL
pk *rsa.PrivateKey
}
This declares a
struct with three fields. Because the field names start with lowercase letters, they are all
private fields, which means that they can be seen in the methods in this file, but not elsewhere (I think, I'm not entirely sure what the rules are).
The three fields are:
- node is the URL of the server we are intending to connect to;
- iam is the URL which identifies the submitting user;
- pk is the private signing key of the submitting user.
func NewSubmitter(node string, id string, pk *rsa.PrivateKey) (*Submitter, error) {
nodeAddr, e1 := url.Parse(node)
if e1 != nil {
return nil, e1
}
iam, err := url.Parse(id)
if err != nil {
return nil, err
}
return &Submitter{node: nodeAddr, iam: iam, pk: pk}, nil
}
Go does not have constructors as such. Instead a new instance of a struct is created using the syntax shown on the last line of this function: the
struct name, followed by field assignments in curly braces. The equivalent to constructors are the kind of functions shown here, which take arguments and then create the appropriate object. So here, we have a function that takes
string versions of
node and user
id, and confirms that they are in fact valid urls before placing them in a
Submitter structure.
Note that in order to return a
*Submitter, we have to use the
& operator to convert an instance into a pointer (or some such technical language).
func (s *Submitter) Submit(tx *api.Transaction) error {
var e error = tx.Signer(s.iam)
if e != nil {
return e
}
e = tx.Sign(s.iam, s.pk)
if e != nil {
return e
}
json, e2 := tx.JsonReader()
if e2 != nil {
return e2
}
cli := http.Client{}
_, e3 := cli.Post(s.node.JoinPath("/store").String(), "application/json", json)
return e3
}
This is how methods are defined and attached to a
struct. The parentheses after the
func keyword indicate that there is a "target object" (i.e. the instance) which is implicit in the function call, but other than that has an argument name just like any other (there is no built-in
this variable, although there would be nothing stopping you calling the variable here
this or
self everywhere if you wanted to).
This first makes sure that the submitting user is listed as a
Signer, and then uses their private key to
Sign the key fields in this block. Note that it is the responsibility of the
Transaction code to make sure that all the users sign
exactly the same block, although this first version of the code doesn't do that: we'll come back to that in a bit.
Finally, the code submits a JSON version of the
Transaction struct to the server. Once again, it feels like the error handling stops me writing the fluid code I would want to; it is not clear to me how to put the
tx.JsonReader() in the call to
cli.Post because I cannot handle the error return.
internal/api/transaction.go
Once again, the boilerplate for completeness so that you can't say I don't show you everything.
package api
import (
"bytes"
"crypto"
"crypto/rand"
"crypto/rsa"
"crypto/sha512"
"encoding/json"
"fmt"
"hash"
"io"
"net/url"
"github.com/gmmapowell/ChainLedger/internal/types"
)
The purpose of the
Transaction struct is to manage the lifecycle of a transaction on the client side for submission to a server. It will also be stored on the server side until the server has one copy from each of the signatories. We'll get to that later. Quite a bit later.
type Transaction struct {
ContentLink *url.URL
ContentHash hash.Hash
Signatories []*types.Signatory
}
A transaction consists of three things: the link to the content,
ContentLink, the hash of said content
ContentHash and the signature block
Signatories.
For now, I have made all of these public fields, but I think that just reflects the fact that I am still at the experimental phase: do not be surprised if they are made private at some point in the future.
func NewTransaction(linkStr string, h hash.Hash) (*Transaction, error) {
link, err := url.Parse(linkStr)
if err != nil {
return nil, err
}
return &Transaction{ContentLink: link, ContentHash: h, Signatories: make([]*types.Signatory, 0)}, nil
}
Again, we have here a "constructor pattern" where
NewTransaction is responsible for taking the expected arguments, validating them, and returning a pointer to the
Transaction block.
Note the use here of
make to create a 0-length array of
Signatory items. We will add the signatories individually below, so for now we don't have any.
func (tx *Transaction) SignerId(signerId string) error {
signer, err := types.OtherSignerId(signerId)
return tx.addSigner(signer, err)
}
func (tx *Transaction) Signer(signerURL *url.URL) error {
signer, err := types.OtherSignerURL(signerURL)
return tx.addSigner(signer, err)
}
func (tx *Transaction) addSigner(signer *types.Signatory, err error) error {
if err != nil {
return err
}
tx.Signatories = append(tx.Signatories, signer)
return nil
}
Each of the signers for the transaction needs to be identified. During submission, the
Signer method is called with the submitter's URL. All the others need to be explicited added, as happens in
main.
The code here has two possible paths, with either with a string or a URL being passed in. The appropriate method in the
types module (see below) is called to obtain a
Signatory struct, and then the common path in
addSigner (its name begins with a lower case letter, because it's an internal method) is called to add the
Signatory to the
Transaction.
The
append function extends the array slice to make room for an additional element at the end, and then puts this new
Signatory there.
func (tx *Transaction) Sign(signerURL *url.URL, pk *rsa.PrivateKey) error {
return tx.doSign(signerURL, pk, nil)
}
func (tx *Transaction) doSign(signer *url.URL, pk *rsa.PrivateKey, e1 error) error {
if e1 != nil {
return e1
}
h, e2 := tx.makeSignableHash()
if e2 != nil {
return e2
}
sign, e3 := makeSignature(pk, h)
if e3 != nil {
return e3
}
done := false
for _, signatory := range tx.Signatories {
if signatory.Signer == signer {
signatory.Signature = sign
done = true
break
}
}
if !done {
return fmt.Errorf("there is no signatory %v", signer)
}
return nil
}
func (tx *Transaction) makeSignableHash() (hash.Hash, error) {
var h = sha512.New()
h.Write([]byte("hello, world"))
return h, nil
}
func makeSignature(pk *rsa.PrivateKey, h hash.Hash) (*types.Signature, error) {
sum := h.Sum(nil)
sig, err := rsa.SignPSS(rand.Reader, pk, crypto.SHA512, sum, nil)
if err != nil {
return nil, err
}
var ret types.Signature = sig
return &ret, nil
}
This is responsible for doing all of the work to create a signature - and even then it doesn't sign the right thing!
The first method follows the same pattern as the block above in order to allow for a
Sign method with just a string, but that method does not exist.
The second method (
doSign), first calls
makeSignableHash to figure out what to sign, then calls
makeSignature to sign that hash. Finally, it scans through all the signatory blocks and finds the one whose signatory is the submitting user. It is, of course, an error for there not to be one, and the
fmt.Errorf is a special version of
Printf which creates an error object.
The
makeSignableHash object is supposed to lay out and organise the contents of the
Transaction object into a consistent block, regardless of how it was constructed, and then produce a SHA-512 hash of it. At the moment, it does not do this, but rather hashes "hello, world". We'll come back to that later.
Finally,
makeSignature takes a private key and the hash, and generates the
Sum of the hash and calls
SignPSS to sign it. This is all very easy code to write, although I'm not 100% convinced I actually understand what I've done. At some point we will write code on the server side to verify the signatures. If it turns out that we encounter some problems, we will come back and revisit it.
Signature is my own internal type (we'll see it below) and it is for all intents and purposes "the same as"
[]byte, which is what we get back from
SignPSS. However, I cannot directly return the address of
sig but have to first pass it to an explicitly typed variable
ret before I can take the address. I am sure that there are good reasons for this (that probably involve the word 'contravariance') but I do not know what they are yet.
func (tx *Transaction) JsonReader() (io.Reader, error) {
json, err := json.Marshal(tx)
if err != nil {
return nil, err
}
return bytes.NewReader(json), nil
}
func (tx *Transaction) String() string {
return fmt.Sprintf("Tx[%s]", tx.ContentLink)
}
Finally we have a couple of methods to return a
Transaction in a readable format. The
JsonReader function simply marshals the transaction to JSON, which is
very nicely handled in Go.
Meanwhile the
String function is just like a
toString method in Java or JavaScript: for any given object, it will convert the contents to a string on demand. This behaviour is defined by the
Stringer interface, which
Transaction implicitly implements by providing the
String method. Clever.
internal/client/repo.go
The client repository is a placeholder for a configurable information store which is intended to hold all of the information needed by the client. For now, we are just using it as a place to hide the public/private key for a user.
We start with the boilerplate:
package client
import (
"crypto/rand"
"crypto/rsa"
"fmt"
"net/url"
)
Because we will want to have many different ways of collecting the data, we are going to define
ClientRepository as an
interface:
type ClientRepository interface {
PrivateKey(user *url.URL) (*rsa.PrivateKey, error)
}
For now, this interface is very simple, just allowing us to ask for the private key of a given user URL.
Internally, we are going to store records about all the users in the system, which are in
ClientInfo structs. As I write this, I wonder if the name might want to change to
UserInfo. Don't be surprised if it does at some point.
type ClientInfo struct {
user *url.URL
privateKey *rsa.PrivateKey
publicKey *rsa.PublicKey
}
Each user has three fields associated with them for now - their unique URL id, their private key and their public key. I am storing all three of these, even though for now, we will only use the private key.
type MemoryClientRepository struct {
clients map[url.URL]*ClientInfo
}
func MakeMemoryRepo() (ClientRepository, error) {
mcr := MemoryClientRepository{clients: make(map[url.URL]*ClientInfo)}
mcr.NewUser("https://user1.com/")
return mcr, nil
}
MemoryClientRepository is a concrete implementation of
ClientRepository and is one that is intended for "toy" use: testing or demonstration purposes. We will come back and build better things as we need them. This basically stores in memory a map of URLs to
ClientInfo records. The
MakeMemoryRepo function is a constructor of sorts, but it also initializes the repository with our chosen user.
Note that the key of the
map here is a
url.URL not a
*url.URL. That was my original implementation, but it did not work. Why not? Because the keys in a map have to match by equality, and if you use pointers, they don't. I'm not entirely sure how equality works in Go (yet), but it is reasonable that you say that two pointers are equal if they are the same pointer, but that two URLs are the same if they have the same "value", i.e. represent the same address. That's what seems to happen here.
func (cr MemoryClientRepository) PrivateKey(user *url.URL) (pk *rsa.PrivateKey, e error) {
entry := cr.clients[*user]
if entry == nil {
e = fmt.Errorf("there is no user %s", user.String())
} else {
pk = entry.privateKey
}
return
}
This method recovers the private key for a given user by accessing the map for the URL passed in. Again, note that we are using
*user as the key, since we are given a pointer. This still seems weird in my head, but I'm sure I'll wrap my head around it in the end.
If we don't find an entry in the map, we will return an error; otherwise we extract the private key. This assumes that there is a private key associated with the user; we should probably (and probably will at some point) check if the user
has a private key and return an error if they do not. Of course, we could say that returning the pair
nil, nil means that we did find the user, but they didn't have a private key - test for that case! It depends on how we want to code things.
func (cr *MemoryClientRepository) NewUser(user string) error {
u, e1 := url.Parse(user)
if e1 != nil {
return e1
}
if cr.clients[*u] != nil {
return fmt.Errorf("user %s already exists in the repo", user)
}
pk, e2 := rsa.GenerateKey(rand.Reader, 2048)
if e2 != nil {
return e2
}
cr.clients[*u] = &ClientInfo{user: u, privateKey: pk, publicKey: &pk.PublicKey}
return nil
}
The final method is the one that adds a new user to the repository. Since it takes a string url, it must first parse that (and returns an error if it does not parse correctly). We check that this is not a duplicate user, because that is kind of the definition of a "new" user. Then we generate a key pair and store the new triple of user, public key and private key in the list of clients.
internal/types/signatory.go
The signatory type is just a
struct combining the URL id of a signer and their (optional) signature.
We start with the usual boilerplate.
package types
import (
"net/url"
)
The actual signatory struct is very simple.
type Signatory struct {
Signer *url.URL
Signature *Signature
}
func OtherSignerURL(u *url.URL) (*Signatory, error) {
return &Signatory{Signer: u}, nil
}
func OtherSignerId(id string) (*Signatory, error) {
u, err := url.Parse(id)
if err != nil {
return nil, err
}
return OtherSignerURL(u)
}
And then there are two constructor methods, one that takes a URL signer id and one that parses it from a string. In both cases, the
Signature is initally left blank.
internal/types/signature.go
This final client file is not that complicated. It basically amounts to a type alias.
package types
type Signature []byte
This just says that I can write
types.Signature anywhere and it is basically the same as writing
[]byte, although, as we saw when we used it, that is not true when using the
& operator.
The Server
Moving on from the client side, we are going to have a server which initially doesn't do very much. It starts up and makes itself available for clients to submit transactions. It reports that it has done so.
cmd/chainledger/main.go
As with the client, we put the
main() function in a file called
main.go in the directory
cmd/chainledger which causes a binary
chainledger to be produced with
go build.
It starts with the usual boilerplate:
package main
import (
"errors"
"fmt"
"log"
"net/http"
"github.com/gmmapowell/ChainLedger/internal/clienthandler"
)
and then has the
main() function which depends heavily on code included from
internal:
func main() {
log.Println("starting chainledger")
storeRecord := clienthandler.NewRecordStorage()
cliapi := http.NewServeMux()
cliapi.Handle("/store", storeRecord)
err := http.ListenAndServe(":5001", cliapi)
if err != nil && !errors.Is(err, http.ErrServerClosed) {
fmt.Printf("error starting server: %s\n", err)
}
}
Because this is a server, I will be very inclined to
log everything that happens. So, we start by acknowledging that we have started.
The
NewRecordStorage class is the class which will be responsible for handling the receipt and processing of a transaction coming from the client. I'm not entirely sold on the name at this point, but the whole thing is really just a placeholder at the moment, so I'll probably change it when I feel the need.
The rest of the code just sets up a web server.
cliapi is a
http.ServeMux, which I think is what you need in order to have different services listen on different ports (but I'm not sure). There is something to be said for the fact that this is a case of "YAGNI", but since I believe I am more likely to need it (and forget that it exists and waste time trying to figure out what's going wrong) that I am to not need it, I have put it there.
This
ServeMux then takes the
Handle method to associate a web path with a handler (which must implement the
ServerHTTP method in order to be an
http.Handler); this is what the
RecordStorage class does.
The
ListenAndServe method opens up a socket to listen on the named port and dispatches any incoming requests to the identified muxer. It's important to note that
this method blocks dispatching the incoming messages, so it will be important to call this method in a
goroutine if we want to have multiple running concurrently.
The final couple of lines deal with error cases, the most likely of which is that we already have the server running on the identified port.
internal/clienthandler/recordstorage.go
This is the handler for the
/store web path.
The usual boilerplate gets us going:
package clienthandler
import (
"log"
"net/http"
)
We need to declare a
struct and have a constructor for it.
type RecordStorage struct {
}
func NewRecordStorage() RecordStorage {
return RecordStorage{}
}
This is all very simple, mainly because it is a dummy implementation at the moment. We will come back to it next time when we start doing some serious implementation.
// ServeHTTP implements http.Handler.
func (r RecordStorage) ServeHTTP(http.ResponseWriter, *http.Request) {
log.Println("asked to store record")
}
The one thing I did manage to get the VSCode plugin to "Quick Fix" for me was declaring
RecordStorage and then using it where an
http.Handler was needed. It told me it didn't have this method and implemented it for me (including adding the comment). I then added the one line "implementation" which is basically logging the fact that we have received a request to store a record, the said request being a JSON object in the body of the request. Again, we will come back to this next time.
internal/records/storedtransaction.go
Start with the boilerplate:
package records
import (
"hash"
"net/url"
"github.com/gmmapowell/ChainLedger/internal/types"
)
At the moment, this is a bit speculative, because it's not used anywhere, but it feels so key to me in terms of everything we are trying to accomplish that I couldn't imagine
not defining it early. Apologies to all the YAGNI folks, but I am going to need it, and I want it there early. Having said that, I believe more fields will ultimately be added to this (in particular the node will sign it), so I'm certainly not going all waterfall on you.
type StoredTransaction struct {
txid hash.Hash
whenReceived types.Timestamp
contentLink url.URL
contentHash hash.Hash
signatories []types.Signatory
}
This is the version of the transaction that we are going to log on the permanent record. In order to make this happen, we need to collect all the copies that are sent across from the clients and match them together. The last three fields obviously match the fields that are in the
Transaction object that will be coming over from the client in the body of the request; the other two will be attached when we get around to processing it.
internal/types/timestamp.go
Again, this is a very simple type alias at the moment, although I will probably add more functionality to it later.
The purpose of this is to allow me to store a timestamp using the semantics of JavaScript (milliseconds centred around midnight at the start of Jan 1, 1970 GMT) in a single
int64.
package types
type Timestamp int64
Checked In
All of this code is checked in to git at
git@github.com:gmmapowell/ChainLedger.git and this version of the code is tagged
FIRST_CLIENT_SERVER.
