So far, we have written a minimal server and an inadequate client. But for sure, the client is a long way ahead of the server and we'll only be making changes there that we need in order to make progress on the server side. So now, we're going to slow down (a bit) and tackle the basic server functionality. Remember, what I still really want to do is get into the nitty-gritty of testing that this can operate at scale and keep functioning in the face of multiple network problems. Apologies if I'm still going faster than you want - get back to me in the comments and I'll try and address your concerns.
At the end of the last episode, we claimed to have a working client and server, but in reality we have put a JSON transaction onto the wire and alert the server. Before we can go any further we need to unpack it on the server side. I was going to "just do this", but it turns out to be trickier than I thought.
The handler code
We left our handler last time with this code:// ServeHTTP implements http.Handler.
func (r RecordStorage) ServeHTTP(http.ResponseWriter, *http.Request) {
log.Println("asked to store record")
}
FIRST_CLIENT_SERVER:internal/clienthandler/recordstorage.go
The http.Request is some abstract thing which, given the context, we assume is a POST of a transaction as a JSON message. We need parse the JSON into a "client" transaction.(We should (arguably?) also check things like whether it is a POST request, and we should also probably have some kind of session ID - if I don't find I need them before the end of this blog, adding them will be left as an exercise for the reader. This is true of a lot of validation steps at the moment, but I imagine I will end up checking all the signatures I care about.)
On the face of it, it would seem we can read the body of the request using io.ReadAll and convert it from JSON into a Transaction object using json.Unmarshal:
// ServeHTTP implements http.Handler.
func (r RecordStorage) ServeHTTP(resp http.ResponseWriter, req *http.Request) {
log.Printf("asked to store record with length %d\n", req.ContentLength)
body, err := io.ReadAll(req.Body)
if err != nil {
log.Printf("Error: %v\n", err)
resp.WriteHeader(http.StatusBadRequest)
return
}
log.Printf("have json input %s\n", string(body))
var tx = api.Transaction{}
err = json.Unmarshal(body, &tx)
if err != nil {
log.Printf("Error unmarshalling: %v\n", err)
resp.WriteHeader(http.StatusBadRequest)
return
}
log.Printf("Have transaction %v\n", tx)
}
UNPACK_JSON_1:internal/clienthandler/recordstorage.go
Sadly this does not work. There are two problems: let's tackle the trivial one first. I assumed that when it came to sending a URL over the wire, the marshaller would automatically transform it back into a string, because that's what we all think URLs are. But not so much - this is what we see when we dump the body:{"ContentLink":{"Scheme":"http","Opaque":"","User":null,Now, the json.Marshal function can adapt its behaviour based on tags associated with a struct field, but as far as I can see, there isn't one that says "use the Stringer interface" (there is a ",string", but that says it only works for primitive fields).
"Host":"tx.info","Path":"/msg1","RawPath":"","OmitHost":false,
"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},
"ContentHash":{},
"Signatories":[{"Signer":{"Scheme":"https","Opaque":"","User":null,
"Host":"user2.com","Path":"","RawPath":"","OmitHost":false,
"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},
"Signature":null},
{"Signer":{"Scheme":"https","Opaque":"","User":null,
"Host":"user1.com","Path":"/","RawPath":"","OmitHost":false,
"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},
"Signature":"oxjfkiItXMa/jME0zSuVwlTjlyAd3ITYxZ/JslmT5o4Aj+cwiSN7tQX7OKBAE+tX4DZ9Qe+yEclwIKJUeCMvjXdQ9zPDBktkC0njdkybjaUbiNXrhLSCWLOz2861lzuNQd92GAoMErwy3bBI9qthhoBG47gMTC/bEzesH33WL+ZenDYQhsyrk0DXlT/BokghGVI9H1XTSpfespOFxOEH7SZMAw1HSBtNFkF5VsE66Je66suirLk3Pb2+ClTjOs4NXwlTGv11a1BZWZA9lKqs4M3rNmzLWJiz8FmGL4XTNpyAS/XNL1o9UOLxK+OnckU+pd4md8fTRTLK0su16I6rwA=="
}]}
So, it looks like I need to write my own marshalling (and unmarshalling) code for Transaction and Signatory. Fortunately, it's not too hard. It would seem that you can copy the fields into a map and then marshal that:
func (tx Transaction) MarshalJSON() ([]byte, error) {
var m = make(map[string]any)
m["ContentLink"] = tx.ContentLink.String()
m["ContentHash"] = tx.ContentHash
m["Signatories"] = tx.Signatories
return json.Marshal(m)
}
CUSTOM_MARSHAL_URL:internal/api/transaction.go
Likewise for Signatory:func (sig Signatory) MarshalJSON() ([]byte, error) {
var m = make(map[string]any)
m["Signer"] = sig.Signer.String()
m["Signature"] = sig.Signature
return json.Marshal(m)
}
CUSTOM_MARSHAL_URL:internal/types/signatory.go
Of course, that now means that you have one more moving part you need to keep synchronized every time you add a field :-(Now we see the following output from the server when it unpacks the message:
{"ContentHash":{},The bigger problem is that Unmarshalling just fails:
"ContentLink":"http://tx.info/msg1",
"Signatories":[
{"Signature":null,"Signer":"https://user2.com"},
{"Signature":"E6x5u3lOi6PwiXuDUCPMd4sv87LVxZVCng50MTe/dtIG9e8HuZFS4Z1K11t/VoKR1PtWPyNMLtIQnsd+dYRhnPFOA1HUErH0Xnd2rwS/tX4DHrISYLj9ioyuD4f0kJyGTEZIORm9nnL01vYBWukwZ+2Ghbfvy65ElzCjmhyexCtwGMOCpL3ao3z0WmRY+RHn4XRTZrMyLX0NkX96Mcpz75WRlu+uwxnbUvJnfpeXJYFT5XA1K/8r/iEhq71EGEg4mKQgJcrwVid9rZDeKu1s8HU1hpJ2zDQQGcxpdFdxbgcGPby0vQ/J1VzmEX0OTpqrgL0WF9JnuVBysVx7mrwN0w==","Signer":"https://user1.com/"}
]}
Error unmarshalling: json: cannot unmarshal object into Go struct field Transaction.ContentHash of type hash.HashLooking back at the JSON we printed out, that would seem to be most likely connected to the fact that ContentHash is represented as {}. In all likelihood, we need to do something with that as well in order to marshal it correctly.
But, no. It turns out that I have misunderstood the term Hash as used in Go. A hash.Hash is a hasher, not the resultant hash. In order to get that, I need to call Sum, which returns a []byte. That's what I want to store.
But I don't want to declare to all and sundry that it is a []byte, so I'm going to introduce another type in my own space called Hash. This has a ripple effect through the code, but the main thing is that we need to call Sum on the hash in main() and pass that.
var hasher maphash.Hash
hasher.WriteString("hello, world")
h := hasher.Sum(nil)
tx, err := api.NewTransaction("http://tx.info/msg1", h)
INTRODUCED_HASH_TYPE:cmd/ledgerclient/main.go
It's a somewhat complicated and subtle thing, but apart from the "clarity" that you get from renaming a type in this way, there are practical benefits in Go: "methods" can be associated with types, even if the type is, as here, just a slice of bytes. Because although you could argue that's what it is, the compiler carries around a static type that enables it to understand a "method application" and turn it into a function call with the array of bytes as the method receiver. I don't understand enough yet to know whether this persists at runtime, or whether this is forgotten in the code and it really is just a "byte slice".Since this is a change on the client side, it automatically propagates to the server and we now see on the that the JSON to be unmarshalled looks like this:
{"ContentHash":"/pWORCcEdbM=",But it still doesn't unmarshal successfully, because those strings cannot be automatically unmarshalled back into url.URL fields in Transaction. To handle this, we need a custom UnmarshalJson method:
"ContentLink":"http://tx.info/msg1",
"Signatories":[
{"Signature":null,"Signer":"https://user2.com"},
{"Signature":"Mizxonkpsi/tGykneTeprJ0LovyxMBDWZ3uF6S4LlLX+Ssy+rhPTfq26ILGhDJiIOlPHNebyoGW3+yFQCYu3NFRNuGwZnAGoFajnBDO7oTf2ctk8GBLypb5Ow8IzkiSNU1LPD8c/ZoUkE1qlx6niQXAbH3UpQ1drvh0Td0JP2ja3RktxFKCz9B36X/Hkj3lBOb0hv+ztCD4LVTbd49RSh4ROoLcBFkaqNXx7OuJGIUEMXwBofbvVSnuzZZV4oc6OP/JIl/MUqRHa+uqIGSSOHiZXZZjJ0OVFJKTUYbu5+lTxmzdX/TxcOx8svkbnnst2w1mxBxVVpmCql3VgD3PIDw==","Signer":"https://user1.com/"}
]}
func (tx *Transaction) UnmarshalJSON(bs []byte) error {
var wire struct {
ContentLink string
ContentHash []byte
Signatories []*types.Signatory
}
if err := json.Unmarshal(bs, &wire); err != nil {
return err
}
if url, err := url.Parse(wire.ContentLink); err == nil {
tx.ContentLink = url
} else {
return err
}
tx.ContentHash = wire.ContentHash
tx.Signatories = wire.Signatories
return nil
}
CUSTOM_UNMARSHAL_URL:internal/api/transaction.go
I was actually surprised how hard this was to do; I was thinking there would be some way to specify that you wanted to intervene in the standard unmarshalling process, but you basically need to rewrite the whole unmarshaller, including specifying the wire type. So what this does is to declare an anonymous type which has fields with the same names as the ones transmitted on the wire (this is obviously very important) and the types which were marshalled.It then parses the URL ContentLink field and assigns it to the Transaction, followed by the other two fields which can just be copied.
There are a couple of things about Go which I've used here that I didn't know yesterday. When I was reading one of the books about go, it talked about the form of the if statement I've used here and it seemed weird to me although it did describe it as "common in idiomatic Go". But seeing this in the example I was referencing from StackOverflow today, it made perfect sense, at least as long as you read it correctly. And, of course, learning to read a new programming language correctly is part of learning a new programming language :-)
It allows you to place the assignment of url and err in a scope which lasts just for the duration of the if and else blocks (and, presumably, any else if blocks). This means that you can reuse the same name err in multiple if statements without conflict, which is something I've been doing up until now that has been annoying me. Look for that to quietly change off camera as we go along.
The other thing is the anonymous type. I haven't thoroughly understood this yet, but it seems to be that the Go compiler builds up a "real" typename which is some kind of view of the structure of the declaration and can decide to use that as opposed to just going off the declared type name, although often it will choose to say that two types are different just because of the name they have been given. Anyway, the important point is that you can create an anonymous type here, in a very lightweight way, and then use it as a stepping stone to your real objective.
We then obviously have to do the same thing with the Signatory struct:
func (sig *Signatory) UnmarshalJSON(bs []byte) error {
var wire struct {
Signer string
Signature *Signature
}
if err := json.Unmarshal(bs, &wire); err != nil {
return err
}
if url, err := url.Parse(wire.Signer); err == nil {
sig.Signer = url
} else {
return err
}
sig.Signature = wire.Signature
return nil
}
CUSTOM_UNMARSHAL_URL:internal/types/signatory.go
Reflections
I am still having some issues with getting used to Go, and in particular the VSCode environment I've chosen to use.One of the general problems of developing client/server applications is that (by definition) you have two binaries that you need to run simultaneously. This is quite a lot of clicking, and I haven't yet seen a way of reducing it to one click (although believe that such a thing may be possible in VSCode). When running the client, you run it and then it's done. But when it comes to debugging the server, you have to remember to stop it and restart it, otherwise your changes have no effect.
VSCode does have a warning for this, but I turned it off because it warns you all the time when you are writing code; even when you are changing client code that doesn't affect the running server. So there isn't a perfect solution (if there is a perfect solution, please let me know?).
Hopefully most of these issues will go away in a short while when I start writing and running automated tests instead of testing manually.
No comments:
Post a Comment