So far, I've managed to build a trivial client/server application and a full-on test harness. I seem to remember setting out to build a blockchain. So this episode is going to focus on assembling the code needed to build a blockchain - driven by tests. It will take a few more episodes to build out all the code we need to have that running in a live node.
The basic idea of a blockchain is what the name implies: there is a sequence of blocks, and they are chained together. The purpose of the blockchain is to provide assurance than nothing has been altered - the very idea of a ledger. Each block gives us assurance about the messages contained in it. It is therefore important that we can validate each block. We do this by looking at the block's hash and signature. The hash is a very short summary of what happened in the last time slice combined with the previous block's hash. Inductively, this includes every message in every block back to the beginning of time. This makes it impossible to change any message or block anywhere in the chain without rewriting all of the subsequent blocks and changing all of the hashes and then resigning everything with the node's original signing key. The security comes from the fact that you can't do that on a public blockchain without somebody noticing (or at least being able to notice). (We will come back to the nodes cross-checking each other's work later; the blocks we build here do not guarantee that, just make it possible for other nodes to do that.)
So what we are going to do here is to build a "block" at regular intervals which includes: the hash of the previous block; the name of the node building the block; the time at which the block "closed"; a list of hashes, one for each message published in the timeframe of this block; the hash of all this information, which constitutes this block's ID; a signature of this hash.
So what about block 0? It would be possible to just "start" and say the only thing that is special about block 0 is that it doesn't have a previous block hash. I don't like that for a number of reasons, but the most obvious is that just seems imprecise. I don't think it actually opens any security holes that aren't already there, but it may do.
So my block 0 is going to all the information we do have: the node name; the start time; and a hash and a signature. This hash is what will be used in block 1.
Running the Block Builder
I've said that the block builder needs to build each new block at regular intervals. On a single machine in Go, the easiest thing to do is to start a goroutine to make that happen.In the last episode, we moved almost all of the Node code into node.go. I now want to add code in here to create and start a block builder, but as I look at it, I realize that it is a bit cluttered, so I'm going to refactor what's there first, and extract all the code that sets up the HTTP API handler into its own method. I'm not doing this because I'm planning to share or reuse it; just for clarity.
func (node *ListenerNode) Start() {
log.Println("starting chainledger node")
clock := helpers.ClockLive{}
config, err := config.ReadNodeConfig()
if err != nil {
fmt.Printf("error reading config: %s\n", err)
return
}
pending := storage.NewMemoryPendingStorage()
resolver := NewResolver(&clock, config.NodeKey, pending)
journaller := storage.NewJournaller()
node.startAPIListener(resolver, journaller)
}
func (node *ListenerNode) startAPIListener(resolver Resolver, journaller storage.Journaller) {
cliapi := http.NewServeMux()
pingMe := PingHandler{}
cliapi.Handle("/ping", pingMe)
storeRecord := NewRecordStorage(resolver, journaller)
cliapi.Handle("/store", storeRecord)
err := http.ListenAndServe(node.addr, cliapi)
if err != nil && !errors.Is(err, http.ErrServerClosed) {
fmt.Printf("error starting server: %s\n", err)
}
}
NODE_HANDLER_EXTRACT_API:internal/clienthandler/node.go
I believe I mentioned at the time I moved the code out of the cmd package that clienthandler didn't feel like the right place for this. Now that I want to add BlockBuilder it certainly doesn't seem right. But I'm going to refrain from moving it until I have a good idea where I do want to put it. In the meantime, I am going to add the code for the BlockBuilder and put that in its own package block.func (node *ListenerNode) Start() {
log.Println("starting chainledger node")
clock := helpers.ClockLive{}
config, err := config.ReadNodeConfig()
if err != nil {
fmt.Printf("error reading config: %s\n", err)
return
}
pending := storage.NewMemoryPendingStorage()
resolver := NewResolver(&clock, config.NodeKey, pending)
journaller := storage.NewJournaller()
node.runBlockBuilder(&clock, journaller)
node.startAPIListener(resolver, journaller)
}
func (node ListenerNode) runBlockBuilder(clock helpers.Clock, journaller storage.Journaller) {
builder := block.NewBlockBuilder(clock, journaller)
builder.Start()
}
INTRODUCE_BLOCK_BUILDER:internal/clienthandler/node.go
And the code in block.go is absolutely minimal at the moment:package block
import (
"github.com/gmmapowell/ChainLedger/internal/helpers"
"github.com/gmmapowell/ChainLedger/internal/storage"
)
type BlockBuilder interface {
Start()
}
type SleepBlockBuilder struct {
}
func (builder *SleepBlockBuilder) Start() {
}
func NewBlockBuilder(clock helpers.Clock, journal storage.Journaller) BlockBuilder {
return &SleepBlockBuilder{}
}
INTRODUCE_BLOCK_BUILDER:internal/block/builder.go
Start wants to kick off a goroutine which spends most of its time asleep, but once in a while (for now, let's say every 5s but come back and configure it later) it kicks off a task to build the most recent block. To allow time for the jounaller to finish storing all of the transactions we have received, we will "pause" for one second before assembling the block. We don't need to take this into consideration (or the time spent assembling the block) because the moment we have waited for our "official" sleep, we call After again to set up the next interval.Let's start with at least putting those "constants" at the top of the file:
const delay = 5 * time.Second
const pause = 1 * time.Second
BLOCK_BUILDER_DELAY_LOOP:internal/block/builder.go
Because I want things in general to be testable, I want to do all of this work with times to go through my testable abstraction Clock. We need to pass this in to the constructor and store it in the SleepBlockBuilder. Then we can implement Start as a simple go to fork Run() in a goroutine.type SleepBlockBuilder struct {
clock helpers.Clock
}
func (builder *SleepBlockBuilder) Start() {
go builder.Run()
}
func (builder *SleepBlockBuilder) Run() {
timer := builder.clock.After(delay)
for {
blocktime := <-timer
timer = builder.clock.After(delay)
runAt := <-builder.clock.After(pause)
fmt.Printf("Building block ending %s at %s\n", blocktime.IsoTime(), runAt.IsoTime())
}
}
func NewBlockBuilder(clock helpers.Clock, journal storage.Journaller) BlockBuilder {
return &SleepBlockBuilder{clock: clock}
}
BLOCK_BUILDER_DELAY_LOOP:internal/block/builder.go
In Run, we start off by creating a timer. In theory at least, clock.After is just a simple wrapper around time.After which works by creating and providing a channel down which exactly one message will come, indicating the current time, which is the time from "now", "after" the requested delay has happened.We read this time, which is the "end time" of the block we want to create. Then we re-initialize the timer for the next 5s interval. We then call After again, to pause for the small delay to allow all the transactions to be stored. For now, we then just print out the current time and the end time of the block. Everything works exactly as we expected:
2024/12/16 14:36:05 starting chainledger nodeThere is one message every 5s, and there is a clear 1s gap (actually a few ms more, as you might expect) between the "end" time and the execution time.
2024/12/16 14:36:11 Building block ending 2024-12-16_14:36:10.395 at 2024-12-16_14:36:11.399
2024/12/16 14:36:16 Building block ending 2024-12-16_14:36:15.399 at 2024-12-16_14:36:16.402
2024/12/16 14:36:21 Building block ending 2024-12-16_14:36:20.402 at 2024-12-16_14:36:21.405
2024/12/16 14:36:26 Building block ending 2024-12-16_14:36:25.405 at 2024-12-16_14:36:26.408
2024/12/16 14:36:31 Building block ending 2024-12-16_14:36:30.408 at 2024-12-16_14:36:31.411
While I am attempting to write this code in a way which will allow it to be tested - using Clock - I don't actually plan on testing it: the code I am interested in testing is the code that actually builds the block.
Even so, I need to provide both "live" and "double" implementations of Clock.After:
type Clock interface {
Time() types.Timestamp
After(d time.Duration) <-chan types.Timestamp
}
type ClockDouble struct {
Times []types.Timestamp
next int
afters []chan types.Timestamp
}
BLOCK_BUILDER_DELAY_LOOP:internal/helpers/clock.go
The purpose of the afters member is to give us somewhere to store the channels we return to users. The idea would be that there will be a method on the ClockDouble that would allow the test method to put a message on this channel when it wants the code to execute.func (clock *ClockDouble) After(d time.Duration) <-chan types.Timestamp {
ret := make(chan types.Timestamp)
clock.afters = append(clock.afters, ret)
return ret
}
BLOCK_BUILDER_DELAY_LOOP:internal/helpers/clock.go
And the After method creates and returns a channel, and then stores it in that slice.The live version is just a simple wrapper:
func (clock *ClockLive) After(d time.Duration) <-chan types.Timestamp {
ret := make(chan types.Timestamp)
mine := time.After(d)
go func() {
endTime := <-mine
ret <- types.Timestamp(endTime.UnixMilli())
close(ret)
}()
return ret
}
BLOCK_BUILDER_DELAY_LOOP:internal/helpers/clock.go
I say simple, but this actually a little complicated by the fact that the After code does not block, but you need to block in order to wait for the event to fire. So that code has to be shoved off into a goroutine.Asking to Build a Block
So now we have an infinite loop, but it doesn't do anything as yet. What we want is for it to build a block each time it goes around in the loop, and we also want to build "Block 0" just before the loop starts.If you are thinking, "hang on, what do you mean always build block 0, don't you need to pick up where you left off?", this version of the code is strictly memory-only and doesn't have any ability to store or recover any of the records it creates. So every time we start the node, we will start building blocks from scratch all over again. So, yes, we do want to build block 0, every single time. And, yes, we will need to revisit this when we go web-scale.
type SleepBlockBuilder struct {
journaller storage.Journaller
blocker *Blocker
clock helpers.Clock
}
func (builder *SleepBlockBuilder) Start() {
go builder.Run()
}
func (builder *SleepBlockBuilder) Run() {
blocktime := builder.clock.Time()
timer := builder.clock.After(delay)
lastBlock := builder.blocker.Build(blocktime, nil, nil)
for {
prev := blocktime
blocktime = <-timer
timer = builder.clock.After(delay)
<-builder.clock.After(pause)
txs, _ := builder.journaller.ReadTransactionsBetween(prev, blocktime)
lastBlock = builder.blocker.Build(blocktime, lastBlock, txs)
}
}
func NewBlockBuilder(clock helpers.Clock, journal storage.Journaller) BlockBuilder {
url, _ := url.Parse("https://node1.com")
pk, _ := rsa.GenerateKey(rand.Reader, 16)
return &SleepBlockBuilder{clock: clock, journaller: journal, blocker: &Blocker{name: url, pk: pk}}
}
BASIC_BLOCK_BUILDER:internal/block/builder.go
NewBlockBuilder now uses the clock and the journaller to build the SleepBlockBuilder. It also constructs a Blocker (see below), passing it a "name" and "private key" (hardcoded for now, to be configured later) and stores that in the struct.The code inside Run() now tracks the blocktime, the last time before which transactions will be included in the block; and a lastBlock which is initially created as the empty "block 0". Inside the loop, the transactions stored during the half-closed interval from prev (the previous value of blocktime) up to but not including blocktime are recovered, and then the blocker asked to create a new block based on the blocktime, the previous block and this set of transactions. The memory of the previous block is then replaced with the information about the current block before the loop repeats.
This requires the Journaller to have a ReadTransactionsBetween method, although we haven't implemented it yet.
type Journaller interface {
RecordTx(tx *records.StoredTransaction) error
ReadTransactionsBetween(from types.Timestamp, upto types.Timestamp) ([]records.StoredTransaction, error)
}
func (d DummyJournaller) ReadTransactionsBetween(from types.Timestamp, upto types.Timestamp) ([]records.StoredTransaction, error) {
return nil, nil
}
BASIC_BLOCK_BUILDER:internal/storage/journal.go
Then we need an implementation of Blocker, which is outlined here:package block
import (
"crypto/rsa"
"log"
"net/url"
"github.com/gmmapowell/ChainLedger/internal/records"
"github.com/gmmapowell/ChainLedger/internal/types"
)
type Blocker struct {
name *url.URL
pk *rsa.PrivateKey
}
func (b Blocker) Build(to types.Timestamp, last *records.Block, txs []records.StoredTransaction) *records.Block {
ls := "<none>"
if last != nil {
ls = last.String()
}
log.Printf("Building block before %s, following %s with %d records\n", to.IsoTime(), ls, len(txs))
return &records.Block{}
}
BASIC_BLOCK_BUILDER:internal/block/blocker.go
And we need a struct to represent the Blocks we want to create:package records
type Block struct {
}
func (b Block) String() string {
return "Block"
}
BASIC_BLOCK_BUILDER:internal/records/block.go
And once again, this minimal thing logs exactly what we expect:2024/12/17 13:10:00 starting chainledger node
2024/12/17 13:10:00 Building block before 2024-12-17_13:10:00.22, following <none> with 0 records
2024/12/17 13:10:06 Building block before 2024-12-17_13:10:05.22, following Block with 0 records
2024/12/17 13:10:11 Building block before 2024-12-17_13:10:10.224, following Block with 0 records
We Can Now Build a Block
This has taken care of all of mechanisms around building blocks, and now we need to actually do the job we came here to do: build a block On the upside, although we haven't tested anything up to this point, we have put ourselves in a position where the code we are about to write is testable.Of course, we are back in that hole where I say "the code is testable" and then say "but testing that you hash something correctly is hard". This time, however, I am not going to give in and run away - I am going to test that the hashing is done correctly (yes, what I am doing here could have been done before; having put the effort in here, I may well go back and retrofit tests). The thing is, of course, we need a "test double" for our hashing algorithm: the hashing algorithm itself is not under test: that is code provided by the Go library and I have no doubts that it does correctly implement the SHA-512 algorithm.
The problem of course, is exactly the same one that we faced with Clock: we want to just create a SHA-512 hasher in place, but that ties us to a specific implementation. So instead, we need to retrofit our code to pass it a "hasher factory": in the live environment, this will call the same constructor for a SHA-512 hasher we would naturally call; but in the test environment it returns a test double we have prepared for it.
Here our double will be most similar to a mock in that we will create a double which knows what calls it expects to be made and checks that those calls are made. The final call to Sum will then return a value passed in to the mock when it was created, and this can then be used further downstream in the testing.
Hopefully all this will become clear as we play the game. Let's start that by writing a test:
package block_test
import (
"bytes"
"crypto/rand"
"crypto/rsa"
"net/url"
"testing"
"github.com/gmmapowell/ChainLedger/internal/block"
"github.com/gmmapowell/ChainLedger/internal/types"
)
func TestBuildingBlock0(t *testing.T) {
nodeName, _ := url.Parse("https://node1.com")
pk, _ := rsa.GenerateKey(rand.Reader, 32)
blocker := block.NewBlocker(nodeName, pk)
buildTo, _ := types.ParseTimestamp("2024-12-12_18:00:00.000")
retHash := types.Hash([]byte("computed-hash"))
retSig := types.Hash([]byte("signed as"))
block0 := blocker.Build(buildTo, nil, nil)
if block0.PrevID != nil {
t.Fatalf("Block0 should have a nil previous block")
}
if block0.UpUntil != buildTo {
t.Fatalf("the stored block time was not correct")
}
if len(block0.Txs) != 0 {
t.Fatalf("Block0 should not have any messages")
}
if !bytes.Equal(block0.ID, retHash) {
t.Fatalf("the computed hash was incorrect")
}
if !bytes.Equal(block0.Signature, retSig) {
t.Fatalf("the computed hash was incorrect")
}
}
BLOCKER_MINIMAL_TEST:internal/block/blocker_test.go
As yet, this doesn't have the "hasher factory" in it, but we have written enough to have a failing test. It also forces us to write a little more production code. It calls a NewBlocker function to create the Blocker:func NewBlocker(name *url.URL, pk *rsa.PrivateKey) *Blocker {
return &Blocker{name: name, pk: pk}
}
BLOCKER_MINIMAL_TEST:internal/block/blocker.go
and it depends on the Block type having the fields it wants to test:type Block struct {
ID types.Hash
PrevID types.Hash
BuiltBy *url.URL
UpUntil types.Timestamp
Txs []types.Hash
Signature types.Signature
}
BLOCKER_MINIMAL_TEST:internal/records/block.go
It would either be laughable or scary if this test passed, but I'm glad to report that it doesn't.Let's do the easy bits first:
func (b Blocker) Build(to types.Timestamp, last *records.Block, txs []records.StoredTransaction) *records.Block {
ls := "<none>"
if last != nil {
ls = last.String()
}
log.Printf("Building block before %s, following %s with %d records\n", to.IsoTime(), ls, len(txs))
return &records.Block{
UpUntil: to,
BuiltBy: b.name,
PrevID: nil,
Txs: nil,
}
}
BLOCKER_SIMPLE_FIELDS:internal/block/blocker.go
This gets through the first few assertions, but fails at the hash. OK, we need that hasher factory now.First, let's work through all the plumbing. In order to do this, we need an interface for the factory, two implementations (one for SHA512, one for the mock) and we need to instantiate the mock in the test. Then we need to create a mock factory and pass it in through the test, backfilling the live version as appropriate. Deep breath!
All the new code is going in one file: helpers/hashing.go
The preamble and the interface:
package helpers
import (
"crypto/sha512"
"hash"
)
type HasherFactory interface {
NewHasher() hash.Hash
}
BLOCKER_MOCK_HASH:internal/helpers/hashing.go
The SHA512 implementation:type SHA512Factory struct {
}
func (f SHA512Factory) NewHasher() hash.Hash {
return sha512.New()
}
BLOCKER_MOCK_HASH:internal/helpers/hashing.go
(Note that while I am thinking about this as the live implementation, if you wanted other instances for other hashing algorithms, you could do that.)And the outline mock implementation:
type MockHasherFactory struct {
hashers []MockHasher
next int
}
func (f *MockHasherFactory) NewHasher() hash.Hash {
r := f.hashers[f.next]
f.next++
return r
}
type MockHasher struct {
}
// BlockSize implements hash.Hash.
func (m MockHasher) BlockSize() int {
panic("unimplemented")
}
// Reset implements hash.Hash.
func (m MockHasher) Reset() {
panic("unimplemented")
}
// Size implements hash.Hash.
func (m MockHasher) Size() int {
panic("unimplemented")
}
// Sum implements hash.Hash.
func (m MockHasher) Sum(b []byte) []byte {
panic("unimplemented")
}
// Write implements hash.Hash.
func (m MockHasher) Write(p []byte) (n int, err error) {
panic("unimplemented")
}
BLOCKER_MOCK_HASH:internal/helpers/hashing.go
We will want to use one of these in the Blocker, so we need to update that to store one and take one during construction:type Blocker struct {
hasher helpers.HasherFactory
name *url.URL
pk *rsa.PrivateKey
}
func NewBlocker(hasher helpers.HasherFactory, name *url.URL, pk *rsa.PrivateKey) *Blocker {
return &Blocker{hasher: hasher, name: name, pk: pk}
}
BLOCKER_MOCK_HASH:internal/block/blocker.go
In the block builder, we need to pass this in to NewBlocker. Because we aren't (currently) planning on testing that code, the easiest thing to do is to create a SHA512 instance on the fly and pass that in. If we did (or ever do) want to test that code, we would need to externalize that so that either a live or a mock instance could be passed in along with clock and journaller.func NewBlockBuilder(clock helpers.Clock, journal storage.Journaller) BlockBuilder {
url, _ := url.Parse("https://node1.com")
pk, _ := rsa.GenerateKey(rand.Reader, 16)
hf := helpers.SHA512Factory{}
blocker := NewBlocker(&hf, url, pk)
return &SleepBlockBuilder{clock: clock, journaller: journal, blocker: blocker}
}
BLOCKER_MOCK_HASH:internal/block/builder.go
And, finally, we can update our test:func TestBuildingBlock0(t *testing.T) {
nodeName, _ := url.Parse("https://node1.com")
pk, _ := rsa.GenerateKey(rand.Reader, 32)
hasher := helpers.MockHasherFactory{}
blocker := block.NewBlocker(&hasher, nodeName, pk)
buildTo, _ := types.ParseTimestamp("2024-12-12_18:00:00.000")
retHash := types.Hash([]byte("computed-hash"))
retSig := types.Hash([]byte("signed as"))
block0 := blocker.Build(buildTo, nil, nil)
if block0.PrevID != nil {
t.Fatalf("Block0 should have a nil previous block")
}
if block0.UpUntil != buildTo {
t.Fatalf("the stored block time was not correct")
}
if len(block0.Txs) != 0 {
t.Fatalf("Block0 should not have any messages")
}
if !bytes.Equal(block0.ID, retHash) {
t.Fatalf("the computed hash was incorrect")
}
if !bytes.Equal(block0.Signature, retSig) {
t.Fatalf("the computed hash was incorrect")
}
}
BLOCKER_MOCK_HASH:internal/block/blocker_test.go
Nobody will be surprised that the test still fails. Some may be surprised that it still fails with the same assertion:=== RUN TestBuildingBlock0The reason, of course, is that although we have done all this hard work to get a hasher in place, we don't actually use it!
2024/12/19 11:58:54 Building block before 2024-12-12_18:00:00, following <none> with 0 records
blocker_test.go:34: the computed hash was incorrect
--- FAIL: TestBuildingBlock0 (0.00s)
Adding a trivial call to invoke the HasherFactory:
func (b Blocker) Build(to types.Timestamp, last *records.Block, txs []records.StoredTransaction) *records.Block {
ls := "<none>"
if last != nil {
ls = last.String()
}
log.Printf("Building block before %s, following %s with %d records\n", to.IsoTime(), ls, len(txs))
hasher := b.hasher.NewHasher()
hasher.Sum(nil)
return &records.Block{
UpUntil: to,
BuiltBy: b.name,
PrevID: nil,
Txs: nil,
}
}
HASHER_TRIVIAL_CALL:internal/block/blocker.go
Gives us the desired failure that we don't have one set up:=== RUN TestBuildingBlock0So let's sort this out. We need to add a new mock in the unit test. I'm going to design this as a mock where the constructor takes the value to be returned by Sum, and then we will add the other calls we expect to receive (in the correct order) onto the mock directly using methods.
2024/12/19 12:02:31 Building block before 2024-12-12_18:00:00, following <none> with 0 records
--- FAIL: TestBuildingBlock0 (0.00s)
panic: runtime error: index out of range [0] with length 0 [recovered]
panic: runtime error: index out of range [0] with length 0
goroutine 4 [running]:
testing.tRunner.func1.2({0x574f40, 0xc00001a1b0})
/usr/local/go/src/testing/testing.go:1632 +0x230
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:1635 +0x35e
panic({0x574f40?, 0xc00001a1b0?})
/usr/local/go/src/runtime/panic.go:785 +0x132
github.com/gmmapowell/ChainLedger/internal/helpers.(*MockHasherFactory).NewHasher(0xc0001001b0?)
/home/gareth/Projects/ChainLedger/internal/helpers/hashing.go:25 +0x39
github.com/gmmapowell/ChainLedger/internal/block.Blocker.Build({{0x5beaa0?, 0xc00011c1a0?}, 0xc00012e000?, 0xc000130000?}, 0x193bc071100, 0x0, {0x0?, 0x0, 0x683670?})
/home/gareth/Projects/ChainLedger/internal/block/blocker.go:26 +0x17b
github.com/gmmapowell/ChainLedger/internal/block_test.TestBuildingBlock0(0xc00011e4e0)
/home/gareth/Projects/ChainLedger/internal/block/blocker_test.go:23 +0xe5
testing.tRunner(0xc00011e4e0, 0x5916d8)
/usr/local/go/src/testing/testing.go:1690 +0xf4
created by testing.(*T).Run in goroutine 1
/usr/local/go/src/testing/testing.go:1743 +0x390
FAIL github.com/gmmapowell/ChainLedger/internal/block 0.006s
For now, we only need the return value, so we can do this:
func TestBuildingBlock0(t *testing.T) {
nodeName, _ := url.Parse("https://node1.com")
pk, _ := rsa.GenerateKey(rand.Reader, 32)
hasher := helpers.MockHasherFactory{}
hasher.AddMock("computed-hash")
blocker := block.NewBlocker(&hasher, nodeName, pk)
buildTo, _ := types.ParseTimestamp("2024-12-12_18:00:00.000")
retHash := types.Hash([]byte("computed-hash"))
retSig := types.Hash([]byte("signed as"))
block0 := blocker.Build(buildTo, nil, nil)
if block0.PrevID != nil {
t.Fatalf("Block0 should have a nil previous block")
}
if block0.UpUntil != buildTo {
t.Fatalf("the stored block time was not correct")
}
if len(block0.Txs) != 0 {
t.Fatalf("Block0 should not have any messages")
}
if !bytes.Equal(block0.ID, retHash) {
t.Fatalf("the computed hash was incorrect")
}
if !bytes.Equal(block0.Signature, retSig) {
t.Fatalf("the computed signature was incorrect")
}
}
BLOCKER_FIRST_MOCK:internal/block/blocker_test.go
The AddMock method needs implementing:func (f *MockHasherFactory) AddMock(hashesTo string) MockHasher {
ret := MockHasher{hashesTo: hashesTo}
f.hashers = append(f.hashers, ret)
return ret
}
BLOCKER_FIRST_MOCK:internal/helpers/hashing.go
which requires us to add hashesTo into the struct:type MockHasher struct {
hashesTo string
}
BLOCKER_FIRST_MOCK:internal/helpers/hashing.go
Now we will get the MockHasher back, but it will panic as soon as we try and use it to call Sum, so let's implement that by returning the hashesTo value.func (m MockHasher) Sum(b []byte) []byte {
// This is an implementation detail
// and can easily be checked by adding another argument to the constructor
if b != nil {
panic("mock always expects final block to be nil")
}
return []byte(m.hashesTo)
}
BLOCKER_FIRST_MOCK:internal/helpers/hashing.go
Note that this is converting a string to a []byte just in order to make it easy to write our test code - this allows us to configure the hasher with a string. The bytes of a string are, of course, a perfectly valid hash value - just an unlikely one to turn up in the real world.In the Blocker itself, we need to capture and store the return value from Sum:
func (b Blocker) Build(to types.Timestamp, last *records.Block, txs []records.StoredTransaction) *records.Block {
ls := "<none>"
if last != nil {
ls = last.String()
}
log.Printf("Building block before %s, following %s with %d records\n", to.IsoTime(), ls, len(txs))
hasher := b.hasher.NewHasher()
hash := hasher.Sum(nil)
return &records.Block{
ID: hash,
UpUntil: to,
BuiltBy: b.name,
PrevID: nil,
Txs: nil,
}
}
BLOCKER_FIRST_MOCK:internal/block/blocker.go
The test now accepts the hash and moves on to complaining about the signature:=== RUN TestBuildingBlock0(Again, kudos to the eagle-eyed who had spotted that I had originally duplicated the error message "the computed hash was incorrect". I had not spotted this and was confused why the hash was incorrect until I looked at the line where the panic occurred. Whoops!)
2024/12/19 12:16:01 Building block before 2024-12-12_18:00:00, following <none> with 0 records
blocker_test.go:38: the computed signature was incorrect
--- FAIL: TestBuildingBlock0 (0.00s)
Building the Correct Hash
While we are testing that the hash is being stored correctly - and that Sum is being called - we are not testing that the block hash is being built correctly.Remember that for block 0, we just want to hash the node name and the block timestamp (in that order). As before, the node name will be hashed with a newline on the end.
We can assert this in our test:
func TestBuildingBlock0(t *testing.T) {
nodeName, _ := url.Parse("https://node1.com")
pk, _ := rsa.GenerateKey(rand.Reader, 32)
hasher := helpers.MockHasherFactory{}
buildTo, _ := types.ParseTimestamp("2024-12-12_18:00:00.000")
mock1 := hasher.AddMock("computed-hash")
mock1.ExpectString(nodeName.String())
mock1.ExpectTimestamp(buildTo)
blocker := block.NewBlocker(&hasher, nodeName, pk)
retHash := types.Hash([]byte("computed-hash"))
retSig := types.Hash([]byte("signed as"))
block0 := blocker.Build(buildTo, nil, nil)
if block0.PrevID != nil {
t.Fatalf("Block0 should have a nil previous block")
}
if block0.UpUntil != buildTo {
t.Fatalf("the stored block time was not correct")
}
if len(block0.Txs) != 0 {
t.Fatalf("Block0 should not have any messages")
}
if !bytes.Equal(block0.ID, retHash) {
t.Fatalf("the computed hash was incorrect")
}
if !bytes.Equal(block0.Signature, retSig) {
t.Fatalf("the computed signature was incorrect")
}
}
BLOCKER_BLOCK_0:internal/block/blocker_test.go
where ExpectString and ExpectTimestamp are new (see below).We can then implement the hashing here:
func (b Blocker) Build(to types.Timestamp, last *records.Block, txs []records.StoredTransaction) *records.Block {
ls := "<none>"
if last != nil {
ls = last.String()
}
log.Printf("Building block before %s, following %s with %d records\n", to.IsoTime(), ls, len(txs))
hasher := b.hasher.NewHasher()
hasher.Write([]byte(b.name.String()))
hasher.Write([]byte("\n"))
hasher.Write(to.AsBytes())
hash := hasher.Sum(nil)
return &records.Block{
ID: hash,
UpUntil: to,
BuiltBy: b.name,
PrevID: nil,
Txs: nil,
}
}
BLOCKER_BLOCK_0:internal/block/blocker.go
This encodes the Timestamp by asking it to convert itself to a slice of bytes, which it does using the AsBytes method defined on Timestamp:func (ts Timestamp) AsBytes() []byte {
var s = make([]byte, 8)
binary.LittleEndian.PutUint64(s, uint64(ts))
return s
}
BLOCKER_BLOCK_0:internal/types/timestamp.go
As noted before, the timestamp is really just a 64-bit integer, so we can treat it as that and convert it to bytes by storing it as a little-endian 8-byte value (we could equally use big endian, just as long as we are consistent).So far, so good.
We now need to go back and implement the necessary methods on the mock, and it starts to get a bit more tricky. First off, we need to add a few fields to the MockHasher:
type MockHasher struct {
t testing.T
hashesTo string
blobs []byte
written []byte
}
BLOCKER_BLOCK_0:internal/helpers/hashing.go
blobs is all the bytes that we expect to be written and that is assembled by ExpectString and ExpectTimestamp:func (m *MockHasher) ExpectString(s string) *MockHasher {
m.blobs = append(m.blobs, []byte(s)...)
return m
}
func (m *MockHasher) ExpectTimestamp(ts types.Timestamp) *MockHasher {
m.blobs = append(m.blobs, ts.AsBytes()...)
return m
}
BLOCKER_BLOCK_0:internal/helpers/hashing.go
written meanwhile, tracks all the bytes that are actually written in Write:func (m *MockHasher) Write(p []byte) (n int, err error) {
m.written = append(m.written, p...)
return len(p), nil
}
BLOCKER_BLOCK_0:internal/helpers/hashing.go
Now, this requires us to change the signature of the method receiver from (m MockHasher) to (m *MockHasher) in order that the assignment to written can update the value passed in. If we don't do this, when the method is called, the object on which it is called will be copied and any updates will only affect the copy and not the original. The compiler spots this and warns about an "ineffective assignment":ineffective assignment to field MockHasher.written (SA4005)Fair enough. But what happens next confuses me. It's confused me before and led me to believe strange things about Go, but I think it really just comes down to the error messages being confusing.
In NewHasher for the MockHasherFactory, we now get this error message:
cannot use r (variable of type MockHasher) as hash.Hash value in return statement: MockHasher does not implement hash.Hash (method Write has pointer receiver)Why can't I have a pointer receiver if I want one? It turns out that the problem here is a very subtle issue in the Go type system, which is in and around the area I keep being confused with the relationship between interfaces, structs and pointers to structs. If all the methods in a struct take non-pointer method receivers, the struct can "implement" an interface. Otherwise, only a pointer to the struct can implement the interface. No, even as I write this it still doesn't make sense to me. But it is, apparently, true.
So what I need to do is to change the code that creates, stores and returns a MockHasher here:
type MockHasherFactory struct {to deal with *MockHasher references everywhere:
hashers []MockHasher
next int
}
func (f *MockHasherFactory) AddMock(hashesTo string) MockHasher {
ret := MockHasher{hashesTo: hashesTo}
f.hashers = append(f.hashers, ret)
return ret
}
func (f *MockHasherFactory) NewHasher() hash.Hash {
r := f.hashers[f.next]
f.next++
return r
}
type MockHasherFactory struct {including taking the address of the MockHasher the moment it is created.
hashers []*MockHasher
next int
}
func (f *MockHasherFactory) AddMock(hashesTo string) *MockHasher {
ret := &MockHasher{hashesTo: hashesTo}
f.hashers = append(f.hashers, ret)
return ret
}
func (f *MockHasherFactory) NewHasher() hash.Hash {
r := f.hashers[f.next]
f.next++
return r
}
Finally, we need to check that the bytes that are actually written are the same as the ones that were expected. We can do this in Sum as follows:
func (m *MockHasher) Sum(b []byte) []byte {In order to tie this in with the testing framework, we need to pass in a testing.T, and obviously we need to pass it in through the mock, since we can't reasonably ask the client code to do that. The moment we do this (see above in the struct definition), we get errors on all the other methods which are still taking non-pointer receivers:
// This is an implementation detail
// and can easily be checked by adding another argument to the constructor
if b != nil {
panic("mock always expects final block to be nil")
}
if !bytes.Equal(m.blobs, m.written) {
m.t.Log("the written blobs were not the expected blobs")
m.t.Logf("expected: %v\n", m.blobs)
m.t.Logf("written: %v\n", m.written)
m.t.Fail()
}
return []byte(m.hashesTo)
}
BlockSize passes lock by value: github.com/gmmapowell/ChainLedger/internal/helpers.MockHasher contains testing.T contains testing.common contains sync.RWMutexI'm not sure I fully understand this message, but it is basically saying it's not a good idea to copy a Mutex by value, which makes sense. And, when I think about it, I realize that I should have passed a pointer to *testing.T, not an actual object. So I'll change that.
I'm not sure how I feel about all of this. I have sort-of-sensed that you should put a pointer on a method if it wants to be a "write" method and not if it doesn't. But the more I think about it, if you are going to copy an object (even a shallow copy) every time you call a read-only method, that seems expensive if the object is any significant size. I'm tempted to think that I want a pointer receiver for every method. Maybe I should think about this some more.
Anyway, for all the work we have done here, the test still fails to have the right signature.
Signing is Much the Same
Signing is basically no different from hashing. It's something where it's very hard to know what the "correct" answer should be, but you do know what the correct steps for any given test are. So it makes sense to put a mock in place that checks the input and produces a "desired" about that can then be checked.It's so similar, I'm just going to quickly present an overview, rather than going into the same level of detail I did with the Hasher.
First, we add the signer to the test case:
func TestBuildingBlock0(t *testing.T) {
nodeName, _ := url.Parse("https://node1.com")
pk, _ := rsa.GenerateKey(rand.Reader, 32)
hasher := helpers.NewMockHasherFactory(t)
buildTo, _ := types.ParseTimestamp("2024-12-12_18:00:00.000")
mock1 := hasher.AddMock("computed-hash")
mock1.ExpectString(nodeName.String() + "\n")
mock1.ExpectTimestamp(buildTo)
signer := helpers.MockSigner{}
retHash := types.Hash([]byte("computed-hash"))
retSig := types.Hash([]byte("signed as"))
signer.Expect(retSig, pk, retHash)
blocker := block.NewBlocker(hasher, &signer, nodeName, pk)
block0, _ := blocker.Build(buildTo, nil, nil)
if block0.PrevID != nil {
t.Fatalf("Block0 should have a nil previous block")
}
if block0.UpUntil != buildTo {
t.Fatalf("the stored block time was not correct")
}
if len(block0.Txs) != 0 {
t.Fatalf("Block0 should not have any messages")
}
if !bytes.Equal(block0.ID, retHash) {
t.Fatalf("the computed hash was incorrect")
}
if !bytes.Equal(block0.Signature, retSig) {
t.Logf("expected sig: %v\n", retSig)
t.Logf("actual sig: %v\n", block0.Signature)
t.Fatalf("the computed signature was incorrect")
}
}
BLOCKER_MOCK_SIGNER:internal/block/blocker_test.go
And use it in the Blocker:type Blocker struct {
hasher helpers.HasherFactory
signer helpers.Signer
name *url.URL
pk *rsa.PrivateKey
}
func (b Blocker) Build(to types.Timestamp, last *records.Block, txs []records.StoredTransaction) (*records.Block, error) {
ls := "<none>"
if last != nil {
ls = last.String()
}
log.Printf("Building block before %s, following %s with %d records\n", to.IsoTime(), ls, len(txs))
hasher := b.hasher.NewHasher()
hasher.Write([]byte(b.name.String()))
hasher.Write([]byte("\n"))
hasher.Write(to.AsBytes())
hash := hasher.Sum(nil)
sig, err := b.signer.Sign(b.pk, (*types.Hash)(&hash))
if err != nil {
return nil, err
}
return &records.Block{
ID: hash,
UpUntil: to,
BuiltBy: b.name,
PrevID: nil,
Txs: nil,
Signature: *sig,
}, nil
}
BLOCKER_MOCK_SIGNER:internal/block/blocker.go
Note that since signing can fail, we have to handle the err return code. This propagates through the block building process, but I have not shown that since at the moment we just panic. In the fulness of time, we will need to properly handle all errors that occur in a way which respects the principle that a server should keep going even in the face of difficulties, rather than crashing.And obviously we need to define everything to do with signing in the helpers package:
package helpersThis is simpler than its equivalent hasher, because there is no need to "customize" the expected signatures: we know the input and provide the output at initialization. Other than that, we have an interface, a "live" version corresponding to the rsa.SignPSS method with specific customizations, and then the mock works in the expected way, handling Expect and Sign. Again, the mock takes a pointer to the testing environment so that it can cause the tests to fail.
import (
"bytes"
"crypto"
"crypto/rand"
"crypto/rsa"
"testing"
"github.com/gmmapowell/ChainLedger/internal/types"
)
type Signer interface {
Sign(pk *rsa.PrivateKey, hash *types.Hash) (*types.Signature, error)
}
type RSASigner struct {
}
func (s RSASigner) Sign(pk *rsa.PrivateKey, hash *types.Hash) (*types.Signature, error) {
sig, err := rsa.SignPSS(rand.Reader, pk, crypto.SHA512, []byte(*hash), nil)
if err != nil {
return nil, err
}
var ret types.Signature = sig
return &ret, nil
}
type MockSigner struct {
t *testing.T
sigs []*MockExpectedSig
next int
}
func (f *MockSigner) Expect(signature types.Hash, pk *rsa.PrivateKey, hash types.Hash) {
ret := &MockExpectedSig{t: f.t, signature: signature, pk: pk, hash: hash}
f.sigs = append(f.sigs, ret)
}
func (f *MockSigner) Sign(pk *rsa.PrivateKey, hash *types.Hash) (*types.Signature, error) {
r := f.sigs[f.next]
if pk != r.pk { // this is a pointer comparison, which is almost undoubtedly valid for tests
f.t.Log("primary keys did not match")
f.t.Fail()
}
if !bytes.Equal(r.hash, *hash) {
f.t.Log("hash was not correct")
f.t.Fail()
}
f.next++
return (*types.Signature)(&r.signature), nil
}
type MockExpectedSig struct {
t *testing.T
pk *rsa.PrivateKey
hash types.Hash
signature []byte
}
Building Subsequent Blocks
So much for Block 0. It is necessary, but it is not a chain. Fortunately, however, it represents most of the work that we have to do. To demonstrate this, let's add another test case. I'm going to refactor the (shared) initialization out of the test cases so that I don't duplicate it (the extracted setup function is not shown).This test case just tests that we can chain the blocks together; the block does not contain any messages.
func TestBuildingSubsequentBlockWithNoMessages(t *testing.T) {
setup(t)
mock1.ExpectString(string(prevID))
mock1.ExpectString(nodeName.String() + "\n")
mock1.ExpectTimestamp(buildTo)
prev := records.Block{ID: prevID}
block0, _ := blocker.Build(buildTo, &prev, nil)
if !bytes.Equal(block0.PrevID, prevID) {
t.Fatalf("Block1 should have a previous block id %v, not %v", prevID, block0.PrevID)
}
if block0.UpUntil != buildTo {
t.Fatalf("the stored block time was not correct")
}
if len(block0.Txs) != 0 {
t.Fatalf("Block0 should not have any messages")
}
if !bytes.Equal(block0.ID, retHash) {
t.Fatalf("the computed hash was incorrect")
}
if !bytes.Equal(block0.Signature, retSig) {
t.Logf("expected sig: %v\n", retSig)
t.Logf("actual sig: %v\n", block0.Signature)
t.Fatalf("the computed signature was incorrect")
}
}
BLOCK_LATER_EMPTY:internal/block/blocker_test.go
We can then make this test pass by using the ID of the previous block as the PrevID:func (b Blocker) Build(to types.Timestamp, last *records.Block, txs []records.StoredTransaction) (*records.Block, error) {
ls := "<none>"
var lastID types.Hash
if last != nil {
ls = last.String()
lastID = last.ID
}
log.Printf("Building block before %s, following %s with %d records\n", to.IsoTime(), ls, len(txs))
hasher := b.hasher.NewHasher()
hasher.Write(lastID)
hasher.Write([]byte(b.name.String()))
hasher.Write([]byte("\n"))
hasher.Write(to.AsBytes())
hash := hasher.Sum(nil)
sig, err := b.signer.Sign(b.pk, (*types.Hash)(&hash))
if err != nil {
return nil, err
}
return &records.Block{
ID: hash,
UpUntil: to,
BuiltBy: b.name,
PrevID: lastID,
Txs: nil,
Signature: *sig,
}, nil
}
BLOCK_LATER_EMPTY:internal/block/blocker.go
Including Messages in a Block
Believe it or not, we do now have a blockchain - a chain of blocks. But sadly, there are no messages included in the chain. So let's add that and then we will have a functioning, single-node blockchain.func TestBuildingSubsequentBlockWithTwoMessages(t *testing.T) {
setup(t)
mock1.ExpectString(string(prevID))
mock1.ExpectString(nodeName.String() + "\n")
mock1.ExpectTimestamp(buildTo)
prev := records.Block{ID: prevID}
m1id := types.Hash([]byte("msg1"))
m2id := types.Hash([]byte("msg2"))
mock1.ExpectHash(m1id)
mock1.ExpectHash(m2id)
msg1 := records.StoredTransaction{TxID: m1id}
msg2 := records.StoredTransaction{TxID: m2id}
block0, _ := blocker.Build(buildTo, &prev, []records.StoredTransaction{msg1, msg2})
if !bytes.Equal(block0.PrevID, prevID) {
t.Fatalf("Block1 should have a previous block id %v, not %v", prevID, block0.PrevID)
}
if block0.UpUntil != buildTo {
t.Fatalf("the stored block time was not correct")
}
if len(block0.Txs) != 0 {
t.Fatalf("Block0 should not have any messages")
}
if !bytes.Equal(block0.ID, retHash) {
t.Fatalf("the computed hash was incorrect")
}
if !bytes.Equal(block0.Signature, retSig) {
t.Logf("expected sig: %v\n", retSig)
t.Logf("actual sig: %v\n", block0.Signature)
t.Fatalf("the computed signature was incorrect")
}
}
BLOCK_LATER_MSGS:internal/block/blocker_test.go
And then add the code:func (b Blocker) Build(to types.Timestamp, last *records.Block, txs []records.StoredTransaction) (*records.Block, error) {
ls := "<none>"
var lastID types.Hash
if last != nil {
ls = last.String()
lastID = last.ID
}
log.Printf("Building block before %s, following %s with %d records\n", to.IsoTime(), ls, len(txs))
hasher := b.hasher.NewHasher()
hasher.Write(lastID)
hasher.Write([]byte(b.name.String()))
hasher.Write([]byte("\n"))
hasher.Write(to.AsBytes())
for _, m := range txs {
hasher.Write(m.TxID)
}
hash := hasher.Sum(nil)
sig, err := b.signer.Sign(b.pk, (*types.Hash)(&hash))
if err != nil {
return nil, err
}
return &records.Block{
ID: hash,
UpUntil: to,
BuiltBy: b.name,
PrevID: lastID,
Txs: nil,
Signature: *sig,
}, nil
}
BLOCK_LATER_MSGS:internal/block/blocker.go
And we have working tests of a working blocker. Sadly, we don't have a working application. For that, we need to implement the journal functions to remember and retrieve messages. Next time.Running Tests in VSCode
Thanks to this article I was finally able to figure out how to launch my unit tests in the VSCode debugger launch.json configuration.It would seem that the issue has something to do with dlv which I've seen referenced a number of times but not actually delved into (pun intended, I think). So I now have this testing target:
"configurations": [{It's sad that this is not what the tool automatically generates.
"name": "test",
"type": "go",
"request": "launch",
"mode": "test",
"program": "${workspaceFolder}/internal/block"
},
Refactoring and Tidying Up
While doing all of this, I've noticed some things about the code that I don't like anymore. This is basically because my understanding of Go - particularly around what does and does not constitute a "pointer value" - has improved, and I've been taking addresses of things that basically are pointers, or of things like Timestamps that are essentially just 64-bit quantities.I also fixed a bunch of minor annoyances and mistypings, and I also added tests for the hashing and signature functions that previously weren't tested in resolver_test.go: you will notice that some specifically test that the right things are hashed and signed, although others gloss over it; meanwhile the signature test uses the "live" hasher and signer so that it can verify the signature.
If you're interested in this, you can see what I did in the commit tagged REFACTOR_POINTER_THINGS.
No comments:
Post a Comment