Friday, May 30, 2025

Main CT600 Submitter


So now it's time to create the main program to submit the CT600.

I think I'm going to want to be able to do "all" of the various operations - submit, see what's out there, check a submission, delete a submission - from a single program, and that means that I'm going to want an argument that is the "command". But most of the other arguments are going to be configuration files. That being the case, I'm going to work on the assumption that the command will have the syntax:
ct600 <operation> <config>...
But for now, I'm just going to assume the operation "submit" (in fact, just transmitting and expecting an error) and only parse configuration files. I'm going to rejig what we did last time to accept these configuration files and then consolidate the configuration before attempting to generate a message and submit it.

Let's go!

A New Command

We already have a command to generate GnuCash files, so we're going to do something very similar. So similar, in fact, I am just going to copy it and then update it.
package main

import (
    "fmt"
    "os"

    "github.com/gmmapowell/ignorance/accounts/internal/ct600/config"
    "github.com/gmmapowell/ignorance/accounts/internal/ct600/submission"
)

func main() {
    conf := config.MakeBlankConfig()
    for _, f := range os.Args[1:] {
        err := config.IncludeConfig(conf, f)
        if err != nil {
            fmt.Printf("failed to read config %s: %v\n", f, err)
            return
        }
    }
    err := submission.Submit(conf)
    if err != nil {
        fmt.Printf("submission failed: %v\n", err)
        return
    }
}

CT600_MAIN_SUBMITTER:accounts/cmd/ct600/main.go

This is quite big for a cmd program, but I think at this moment I'm happy with it. We'll see how we do when we add multiple commands; I imagine I will want to put basically all of it somewhere else.

So, as promised, I refactored the config stuff to make a blank configuration, then to incorporate each file we provide on the command line, and then we call Submit. I'm not going to review the configuration changes, but here is the submit code that has emerged from what was left of the test case (now deleted):
package submission

import (
    "fmt"
    "io"
    "log"
    "net/http"
    "strings"

    "github.com/gmmapowell/ignorance/accounts/internal/ct600/config"
)

func Submit(conf *config.Config) error {
    send, err := Generate(conf)
    if err != nil {
        return err
    }

    return transmit(send)
}

func transmit(body io.Reader) error {
    cli := &http.Client{}
    resp, err := cli.Post("https://test-transaction-engine.tax.service.gov.uk/submission", "application/x-binary", body)
    if err != nil {
        log.Fatalf("error posting xml file: %v", err)
    }
    respBody, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatalf("error reading response: %v", err)
    }
    msg := string(respBody)
    fmt.Printf("%s", msg)
    if strings.Contains(msg, "GovTalkErrors") {
        return fmt.Errorf("there was a GovTalkErrors block")
    }

    return nil
}

CT600_MAIN_SUBMITTER:accounts/internal/ct600/submission/submit.go

Submit calls the existing Generate function, and then calls transmit, which is the old code from the test to send to the government website. This still has some constants in it (like the URL) that we will want to extract to configuration at some point (we will then need a set of "test" credentials and a set of "live" credentials in separate configuration files, going along with the URL and the GatewayTest flag). But for now, this is enough to get us running.

I put my valid credentials in a test file (of the same form as foo.json) and ran the ct600 script. And just like that, we have a "working" submission:
<?xml version="1.0" encoding="UTF-8"?>
<GovTalkMessage xmlns="http://www.govtalk.gov.uk/CM/envelope">
    <EnvelopeVersion>2.0</EnvelopeVersion>
    <Header>
        <MessageDetails>
            <Class>HMRC-CT-CT600</Class>
            <Qualifier>acknowledgement</Qualifier>
            <Function>submit</Function>
            <TransactionID></TransactionID>
            <CorrelationID>8E73E73CD1F34678BBFDE9B6C1B9139C</CorrelationID>
            <ResponseEndPoint PollInterval="10">https://test-transaction-engine.tax.service.gov.uk/poll</ResponseEndPoint>
            <GatewayTimestamp>2025-05-30T20:34:21.066</GatewayTimestamp>
        </MessageDetails>
        <SenderDetails/>
    </Header>
    <GovTalkDetails>
        <Keys/>
    </GovTalkDetails>
    <Body/>
</GovTalkMessage>
That is, my submission has been accepted for consideration. I imagine when I write the next line of code to go back and examine that submission, I imagine I will see that it was, in fact, rejected.

Conclusion

In what feels like no time at all we have replaced our scaffolding test with a main program and, with the judicious introduction of correct data, we are able to communicate with the government. Excellent progress. We now need to expand this program to accept commands so that we can check for existing submissions, see the results and clean up dead submissions.

Customizing the CT600

Now that we are generating the CT600, we have the ability to customize it. Most obviously, we need to put in our credentials, but there will be other things too as we move along. I think there will ultimately be four sources of data:
  • Things that are at the "app level" and would be the same across all clients, such as the credentials I received from SDST;
  • Things that are specific to the fact that we are submitting a CT600 (the overall gateway supports a lot more things, most obviously SA100s);
  • Things that are specific to this customer (i.e. my company) such as the UTR, but do not change year on year;
  • The actual accounts and tax details themselves.
To avoid falling into the trap that the ct600 program fell into, I am going to try and keep these "as close to where they belong" as I can. I think I need a special (second) config file to pull in my app secrets; this will allow anyone else who wants to use this code (idiots!) to sign up with SDST and fill in that configuration.

All the information related to it being a CT600 can either go in my existing configuraiton file or even a more general one, depending on how specific it is to the business.

I think I want to put things like the UTR into the spreadsheet, in a special "options" sheet.

OK, that's what I think, but I'm happy to be wrong. But what I need is to have these things available in the govtalk.go code that we wrote last time. So let's start there (where everything can be assumed to have been consolidated) and work our way backwards.

We can update our test to generate the file but not bother submitting it (especially since I'm on a train right now with very spotty wifi) but to create a Config and pass that in.
func TestWeCanConfigureSomething(t *testing.T) {
    config := &config.Config{}
    config.Sender = "me"
    config.Password = "secret"
    config.Utr = "1234509876"
    config.Vendor = "Ziniki"
    config.Product = "SendTax"
    config.Version = "2025-05-30-Alpha"
    send, err := submission.Generate(config)

CT600_CONFIG_1:accounts/internal/ct600/submission/submission_test.go

Here's the Config struct, at least so far as we've gone so far:
package config

type Config struct {
    Sender, Password         string
    Utr                      string
    Vendor, Product, Version string
}

CT600_CONFIG_1:accounts/internal/ct600/config/config.go

And in the generation, we can use these configured parameters to customize the "real world" GovTalk object:
func Generate(conf *config.Config) (io.Reader, error) {
    msg := govtalk.MakeGovTalk()
    msg.Identity(conf.Sender, conf.Password)
    msg.Utr(conf.Utr)
    msg.Product(conf.Vendor, conf.Product, conf.Version)
    bs, err := xml.MarshalIndent(msg.AsXML(), "", "  ")
    if err != nil {
        return nil, err
    }
    // fmt.Printf("%s", string(bs))
    return bytes.NewReader(bs), nil
}

CT600_CONFIG_1:accounts/internal/ct600/submission/generate.go

And, in turn, that can then generate the appropriate XML for those parameters:
type GovTalkMessage struct {
    sender, password         string
    utr                      string
    vendor, product, version string
}

func (gtm *GovTalkMessage) Identity(send, pwd string) {
    gtm.sender = send
    gtm.password = pwd
}

func (gtm *GovTalkMessage) Utr(utr string) {
    gtm.utr = utr
}

func (gtm *GovTalkMessage) Product(vendor, product, version string) {
    gtm.vendor = vendor
    gtm.product = product
    gtm.version = version
}

func (gtm *GovTalkMessage) AsXML() any {
    env := ElementWithText("EnvelopeVersion", "2.0")
    msgDetails := ElementWithNesting(
        "MessageDetails",
        ElementWithText("Class", "HMRC-CT-CT600"),
        ElementWithText("Qualifier", "request"),
        ElementWithText("Function", "submit"),
        ElementWithText("Transformation", "XML"),
        ElementWithText("GatewayTest", "1"),
    )
    sndrDetails := ElementWithNesting(
        "SenderDetails",
        ElementWithNesting(
            "IDAuthentication",
            ElementWithText("SenderID", gtm.sender),
            ElementWithNesting(
                "Authentication",
                ElementWithText("Method", "clear"),
                ElementWithText("Role", "Principal"),
                ElementWithText("Value", gtm.password),
            ),
        ),
    )
    gtDetails := ElementWithNesting(
        "GovTalkDetails",
        ElementWithNesting("Keys", Key("UTR", gtm.utr)),
        ElementWithNesting("TargetDetails", ElementWithText("Organisation", "HMRC")),
        ElementWithNesting(
            "ChannelRouting",
            ElementWithNesting(
                "Channel",
                ElementWithText("URI", gtm.vendor),
                ElementWithText("Product", gtm.product),
                ElementWithText("Version", gtm.version),
            ),
        ),
    )
    return MakeGovTalkMessage(env,
        ElementWithNesting("Header", msgDetails, sndrDetails),
        gtDetails,
        gtm.makeBody())
}

func (gtm *GovTalkMessage) makeBody() any {
    body := ElementWithNesting("Body")
    return &body
}

CT600_CONFIG_1:accounts/internal/ct600/govtalk/govtalk.go

Reading the configuration

So let's say that we don't want to set up a configuration in code but instead want to read it from files on the file system. How could we do that?

Well, we already have a lot of that code which reads the configuration to map all our tables, so we can copy, reuse or generalize that to read some other config files and then do that here.

Looking at that reader code, the big problem with it is that it creates its own Configuration object internally, which stops us passing our own in. However, we could extract that and allow the reader portion to take an object (which must at least implement Configuration so that we can extract the verbs). Any additional configuration will need to happen in the wrapper.

Let's update our test:
func TestWeCanConfigureSomething(t *testing.T) {
    config := &config.Config{}
    err := gcconf.ReadAConfiguration(config, "../../../testdata/foo.json")
    if err != nil {
        t.Fatalf("%v", err)
    }
    send, err := submission.Generate(config)

CT600_CONFIG_2:accounts/internal/ct600/submission/submission_test.go

The foo.json file is something I just created with dummy data and put in testdata at the top level. All those .. elements just say "go up to the top level".
{
    "Sender": "a-user",
    "Password": "secret",
    "Utr": "1234509876",
    "Vendor": "Ziniki",
    "Product": "SendTax",
    "Version": "2025-05-30-Alpha"
}

CT600_CONFIG_2:accounts/testdata/foo.json

I decided to refactor the existing configuration reader as discussed above. I ended up with this:
func ReadConfig(file string) (*Configuration, error) {
    ret := Configuration{VerbMap: make(map[string]*Verb)}
    err := ReadAConfiguration(&ret, file)
    if err != nil {
        return nil, err
    } else {
        return &ret, nil
    }
}

func ReadAConfiguration(config any, file string) error {
    bs, err := os.ReadFile(file)
    if err != nil {
        return err
    }
    err = json.Unmarshal(bs, config)
    if err != nil {
        panic(err)
    }
    vc, isConfig := config.(Configuration)
    if isConfig {
        for _, v := range vc.Verbs {
            vc.VerbMap[v.Name] = &v
        }
    }
    return nil
}

CT600_CONFIG_2:accounts/internal/gnucash/config/reader.go

I had intended that the config argument of ReadAConfiguration would be of type Configuration, but it would seem that's just an expectation I have coming from a Java background. But I am able to use any successfully, so I'm fine with that. A caveat, of course, is that the code to initialize the verbs depends on the instance actually being a Configuration, but we can easily test that and only execute that code if it's true.

One of the great things about "key/value" configuration is that we can say "well, there are four configuration files, but I don't really care, just use all four to initialize this object and then check it's done afterwards". It doesn't matter which parameters come from which files. We should be able to call ReadAConfiguration multiple times. Of course, I'm not sure that the code to iterate over the verbs will work perfectly every time, but we can sort that out if it is a problem.

Finally, of course, we need to update our Config to "include" a GnuCash Configuration, and we need to provide an equivalent "create-and-read" function, although in the fulness of time I think we will want this to accept multiple configuration files.
type Config struct {
    config.Configuration
    Sender, Password         string
    Utr                      string
    Vendor, Product, Version string
}

func ReadConfig(file string) (*Config, error) {
    ret := Config{Configuration: config.Configuration{VerbMap: make(map[string]*config.Verb)}}
    err := config.ReadAConfiguration(&ret, file)
    if err != nil {
        return nil, err
    } else {
        return &ret, nil
    }
}

CT600_CONFIG_2:accounts/internal/ct600/config/config.go

The nested configuration initialization seems kind of klunky, so I imagine at some point I will refactor this. I may or may not mention it when I'm doing it.

Conclusion

That feels enough for today. We've customized what we're doing and we've enabled that to be read from a file. I think my next step will be to put my actual parameters in actual files, create a cmd file, and delete this non-test.

I will also want to pull some of the configuration parameters from the spreadsheet as we read that. Oh, what fun!

Thursday, May 29, 2025

Generating the CT600 XML


In retrospect, I think I was a little disingenuous with the ending of the last post.

Obviously, what I want to do is to replace reading the sample file with a full-on protocol communication with the test server. But I can't just do that. I need the flexibility that comes from being able to change the file (most urgently, I need to put my credentials in it). I also need to get my credentials from somewhere. But the shortest path to making a meaningful change is to replace reading the sample XML with generating it.

Having read the documentation in Transaction.pdf, I now feel I have a better understanding of how this whole thing works. This is not a CT600-specific service, but can handle many government forms. So there is a wrapper (an "envelope" that needs to be assembled) around the actual submission.

And, as I suspected before from my brief overview, it also has methods to check on the status of submissions, to obtain the outstanding submissions and to delete "dead" submissions (presumably once you have extracted some kind of authentication code).

(For those who haven't heard this before, I'm really bad at reading almost all kind of documents "in the abstract": specs, descriptions, APIs etc. One of the reasons this blog exists is as my attempt to build a "bridge" between those dry, disconnected documents and the real world; as much for me to look back at as to educate anyone else. Remember that I don't know what I'm doing here, and freely admit it. People often don't believe this, because once I have understood something, I can often quote chapter and verse of the spec in question.)

I mentioned in generating the GnuCash files that I was really unhappy with the class structures that I ended up with and questioned if I could do something more general to represent XML and have my classes be what I would expect them to be with a convenience method AsXML attached. So this time I'm going to try that. I think that also works really well with this whole "envelope"/"body" model here.

Let's get started. I'm not going to list out the whole of the sample message here (it's 126 lines), but I am going to show you excerpts. In particular, I'm not going to worry about the Body portion for now but just try and get an "Envelope" to be accepted.

Changing the Test

So we need to start by updating the test to replace the call to read the file with one that generates the file. Simples:
func TestTheMinimalFileWorks(t *testing.T) {
    send, err := submission.Generate()
    if err != nil {
        log.Fatalf("error generating xml file: %v", err)
    }

    cli := &http.Client{}
    resp, err := cli.Post("https://test-transaction-engine.tax.service.gov.uk/submission", "application/x-binary", send)
    if err != nil {
        log.Fatalf("error posting xml file: %v", err)
    }
    body, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatalf("error reading response: %v", err)
    }
    msg := string(body)
    fmt.Printf("%s", msg)
    if strings.Contains(msg, "GovTalkErrors") {
        t.Fatalf("there was a GovTalkErrors block")
    }
}

CT600_START_GENERATION:accounts/internal/ct600/submission/submission_test.go

Obviously, we also need to provide an implementation of Generate:
package submission

import (
    "bytes"
    "encoding/xml"
    "fmt"
    "io"

    "github.com/gmmapowell/ignorance/accounts/internal/ct600/govtalk"
)

func Generate() (io.Reader, error) {
    msg := govtalk.MakeGovTalk()
    bs, err := xml.MarshalIndent(msg.AsXML(), "", "  ")
    if err != nil {
        return nil, err
    }
    fmt.Printf("%s", string(bs))
    return bytes.NewReader(bs), nil
}

CT600_START_GENERATION:accounts/internal/ct600/submission/generate.go

It seems that using completely general XML structures simply doesn't allow attributes to be rendered properly. Given how important attributes are in XML, this surprises me, but I dug into the code and I still couldn't make any way to make it work. I left my attempt in genxml so that you can check my work, but this is the code I came up with:
package genxml

import (
    "encoding/xml"
)

type Element struct {
    XMLName    xml.Name
    Attributes `xml:",innerxml"`
    Elements
}

type Attributes any
type Elements []any

type GTAttrs struct {
    XMLNS string `xml:"xmlns,attr"`
}

func LocalElement(name string) *Element {
    return &Element{XMLName: xml.Name{Local: name}, Attributes: GTAttrs{XMLNS: "http://www.govtalk.gov.uk/CM/envelope"}}
}

CT600_START_GENERATION:accounts/internal/ct600/genxml/xml.go

and this is the output it gives me:
<GovTalkMessage>
  <Attributes xmlns="http://www.govtalk.gov.uk/CM/envelope"></Attributes>
</GovTalkMessage>
So the whole thing is a non-starter.

But I am going to enforce a two-tier AsXML structure, I'm just going to have to implement 15,000 classes all of which look very similar (although I am going to use a general one if it doesn't have any attributes, which seems to be quite common in this protocol). Somewhat related, I discovered this tool out there on the internet, which will generate Go classes from XML. Useful if you want to go down a path of having one class per Element type.

So this is what I have now:
package govtalk

import "encoding/xml"

type GovTalk interface {
    AsXML() any
}

func MakeGovTalk() GovTalk {
    return &GovTalkMessage{}
}

type GovTalkMessage struct {
}

func (gtm *GovTalkMessage) AsXML() any {
    ret := &GovTalkMessageXML{}
    ret.XMLNS = "http://www.govtalk.gov.uk/CM/envelope"
    ret.XSI = "http://www.w3.org/2001/XMLSchema-instance"
    env := &SimpleElement{XMLName: xml.Name{Local: "EnvelopeVersion"}}
    env.Text = "2.0"
    ret.Elements = append(ret.Elements, env)
    return ret
}

CT600_START_GENERATION:accounts/internal/ct600/govtalk/govtalk.go

The GovTalkMessage looks boring, but that's just because we haven't really done anything yet. This is where most of the logic is going to (eventually) go. In the meantime, all it really does is turn around and create a couple of XML Elements. GovTalkMessageXML is a "specific" struct because it needs the xmlns attributes, while SimpleElement is the general element I am going to use whenever we don't need that.

Both of these are defined in the xml.go file:
package govtalk

import "encoding/xml"

type SimpleElement struct {
    XMLName xml.Name
    Text    string `xml:",chardata"`
    Elements
}

type GovTalkMessageXML struct {
    XMLName xml.Name `xml:"GovTalkMessage"`
    XMLNS   string   `xml:"xmlns,attr"`
    XSI     string   `xml:"xmlns:xsi,attr"`
    Elements
}

type Elements []any

CT600_START_GENERATION:accounts/internal/ct600/govtalk/xml.go

OK, I'm going to dive into a bunnyhole now and generate a whole bunch of those things.

Here we go:
func (gtm *GovTalkMessage) AsXML() any {
    ret := &GovTalkMessageXML{}
    ret.XMLNS = "http://www.govtalk.gov.uk/CM/envelope"
    ret.XSI = "http://www.w3.org/2001/XMLSchema-instance"
    env := &SimpleElement{XMLName: xml.Name{Local: "EnvelopeVersion"}}
    env.Text = "2.0"
    header := &SimpleElement{XMLName: xml.Name{Local: "Header"}}
    msgDetails := &SimpleElement{XMLName: xml.Name{Local: "MessageDetails"}}
    test := &SimpleElement{XMLName: xml.Name{Local: "GatewayTest"}}
    test.Text = "1"
    msgDetails.Elements = append(msgDetails.Elements, test)
    sndrDetails := &SimpleElement{XMLName: xml.Name{Local: "SenderDetails"}}
    header.Elements = append(header.Elements, msgDetails, sndrDetails)
    gtDetails := &SimpleElement{XMLName: xml.Name{Local: "GovTalkDetails"}}
    keys := &SimpleElement{XMLName: xml.Name{Local: "Keys"}}
    utr := &KeyElement{Type: "UTR", Value: "8596148860"}
    keys.Elements = append(gtDetails.Elements, utr)
    gtDetails.Elements = append(gtDetails.Elements, keys)
    body := &SimpleElement{XMLName: xml.Name{Local: "Body"}}
    ret.Elements = append(ret.Elements, env, header, gtDetails, body)
    return ret
}

CT600_XML_ELEMENTS:accounts/internal/ct600/govtalk/govtalk.go

And over in the xml.go file, I defined the struct Key since it needs a Type attribute:
type KeyElement struct {
    XMLName xml.Name `xml:"Key"`
    Type    string   `xml:"Type,attr"`
    Value   string   `xml:",chardata"`
}

CT600_XML_ELEMENTS:accounts/internal/ct600/govtalk/xml.go

That all "works" (as far as it goes), but while I don't know about you, I don't like how much repetition it has in it. Let's fix that.
func (gtm *GovTalkMessage) AsXML() any {
    env := ElementWithText("EnvelopeVersion", "2.0")
    test := ElementWithText("GatewayTest", "1")
    msgDetails := ElementWithNesting("MessageDetails", test)
    sndrDetails := ElementWithNesting("SenderDetails")
    header := ElementWithNesting("Header", msgDetails, sndrDetails)
    utr := Key("UTR", "8596148860")
    keys := ElementWithNesting("Keys", utr)
    gtDetails := ElementWithNesting("GovTalkDetails", keys)
    body := ElementWithNesting("Body")
    return MakeGovTalkMessage(env, header, gtDetails, body)
}

CT600_CLEANED_XML:accounts/internal/ct600/govtalk/govtalk.go

Which obviously requires a little bit of help and support in terms of the functions we are using:
func ElementWithNesting(tag string, elts ...any) *SimpleElement {
    env := &SimpleElement{XMLName: xml.Name{Local: tag}}
    env.Elements = elts
    return env
}

func ElementWithText(tag, value string, elts ...any) *SimpleElement {
    env := &SimpleElement{XMLName: xml.Name{Local: tag}}
    env.Text = value
    env.Elements = elts
    return env
}

func MakeGovTalkMessage(nesting ...any) *GovTalkMessageXML {
    return &GovTalkMessageXML{
        XMLNS:    "http://www.govtalk.gov.uk/CM/envelope",
        XSI:      "http://www.w3.org/2001/XMLSchema-instance",
        Elements: nesting,
    }
}

func Key(ty, value string) *KeyElement {
    return &KeyElement{Type: ty, Value: value}
}

CT600_CLEANED_XML:accounts/internal/ct600/govtalk/xml.go

OK, now that is a bit tidier, I'm going back into my bunnyhole.

This is what I ended up with. I have used a mixture of styles here, from moving the body off into its own function, having things in temporary variables, and doing things inline. I may not have made the best tradeoffs, but it seemed a good idea at the time.
func (gtm *GovTalkMessage) AsXML() any {
    env := ElementWithText("EnvelopeVersion", "2.0")
    msgDetails := ElementWithNesting(
        "MessageDetails",
        ElementWithText("Class", "HMRC-CT-CT600"),
        ElementWithText("Qualifier", "request"),
        ElementWithText("Function", "submit"),
        ElementWithText("Transformation", "XML"),
        ElementWithText("GatewayTest", "1"),
    )
    sndrDetails := ElementWithNesting(
        "SenderDetails",
        ElementWithNesting(
            "IDAuthentication",
            ElementWithText("SenderID", "Provided by the SDST"),
            ElementWithNesting(
                "Authentication",
                ElementWithText("Method", "clear"),
                ElementWithText("Role", "Principal"),
                ElementWithText("Value", "Provided by the SDST"),
            ),
        ),
    )
    gtDetails := ElementWithNesting(
        "GovTalkDetails",
        ElementWithNesting("Keys", Key("UTR", "8596148860")),
        ElementWithNesting("TargetDetails", ElementWithText("Organisation", "HMRC")),
        ElementWithNesting(
            "ChannelRouting",
            ElementWithNesting(
                "Channel",
                ElementWithText("URI", "Vendor ID"),
                ElementWithText("Product", "Product Details"),
                ElementWithText("Version", "Version #"),
            ),
        ),
    )
    return MakeGovTalkMessage(env,
        ElementWithNesting("Header", msgDetails, sndrDetails),
        gtDetails,
        gtm.makeBody())
}

func (gtm *GovTalkMessage) makeBody() any {
    body := ElementWithNesting("Body")
    return &body
}

CT600_HEADER:accounts/internal/ct600/govtalk/govtalk.go

This leaves us with the "envelope" portion much as we had it in the sample, and the body just empty. And when we submit it, it again tells us we can't authenticate. Let's tackle that next time.

Conclusion

We faffed around with XML formats until I found a reasonable compromise between what I really wanted and what was possible. I then stubbed out the header of the message as best as I could with static data. This leaves us in a good position next time to fill in the data more accurately.

Wednesday, May 28, 2025

Working with the Government


So, we've figured out what we can and can't do: it's time to refocus.

I started this project thinking that it would be easy to submit accounts from GnuCash, and made getting my accounts into GnuCash the goal, but this turns out to have been a false trail. The actual goal, of course, is to communicate my company and tax information to the government. In the process of doing what I have so far, I have most of the information I must need in a combination of a configuration file and a spreadsheet: it's just a question of working with the government services to give that to them. Reviewing the information in the ct600 project, it now seems that doing so directly is the way to go.

Now that I'm aware of the Government developer services, let's get started! (Have I learnt nothing from having just gone down a false trail? Not really; only what I already knew which is to try and build a small spike first before committing too many resources down a path.)

Signing up to be a developer

In order to do anything with government services, we need to sign up as a developer on the government's developer hub.

This takes a while and has a few steps, but is a fairly well-described process. There appear to be two different ways of signing up (self-registration and something that requires you to email them). It was initially unclear to me where CT600 lies, but in the end I had to email the SDST (software development support team) anyway. Since I did this over the holiday weekend it all took a while, but eventually I managed to get set up.

CT600 Service

In the meantime, the CT600 service comes with a "local test service": a Java application that you can download and run locally, so let's try that.

For the time being, I am assuming that I can submit both my tax return and my accounts together, thus saving me from looking into how to do my accounts separately. If this turns out to be wrong, I will have to go back and do that.

Installing the Local Test Service

So I downloaded the LTS8.0.zip file and unzipped it. Under there is a directory called HMRCTools with a document call InstallationGuide.doc, which has an explanation of what to do:
$ export LTS_HOME=$/HMRCTools/LTS
$ chmod +x HMRCTools/LTS/RunLTSStandalone.sh
$ HMRCTools/LTS/RunLTSStandalone.sh
Audit Logging to: /Users/gareth/Projects/CT600/HMRCTools/LTS/logs/audit.log
***********************************
Schematron Validator - version: 1.80
Service = http://www.govtalk.gov.uk/CM/envelope|http://www.govtalk.gov.uk/taxation/RIMDemo/07-08/1
***********************************
[main] INFO org.eclipse.jetty.util.log - Logging initialized @771ms to org.eclipse.jetty.util.log.Slf4jLog
Configuring LTS server; Configuration file: /Users/gareth/Projects/CT600/HMRCTools/LTS/resources/config/UserConfigurable/LTSConfig.xml
Starting LTS server...
[main] INFO org.eclipse.jetty.server.Server - jetty-9.4.31.v20200723; built: 2020-07-23T17:57:36.812Z; git: 450ba27947e13e66baa8cd1ce7e85a4461cacc1d; jvm 17.0.2+8-86
[main] INFO org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0
[main] INFO org.eclipse.jetty.server.session - No SessionScavenger set, using defaults
[main] INFO org.eclipse.jetty.server.session - node0 Scavenging every 600000ms
[main] INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@28b576a9{Local Test Service,/LTS,file:///Users/gareth/Projects/CT600/HMRCTools/LTS/Webapp/,AVAILABLE}{Webapp/}
[main] INFO org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@5066d65f{HTTP/1.1, (http/1.1)}{0.0.0.0:5665}
[main] INFO org.eclipse.jetty.server.Server - Started @1090ms
LTS version: 8.0 April 2025
LTS server running.
OK. So far, so good. How do we use this thing?

This is an "XML" service, so we need to submit an XML file. No surprise, there. As it happens, the government has a number of sample files on their website. So we should be able to take one of these and "submit" it to our local test service.

I downloaded the first one, without attachments, and saved it into "ct600/no-attach.xml".

The website has a lot of documents on it, which are arranged in some kind of logical order, but I found it difficult to find what I wanted; possibly because I wanted a nice easy developer introduction and all of the documents are specifications. This seemed to be the closest thing to what I wanted, and gives a URL for submitting the documents to the test or live service at the government.

Going back to the LTS Installation Guide (part of the ZIP file), it seems that the correct URL to use locally is http://localhost:5665/LTS, so let's try that:
$ curl http://localhost:5665/LTS
Not wildly interesting. It turns out that this is intended to be used from a browser and only to validate the input, not to be a serious local test service. OK. Nothing seems to be going my way at the moment.

Even so, bullet 3 of section 4.4 allows for "HTTP POST", so let's try that:
curl -Hcontent-type:application/x-binary -d @accounts/ct600/no-attach.xml http://localhost:5665/LTS/LTSPostServlet
Request Failed : Error initiating/processing the incoming message.
uk.gov.hmrc.aspire.esps.validator.config.ServiceNotRegisteredException: http://www.govtalk.gov.uk/CM/envelope|http://www.govtalk.gov.uk/taxation/CT/5
OK, it may be that I am just missing something, but for now, I'm going to assume that "this local thing does not apply to me" and try and move on to working with the sandbox test service.

Setting up a Project

Having received my credentials from the government (no, I'm not going to share them here) and having chosen and set up a password, I'm ready to go with the online service. It seems that there are two separate services, the "live" one and a "sandbox" one, which is excellent, and they have given me some credentials (UTR and passwords) to work with the sandbox environment, so let's see if we can get started there. The UTR appears to be the same one that is given in the sample on their website (I am not surprised by this).

Section 6 (Appendix D) of the Transaction.pdf document describes the "external test service" and gives the URL for working with that: https://test-transaction-engine.tax.service.gov.uk/submission (as opposed to the live version, https://transaction-engine.tax.service.gov.uk/submission, the difference seeming to be that it has test- on the front). Let's just see what happens if we curl our document to that:
$ curl -Hcontent-type:application/x-binary -d @$HOME/Projects/IgnoranceBlog/accounts/ct600/no-attach.xml https://test-transaction-engine.tax.service.gov.uk/submission
<?xml version="1.0" encoding="UTF-8"?>
<GovTalkMessage xmlns="http://www.govtalk.gov.uk/CM/envelope">
    <EnvelopeVersion>2.0</EnvelopeVersion>
    <Header>
        <MessageDetails>
            <Class>UndefinedClass</Class>
            <Qualifier>error</Qualifier>
            <Function>submit</Function>
            <TransactionID></TransactionID>
            <CorrelationID></CorrelationID>
            <ResponseEndPoint PollInterval="10">https://test-transaction-engine.tax.service.gov.uk/submission</ResponseEndPoint>
            <GatewayTimestamp>2025-05-28T06:58:45.075</GatewayTimestamp>
        </MessageDetails>
        <SenderDetails/>
    </Header>
    <GovTalkDetails>
        <Keys/>
        <GovTalkErrors>
            <Error>
                <RaisedBy>Gateway</RaisedBy>
                <Number>1046</Number>
                <Type>fatal</Type>
                <Text>Authentication Failure. The supplied user credentials failed validation for the requested service.</Text>
                <Location/>
            </Error>
        </GovTalkErrors>
    </GovTalkDetails>
    <Body/>
</GovTalkMessage>
Very good. The user credentials failed validation. Just what I wanted.

I'm going to describe that as a "failing test case". The only problem is that it isn't a failing automated test case. So can we do that?

At the moment, I see two challenges ahead of me:
  • Work with the government system
  • Build the actual XML files I want to send
It's worth pointing out that a lot of what I have read seems to imply it's just a question of "uploading an XML file" but the Transaction.pdf file makes it seem more like a protocol with a number of submissions on the go at once and the ability to store things, check up on things and (presumably) delete things, which seems more logically complete. For now, I'll be happy with being able to get code working which fails to upload the initial document because of failed validation. Then tomorrow, we can come back and generate that and have it fail. Then we can move on to having a file that passes validation.

A Failing Test

So I'm just going to do the minimum to duplicate the above as a failing test in Go. From there, I'll work to extract everything else. (Yes, I know this is not how I normally work here, but I feel a bit stuck at the moment.)
package submission_test

import (
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strings"
    "testing"
)

func TestTheMinimalFileWorks(t *testing.T) {
    file, err := os.Open("../../../ct600/no-attach.xml")
    if err != nil {
        log.Fatalf("error reading xml file: %v", err)
    }

    cli := &http.Client{}
    resp, err := cli.Post("https://test-transaction-engine.tax.service.gov.uk/submission", "application/x-binary", file)
    if err != nil {
        log.Fatalf("error posting xml file: %v", err)
    }
    body, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatalf("error reading response: %v", err)
    }
    msg := string(body)
    fmt.Printf("%s", msg)
    if strings.Contains(msg, "GovTalkErrors") {
        t.Fatalf("there was a GovTalkErrors block")
    }
}

CT600_SUBMIT_FAILS:accounts/internal/ct600/submit/submission_test.go

And when I run this, it fails as I would expect to, but not before repeating the above errors.

Note that I had to specifically look for that error block in the message; the actual communication was very polite and the service returns 200, not the 401 or 403 you might have expected.

Conclusion

I feel like I have been wandering around in the desert, but I now have a direction to go. I have managed to work through the various levels of documentation and have had a "successful" interaction with the test server, and even managed to automate that.

At this point, there are any number of directions we could go with the next step. Come back next time to find out what actually happens!

Monday, May 26, 2025

iXBRL Reporter


The whole reason I started going down this path was because last year when I was submitting accounts, it asked me if I had prepared accounts in iXBRL format. I hadn't of course (I hadn't "prepared" accounts as such at all), but it was enough to put me in research mode and "ah, yes, GnuCash, iXBRL, all sounds reasonable ... maybe if I have some time next year".

So here we are. As I google these things, I find a lot of things are deprecated and I should use something else instead. Anyway, I ended up here and started following the instructions.

Installing

So this is a python project - not my favourite, it has to be said - but I'm only using it, not trying to debug it. Let's see what happens.
$ pip3 install git+https://github.com/cybermaggedon/ixbrl-reporter
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/cybermaggedon/ixbrl-reporter
  Cloning https://github.com/cybermaggedon/ixbrl-reporter to /private/var/folders/sh/l33974n42dd168fg2dvmp600000gn/T/pip-req-build-xz6jyyo
  Running command git clone -q https://github.com/cybermaggedon/ixbrl-reporter /private/var/folders/sh/l33974n42dd168fg2dvmp600000gn/T/pip-req-build-xz6jyyo
  Resolved https://github.com/cybermaggedon/ixbrl-reporter to commit 0751f8b8dcba2765866350c3b4a95e29f7a8aae9
Collecting requests
  Downloading requests-2.32.3-py3-none-any.whl (64 kB)
       ████████████████████████████████| 64 kB 2.9 MB/s
Collecting lxml
  Downloading lxml-5.4.0-cp39-cp39-macosx109x8664.whl (4.4 MB)
       ████████████████████████████████| 4.4 MB 6.7 MB/s
Collecting piecash
  Downloading piecash-1.2.1.tar.gz (769 kB)
       ████████████████████████████████| 769 kB 7.4 MB/s
Collecting PyYAML
  Downloading PyYAML-6.0.2-cp39-cp39-macosx109x8664.whl (184 kB)
       ████████████████████████████████| 184 kB 8.7 MB/s
Collecting SQLAlchemy<1.5,>=1.0
  Downloading SQLAlchemy-1.4.54-cp39-cp39-macosx120x8664.whl (1.6 MB)
       ████████████████████████████████| 1.6 MB 8.9 MB/s
Collecting SQLAlchemy-Utils!=0.36.8
  Downloading SQLAlchemy_Utils-0.41.2-py3-none-any.whl (93 kB)
       ████████████████████████████████| 93 kB 3.2 MB/s
Collecting pytz
  Downloading pytz-2025.2-py2.py3-none-any.whl (509 kB)
       ████████████████████████████████| 509 kB 8.7 MB/s
Collecting tzlocal
  Downloading tzlocal-5.3.1-py3-none-any.whl (18 kB)
Collecting click
  Downloading click-8.1.8-py3-none-any.whl (98 kB)
       ████████████████████████████████| 98 kB 7.0 MB/s
Collecting greenlet!=0.4.17
  Downloading greenlet-3.2.1-cp39-cp39-macosx110_universal2.whl (266 kB)
       ████████████████████████████████| 266 kB 7.4 MB/s
Collecting charset-normalizer<4,>=2
  Downloading charsetnormalizer-3.4.1-cp39-cp39-macosx109universal2.whl (197 kB)
       ████████████████████████████████| 197 kB 7.7 MB/s
Collecting certifi>=2017.4.17
  Downloading certifi-2025.4.26-py3-none-any.whl (159 kB)
       ████████████████████████████████| 159 kB 7.5 MB/s
Collecting urllib3<3,>=1.21.1
  Downloading urllib3-2.4.0-py3-none-any.whl (128 kB)
       ████████████████████████████████| 128 kB 8.3 MB/s
Collecting idna<4,>=2.5
  Downloading idna-3.10-py3-none-any.whl (70 kB)
       ████████████████████████████████| 70 kB 7.2 MB/s
Building wheels for collected packages: ixbrl-reporter, piecash
  Building wheel for ixbrl-reporter (setup.py) ... done
  Created wheel for ixbrl-reporter: filename=ixbrl_reporter-1.1.2-py3-none-any.whl size=57431 sha256=104e9d08018ffde594cb2d2ebb936c0fd85cd496a3ae36892dcfa51d360f52c7
  Stored in directory: /private/var/folders/sh/l33_974n42dd168fg2dvmp600000gn/T/pip-ephem-wheel-cache-bym6oal8/wheels/87/10/24/2a312b06e54c26399ccbf71aa489be34f43f3f765f12e5c4ad
  Building wheel for piecash (setup.py) ... done
  Created wheel for piecash: filename=piecash-1.2.1-py3-none-any.whl size=64622 sha256=fd57e4966f05c66c2583990ea6102512bcef9100d64d54ba95460f1ab06ba189
  Stored in directory: /Users/gareth/Library/Caches/pip/wheels/a0/a2/06/33a71b4c4007e99f46b4e375da1ae745383c8c8495b701a392
Successfully built ixbrl-reporter piecash
Installing collected packages: greenlet, SQLAlchemy, urllib3, tzlocal, SQLAlchemy-Utils, pytz, idna, click, charset-normalizer, certifi, requests, PyYAML, piecash, lxml, ixbrl-reporter
  WARNING: The script normalizer is installed in '/Users/gareth/Library/Python/3.9/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script piecash is installed in '/Users/gareth/Library/Python/3.9/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed PyYAML-6.0.2 SQLAlchemy-1.4.54 SQLAlchemy-Utils-0.41.2 certifi-2025.4.26 charset-normalizer-3.4.1 click-8.1.8 greenlet-3.2.1 idna-3.10 ixbrl-reporter-1.1.2 lxml-5.4.0 piecash-1.2.1 pytz-2025.2 requests-2.32.3 tzlocal-5.3.1 urllib3-2.4.0
WARNING: You are using pip version 21.2.4; however, version 25.1 is available.
You should consider upgrading via the '/Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip' command.
It told me to upgrade pip, so I did.
$ /Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages (21.2.4)
Collecting pip
  Downloading pip-25.1-py3-none-any.whl (1.8 MB)
       ████████████████████████████████| 1.8 MB 5.2 MB/s
Installing collected packages: pip
  WARNING: The scripts pip, pip3 and pip3.9 are installed in '/Users/gareth/Library/Python/3.9/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-25.1
WARNING: You are using pip version 21.2.4; however, version 25.1 is available.
You should consider upgrading via the '/Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip' command.
This also wants me to upgrade pip, but since I've just done it, I have to assume that's a bug.

Generating Accounts

Following on in the instructions, it says I will need to clone the repository to obtain some configuration files.

I can do that.
$ git clone https://github.com/cybermaggedon/ixbrl-reporter
Cloning into 'ixbrl-reporter'...
remote: Enumerating objects: 1924, done.
remote: Counting objects: 100% (225/225), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 1924 (delta 205), reused 198 (delta 195), pack-reused 1699 (from 1)
Receiving objects: 100% (1924/1924), 1.62 MiB | 7.15 MiB/s, done.
Resolving deltas: 100% (1401/1401), done.
And then I should be able to generate my accounts from that. Starting with what they have given me, let's try that.
$ cd ixbrl-reporter/
$ ~/Library/Python/3.9/bin/ixbrl-reporter config.yaml report ixbrl > accts.html
$ open accts.html
OK, yes that works. But everything starts to descend into hell from here.

Customizing the Reports

I was under the impression that the configuration here was all about customizing the form you wanted to generate. And I thought that once I had my data in GnuCash, everything would be fine. Oh, I was so wrong.

Apparently all of your company data needs to be put in the configuration files.

And you need to customize the years that you want the reporting to be for as well. At some level, this is not unreasonable, but it feels like "the year you want" is a different level of option to "this is the way my accounts are structured".

A whole bunch of other stuff which in my mind is "accounting" related, is also in the configuration files.

Also, reading from GnuCash files directly is not supported on a Mac. I could go to Linux, but apparently it is possible to use a CSV file. I can export from GnuCash to CSV, but now I'm asking myself why I put so much effort into generating GnuCash XML files if I'm only using that to generate CSV files.

But at the end of the day, with much hacking, I managed to generate both a set of accounts and a CT600 in iXBRL format that looked valid and similar to what I had last year. I don't really feel that it's worth even recording the kinds of things that I did, because this is not really a scalable way to proceed.

Submitting the Reports

As I said at the top of this episode, the whole reason that I started going down this path was because I thought I could now submit these forms directly. The accounts, yes; the tax forms, no. Ah.

It turns out that there is a further program, ct600 which is capable of doing something like this and working with the HMRC API. But there are two sentences in its description which stop me going there:
  • "This probably won't do what you want"
  • "The first step is to use ct600 to create what I am calling a 'form values' file"
OK, so let's not do that.

Conclusion

I feel I have walked around four sides of a block trying to automatically submit my accounts and CT600 and have done endless transformations to get ... nowhere.

Well, not quite nowhere. Further on, the ct600 README suggests doing this:

"See Company Tax Returns and Corporation Tax online: support for software developers."

Which makes me realize that while I have gone down a false path, not all the work I have done needs to be thrown away.

I want to be clear that I haven't spent "that much" time doing all this - it's less than a week since I started and I've been working anything but full-time on it. But I have certainly wasted time trying to integrate with GnuCash, because it is a false trail. But it's worth pointing out that while I tried to take risks early, I should have used my "hand-built" GnuCash file to try out the ixbrl-reporter and ct600 scripts sooner. Had I done that, I would have reached this point without having to write all the GnuCash XML code that I found so painful.

Next Steps

On the bright side, I have done the following things that I consider useful:
  • Collected together all my company accounts and records in a single Google Sheet;
  • Used GnuCash as a model of how to build that as a reasonable set of "Account"s that can be manipulated into accounts and corporation tax filing;
  • Built code to read the configuration and spreadsheet, and to calculate account values.
The only time I have wasted has been on the output to GnuCash and the taffing with the ixbrl.

Fortunately, I have until Dec 31 to submit my accounts and, if all else fails, I can submit them "by hand" this year (although apparently that facility is going away next year).

I now need to go ahead and understand what it means to use the Government Gateway to submit these documents. More reading ahead!

Sunday, May 25, 2025

Generating Reports

Back when I was younger, there was a computer game called "Company Director" or something like that, and I was fascinated by reports such as Profit and Loss and Balance Sheet. More so, in fact, than actually running my company and selling refrigerators (I have to admit most of the details escape me now).

As I have grown up, these things have become more natural, but I still have a sense of wonder about all sorts of results in number theory, and the balancing of a balance sheet still seems somehow magical for the same reason that 12x12 = 9x16 seems magical.

So, as much as I will tell you that I am doing this so that I can see what I think the results "should be" before I start trying to use someone else's tool, as much as anything, I want to do this for my own amusement.

Setting the Stage

The data as we have stored it in the Gnucash structure is perfect for rendering XML, but not ideal for any other purpose. To repurpose it, we need to choose between interpreting it as it stands and saving it in an alternative format.

In doing this, it seems to me easy enough to find the transactions in the AccountBook, and we already have a pointer to that in Gnucash, so I will do that, but the actual transaction "splits" are so messed up that there is almost no hope, so I will add specific entries to the transaction to hold those:
type Transaction struct {
    XMLName      xml.Name `xml:"gnc:transaction"`
    Version      string   `xml:"version,attr"`
    accountGuids map[string]string
    srcAcct      *TransactionItem
    destAcct     *TransactionItem
    credit       *AccountCredit
    debit        *AccountDebit
    Elements
}

GNUCASH_REPORT_1:accounts/internal/gnucash/writer/gnucash.go

type AccountAction struct {
    When   DateInfo
    Acct   string
    Amount Money
}

type AccountCredit struct {
    AccountAction
}

type AccountDebit struct {
    AccountAction
}

GNUCASH_REPORT_1:accounts/internal/gnucash/writer/gnucash.go

(Each transaction consists of ONE credit and ONE debit, which are a "matching pair".)

And, during the processing of the transaction data, I will just store the relevant information here:
func (g *Gnucash) Transact(date DateInfo, description string, src string, dest string, amount Money) *Transaction {
    tx := &Transaction{Version: "2.0.0", accountGuids: g.accountGuids}
    guid := newGuid()
...
    g.book.Elements = append(g.book.Elements, tx)

    // for regurgitation
    tx.credit = &AccountCredit{AccountAction: AccountAction{When: date, Acct: dest, Amount: amount}}
    tx.debit = &AccountDebit{AccountAction: AccountAction{When: date, Acct: src, Amount: amount}}

    return tx
}

GNUCASH_REPORT_1:accounts/internal/gnucash/writer/gnucash.go

And then it is possible to ask for all these credits and debits back:
func (g *Gnucash) Regurgitate(rcvr TxReceiver) {
    for _, x := range g.book.Elements {
        tx, ok := x.(*Transaction)
        if ok {
            rcvr.Credit(*tx.credit)
            rcvr.Debit(*tx.debit)
        }
    }
}

GNUCASH_REPORT_1:accounts/internal/gnucash/writer/gnucash.go

The loop iterates over all the book elements (counts, accounts, transactions, etc), and finds the ones that are transactions. For each transaction, it then alerts the receiver about the credit and debit portions of the transaction.

A receiver can be any struct implementing TxReceiver:
type TxReceiver interface {
    Credit(c AccountCredit)
    Debit(c AccountDebit)
}

GNUCASH_REPORT_1:accounts/internal/gnucash/writer/deliver.go

Capturing the Data

Having made sure we have a usable source of account data, we can now create a data collector for our reports. Basically, for asset and liability accounts, we want to look at all transactions from the beginning of time until the end of the reporting period; for income and expense accounts, we want to look at transactions within the reporting period.

At the top level, we create the data collector, telling it what the reporting period is, and ask the writer to regurgitate all the transactions and deliver them to the data collector. We can do this for each of the years we are interested in.
func AccountsPipeline(conf *config.Configuration) {
    w := writer.MakeWriter(conf)
    accts := accounts.MakeAccounts(conf, w)
    sheets.ReadSpreadsheet(conf, accts)

    for yr := 2019; yr <= 2019; yr++ {
        r := reporter.NewReporter(yr)
        accts.Regurgitate(r)
    }
}

GNUCASH_REPORT_1:accounts/internal/gnucash/pipeline/accounts.go

I want to be clear that all of this is for only my own benefit. I mean, while it's quite likely that a company has the calendar year as its fiscal year, how likely is it that you want to report from 2019 onwards? Let's be clear: this is a blog about doing interesting and different things and experimenting; I am not talking about production code. If I were, we would be knee-deep in unit tests right now.

And we can implement a first cut of the reporter as follows:
package reporter

import (
    "fmt"

    "github.com/gmmapowell/ignorance/accounts/internal/gnucash/writer"
)

type reporter struct {
    Year int
}

func (r *reporter) Credit(c writer.AccountCredit) {
    fmt.Printf("%s: Credit %s with %s\n", c.When.JustDate(), c.Acct, c.Amount.String())
}

func (r *reporter) Debit(c writer.AccountDebit) {
    fmt.Printf("%s: Debit %s by %s\n", c.When.JustDate(), c.Acct, c.Amount.String())
}

func NewReporter(yr int) writer.TxReceiver {
    return &reporter{Year: yr}
}

GNUCASH_REPORT_1:accounts/internal/gnucash/reporter/reporter.go

So far, so good. I can run this and all the transactions come out with a credit and a debit for each one.

Now we need to start summarizing things. The problem here is that all the different accounts behave differently. For Income and Expense accounts, we only want to collect data for a single year; for the others we want to collect data from the beginning of time. And for Bank and Expense accounts we want to treat credits as positive; for Income, Equity and Liability accounts we want to count debits as positive. This is really hard to explain and understand, but when you apply the rules, everything comes out looking right (again, just like number theory).

So we need to know about what accounts exist and what their types are. In other words, we need the configuration. We can pass the configuration to NewReporter and get that to configure the reporter:
func NewReporter(conf *config.Configuration, yr int) writer.TxReceiver {
    ret := reporter{Year: yr, Accounts: make(map[string]account)}
    ret.Configure(conf.Accounts)
    return &ret
}

GNUCASH_REPORT_2:accounts/internal/gnucash/reporter/reporter.go

And then provide the Configure function along with the data slots to store the configuration:
type reporter struct {
    Accounts map[string]account
    Year     int
}

func (r *reporter) Configure(accts []config.Account) {
    for _, acc := range accts {
        log.Printf("Configuring %s of %s\n", acc.Name, acc.Type)
        r.Accounts[acc.Name] = makeAccount(r.Year, acc.Type)
        r.Configure(acc.Accounts)
    }
}

GNUCASH_REPORT_2:accounts/internal/gnucash/reporter/reporter.go

account is an interface, backed by the poorly named actor struct (I will rename this if I can think of a good name, but this is just basically AccountImpl), which is in a new file.
package reporter

import (
    "fmt"

    "github.com/gmmapowell/ignorance/accounts/internal/gnucash/writer"
)

type account interface {
    Credit(date writer.DateInfo, amount writer.Money)
    Debit(date writer.DateInfo, amount writer.Money)
}

type actor struct {
    debitEffect, creditEffect int
    justYear                  bool
    year                      int
    balance                   writer.Money
}

func (a *actor) Credit(date writer.DateInfo, amount writer.Money) {
    if a.justYear && date.Year != a.year {
        return
    }
    a.balance.Incorporate(a.creditEffect, amount)
    fmt.Printf("Collect credit %s => %s\n", amount, a.balance)
}

func (a *actor) Debit(date writer.DateInfo, amount writer.Money) {
    if a.justYear && date.Year != a.year {
        return
    }
    a.balance.Incorporate(a.debitEffect, amount)
    fmt.Printf("Collect debit %s => %s\n", amount, a.balance)
}

func makeAccount(yr int, ty string) account {
    switch ty {
    case "ASSET":
        fallthrough
    case "BANK":
        return &actor{debitEffect: -1, creditEffect: 1, justYear: false, year: yr}
    case "EXPENSE":
        return &actor{debitEffect: -1, creditEffect: 1, justYear: true, year: yr}
    case "INCOME":
        return &actor{debitEffect: 1, creditEffect: -1, justYear: true, year: yr}
    case "EQUITY":
        return &actor{debitEffect: 1, creditEffect: -1, justYear: false, year: yr}
    case "LIABILITY":
        return &actor{debitEffect: 1, creditEffect: -1, justYear: false, year: yr}
    default:
        panic("no such account type: " + ty)
    }
}
What I'm doing here is defining exactly one "type" of account, to avoid code duplication, but then configuring it with properties to indicate how credits and debits should affect it, along with the "final" year it should collect and whether it should just collect that year. I think I would rather have all these things be polymorphic functions, but this is only an experiment (and, in any case, I'm not sure I know how to polymorphic functions well in Go).

The code for Credit and Debit is very similar and should probably be combined, but is so simple that it didn't seem worth it. Each of them checks if the transaction was for "this year" if the account has been configured this way (note: there is a bug here which I spotted and fixed later in that even the other accounts should only accept transactions prior to this year ending). It then asks the balance to incorporate the money with the appropriate "effect".

That is provided in the Money class:
func (m *Money) Incorporate(effect int, other Money) {
    m.Units += effect * other.Units
    m.Subunits += effect * other.Subunits
    m.Normalize()
}

func (m *Money) Normalize() {
    for m.Subunits < 0 {
        m.Subunits += 100
        m.Units -= 1
    }
    for m.Subunits >= 100 {
        m.Subunits -= 100
        m.Units += 1
    }
}

func (m Money) GCCredit() string {
    return fmt.Sprintf("%d/100", 100*m.Units+m.Subunits)
}

func (m Money) GCDebit() string {
    return "-" + m.GCCredit()
}

func (m Money) String() string {
    return fmt.Sprintf("£%d.%02d", m.Units, m.Subunits)
}

GNUCASH_REPORT_2:accounts/internal/gnucash/writer/money.go

This is mainly boring complexity - the sort of thing that should be wrapped in half-a-dozen unit tests, but ...

Generating Reports

Then it's just a question of asking the data collector to spit out the reports when all the data for a given reporting period has been collected.
func AccountsPipeline(conf *config.Configuration) {
    w := writer.MakeWriter(conf)
    accts := accounts.MakeAccounts(conf, w)
    sheets.ReadSpreadsheet(conf, accts)

    for yr := 2018; yr <= 2025; yr++ {
        r := reporter.NewReporter(conf, yr)
        accts.Regurgitate(r)
        r.ProfitLoss(yr)
        r.BalanceSheet(yr)
    }
}

GNUCASH_REPORT_3:accounts/internal/gnucash/pipeline/accounts.go

These reports are generated inside the reporter (obviously):
func (r *reporter) ProfitLoss(yr int) {
    fmt.Printf("Profit and Loss for %d\n", yr)
    var total writer.Money
    for name, acc := range r.Accounts {
        if acc.IsPL() && acc.HasBalance() {
            fmt.Printf("  %s: %s\n", name, acc.Balance())
            total.Incorporate(acc.PLEffect(), acc.Balance())
        }
    }
    fmt.Printf("Net Profit/Loss: %s\n", total)
}

func (r *reporter) BalanceSheet(yr int) {
    fmt.Printf("Balance Sheet at %d-12-31\n", yr)
    for name, acc := range r.Accounts {
        if !acc.IsPL() && acc.HasBalance() {
            fmt.Printf("  %s: %s\n", name, acc.Balance())
        }
    }
}

GNUCASH_REPORT_3:accounts/internal/gnucash/reporter/reporter.go

And depend, in turn, on additional methods to recover the fields of the accounts.
func (a *actor) HasBalance() bool {
    return a.balance.IsNonZero()
}

func (a *actor) IsPL() bool {
    return a.justYear
}

func (a *actor) Balance() writer.Money {
    return a.balance
}

func (a *actor) PLEffect() int {
    return -a.creditEffect
}

GNUCASH_REPORT_3:accounts/internal/gnucash/reporter/accounts.go

While I'm here, I fixed the bug with not checking for transactions past the end of the reporting period:
func (a *actor) Credit(date writer.DateInfo, amount writer.Money) {
    if a.justYear && date.Year != a.year {
        return
    }
    if date.Year > a.year {
        return
    }
    a.balance.Incorporate(a.creditEffect, amount)
    // fmt.Printf("Collect credit %s => %s\n", amount, a.balance)
}

func (a *actor) Debit(date writer.DateInfo, amount writer.Money) {
    if a.justYear && date.Year != a.year {
        return
    }
    if date.Year > a.year {
        return
    }
    a.balance.Incorporate(a.debitEffect, amount)
    // fmt.Printf("Collect debit %s => %s\n", amount, a.balance)
}

GNUCASH_REPORT_3:accounts/internal/gnucash/reporter/accounts.go

Conclusion

OK, so I have gone on a little detour and figured out that I can calculate all the Profit & Loss and Balance Sheet data for myself without interacting with GnuCash. But I still need to actually submit accounts, and it's my belief that for that, I need iXBRL and the ixbrl-reporter.