On Schemas and Things

It's been over a month since I last worked on this project, so if you've been waiting, apologies. If you're reading it in retrospect, the main thing will be that there is a disconnect in that what happens here is not what I trailed at the end of the last episode. Instead, I'm going to look at the payload of the message.

The UK Government electronic systems go through a single "gateway" and what we have done so far is to assemble an "envelope" which can contain any - and possibly any number - of the supported payloads. Here, we are mainly interested in the "ct600" payload, although I am also interested in submitting my accounts at the same time (as an "attachment"). Previously, we did succeed in hacking together ixbrl accounts using the GnuCash ecosystem; but if we are going to do that again, I am going to generate something directly from the accounts.

But what are the valid fields? We have seen an "example" (and there are a couple more), but that is only really an example of the envelope. What are the fields - all the fields for the CT600 form? It would seem that they are available from the government website as a ZIP file of XSD files. So, I downloaded the latest ZIP file from the HMRC website and unpacked it. Given the relevant schema, we should be able to test any files we generate against it.

Validating Schemas

How do we do that? I'm glad you asked. There are tools that work with popular development tools such as Eclipse and VSCode. But it turns out the there is also a linux command line tool called xmllint. My heart always sinks when I read "linux tool" because the next words are usually "on a Mac, use homebrew or macports" which I always struggle with. But not so on this occasion. In fact, it's already installed on MacOS. Excellent. So now, when I have an XML, I can validate it against the schema using something like this:

$ xmllint --schema schema file

Unfortunately, when I went to take the next step I realized that the schema I had downloaded above was not a match for the sample I had previously downloaded. Specifically, I didn't have a schema for either the GovTalkMessage definition or for the CompanyTaxReturn message. I looked long and hard, but could not find them. In the end, I emailed the government's software development support team who turned around and pointed me to the correct ZIP file.

I unpacked that and found the two schemas I needed. Excellent.

Right then, let's get started.

If I've understood everything correctly, the example that I have should match the schema. That will allow us to ensure we have all our technology working. I'm going to move the two schemas I currently think I care about into a single xsd directory, and then run xmllint:

$ cp HMRC-CT-2014-v1-993/envelope-v2-0-HMRC.xsd xsd/envelope.xsd
$ cp HMRC-CT-2014-v1-993/CT-2014-v1-993.xsd xsd/ct600.xsd
$ xmllint --schema xsd/envelope.xsd --schema xsd/ct600.xsd no-attach.xml

This repeats back to me the whole contents of my input file (exactly the same? subtly changed? I'm not sure) and then outputs two error messages:

no-attach.xml:2: element GovTalkMessage: Schemas validity error : Element '{http://www.govtalk.gov.uk/CM/envelope}GovTalkMessage': No matching global declaration available for the validation root.
no-attach.xml fails to validate

I can't help feeling that this is because it can't handle two --schema arguments but didn't want to say so.

$ xmllint --schema xsd/envelope.xsd no-attach.xml

Again, this spits out the whole file and then gives the more reassuring:

​​no-attach.xml validates

I'm not sure what (if anything) I should do with the output file, but there's an option to save it to a file, so let's do that instead:

$ xmllint --schema xsd/envelope.xsd --output foo no-attach.xml

Since I'm nosy, let's check what differences there might be between the two:

$ diff -b foo no-attach.xml
1d0
< <?xml version="1.0"?>

That's just a header being added.

(without the -b, every line shows up so there are obviously some white space issues as well).

So the answer seems to be that it pretty prints it.

Now, I'm fairly sure that if I'm not using the second schema, it's not going to validate it, and changing the CompanyTaxReturn tag to be CompanyTaxReturnXX proves that. (Meanwhile, changing something like Envelope, Header or Body causes validation to fail). What can we do?

This answer on StackOverflow seems to suggest that I can create my own schema file that "imports" both. So let's try that.

<?xml version="1.0" encoding="UTF-8"?>
<schema elementFormDefault="qualified" xmlns="http://www.w3.org/2001/XMLSchema">
    <import namespace="http://www.govtalk.gov.uk/CM/envelope" schemaLocation="envelope.xsd"/>
    <import namespace="http://www.govtalk.gov.uk/taxation/CT/5" schemaLocation="ct600.xsd"/>
</schema>

CT600_RIM_SCHEMAS:accounts/ct600/xsd/importer.xsd

And then:

$ xmllint --schema xsd/importer.xsd --output foo no-attach.xml
no-attach.xml:59: element CompanyTaxReturnXX: Schemas validity error : Element '{http://www.govtalk.gov.uk/taxation/CT/5}CompanyTaxReturnXX': This element is not expected. Expected is ( {http://www.govtalk.gov.uk/taxation/CT/5}CompanyTaxReturn ).
no-attach.xml fails to validate

(Obviously, the reason for this error is that I deliberately changed the tag in the sample file. Changing it back solves the problem; similar changes prove that the envelope schema is still being checked.)

Including this automatically

I was thinking about running this on the command line, and as I went to do that, I said "I'm not going to copy and paste all that text into a file and validate it; I'll save it". And then I realized I didn't want to run the command either. So I put it all in the program. And then added a test that it gave the correct response (although I'm sure xmllint will actually fail and thus I'll see an error before that).

So I ended up with this:

func Generate(conf *config.Config) (io.Reader, error) {
    msg := govtalk.MakeGovTalk()
    msg.Identity(conf.Sender, conf.Password)
    msg.Utr(conf.Utr)
    msg.Product(conf.Vendor, conf.Product, conf.Version)
    bs, err := xml.MarshalIndent(msg.AsXML(), "", "  ")
    if err != nil {
        return nil, err
    }

    checkAgainstSchema(bs)

    return bytes.NewReader(bs), nil
}

func checkAgainstSchema(bs []byte) {
    file, err := os.Create("submit.xml")
    if err != nil {
        panic(err)
    }
    file.Write(bs)
    file.Close()
    cmd := exec.Command("xmllint", "--schema", "ct600/xsd/importer.xsd", "--output", "out.xml", "submit.xml")
    output, err := cmd.CombinedOutput()
    if err != nil {
        panic(err)
    }
    result := string(output)
    fmt.Println("---- xmllint output")
    fmt.Print(result)
    fmt.Println("----")

    if result != "submit.xml validates\n" {
        panic("xml did not validate against schema; not submitting")
    }
}

CT600_VALIDATE_SCHEMA:accounts/internal/ct600/submission/generate.go

Conclusion

When I thought about this last week, it seemed like an easy win, and an easy way back into this project that had lain abandoned for the past month or more. Sadly, it was harder work than I had expected to track down the relevant XML files, but thanks to the SDST, I'm back in the game!