So, we've figured out what we can and can't do: it's time to refocus.
I started this project thinking that it would be easy to submit accounts from GnuCash, and made getting my accounts into GnuCash the goal, but this turns out to have been a false trail. The actual goal, of course, is to communicate my company and tax information to the government. In the process of doing what I have so far, I have most of the information I must need in a combination of a configuration file and a spreadsheet: it's just a question of working with the government services to give that to them. Reviewing the information in the ct600 project, it now seems that doing so directly is the way to go.
Now that I'm aware of the Government developer services, let's get started! (Have I learnt nothing from having just gone down a false trail? Not really; only what I already knew which is to try and build a small spike first before committing too many resources down a path.)
Signing up to be a developer
In order to do anything with government services, we need to sign up as a developer on the government's developer hub.This takes a while and has a few steps, but is a fairly well-described process. There appear to be two different ways of signing up (self-registration and something that requires you to email them). It was initially unclear to me where CT600 lies, but in the end I had to email the SDST (software development support team) anyway. Since I did this over the holiday weekend it all took a while, but eventually I managed to get set up.
CT600 Service
In the meantime, the CT600 service comes with a "local test service": a Java application that you can download and run locally, so let's try that.For the time being, I am assuming that I can submit both my tax return and my accounts together, thus saving me from looking into how to do my accounts separately. If this turns out to be wrong, I will have to go back and do that.
Installing the Local Test Service
So I downloaded the LTS8.0.zip file and unzipped it. Under there is a directory called HMRCTools with a document call InstallationGuide.doc, which has an explanation of what to do:$ export LTS_HOME=$/HMRCTools/LTSOK. So far, so good. How do we use this thing?
$ chmod +x HMRCTools/LTS/RunLTSStandalone.sh
$ HMRCTools/LTS/RunLTSStandalone.sh
Audit Logging to: /Users/gareth/Projects/CT600/HMRCTools/LTS/logs/audit.log
***********************************
Schematron Validator - version: 1.80
Service = http://www.govtalk.gov.uk/CM/envelope|http://www.govtalk.gov.uk/taxation/RIMDemo/07-08/1
***********************************
[main] INFO org.eclipse.jetty.util.log - Logging initialized @771ms to org.eclipse.jetty.util.log.Slf4jLog
Configuring LTS server; Configuration file: /Users/gareth/Projects/CT600/HMRCTools/LTS/resources/config/UserConfigurable/LTSConfig.xml
Starting LTS server...
[main] INFO org.eclipse.jetty.server.Server - jetty-9.4.31.v20200723; built: 2020-07-23T17:57:36.812Z; git: 450ba27947e13e66baa8cd1ce7e85a4461cacc1d; jvm 17.0.2+8-86
[main] INFO org.eclipse.jetty.server.session - DefaultSessionIdManager workerName=node0
[main] INFO org.eclipse.jetty.server.session - No SessionScavenger set, using defaults
[main] INFO org.eclipse.jetty.server.session - node0 Scavenging every 600000ms
[main] INFO org.eclipse.jetty.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@28b576a9{Local Test Service,/LTS,file:///Users/gareth/Projects/CT600/HMRCTools/LTS/Webapp/,AVAILABLE}{Webapp/}
[main] INFO org.eclipse.jetty.server.AbstractConnector - Started ServerConnector@5066d65f{HTTP/1.1, (http/1.1)}{0.0.0.0:5665}
[main] INFO org.eclipse.jetty.server.Server - Started @1090ms
LTS version: 8.0 April 2025
LTS server running.
This is an "XML" service, so we need to submit an XML file. No surprise, there. As it happens, the government has a number of sample files on their website. So we should be able to take one of these and "submit" it to our local test service.
I downloaded the first one, without attachments, and saved it into "ct600/no-attach.xml".
The website has a lot of documents on it, which are arranged in some kind of logical order, but I found it difficult to find what I wanted; possibly because I wanted a nice easy developer introduction and all of the documents are specifications. This seemed to be the closest thing to what I wanted, and gives a URL for submitting the documents to the test or live service at the government.
Going back to the LTS Installation Guide (part of the ZIP file), it seems that the correct URL to use locally is http://localhost:5665/LTS, so let's try that:
$ curl http://localhost:5665/LTSNot wildly interesting. It turns out that this is intended to be used from a browser and only to validate the input, not to be a serious local test service. OK. Nothing seems to be going my way at the moment.
Even so, bullet 3 of section 4.4 allows for "HTTP POST", so let's try that:
curl -Hcontent-type:application/x-binary -d @accounts/ct600/no-attach.xml http://localhost:5665/LTS/LTSPostServletOK, it may be that I am just missing something, but for now, I'm going to assume that "this local thing does not apply to me" and try and move on to working with the sandbox test service.
Request Failed : Error initiating/processing the incoming message.
uk.gov.hmrc.aspire.esps.validator.config.ServiceNotRegisteredException: http://www.govtalk.gov.uk/CM/envelope|http://www.govtalk.gov.uk/taxation/CT/5
Setting up a Project
Having received my credentials from the government (no, I'm not going to share them here) and having chosen and set up a password, I'm ready to go with the online service. It seems that there are two separate services, the "live" one and a "sandbox" one, which is excellent, and they have given me some credentials (UTR and passwords) to work with the sandbox environment, so let's see if we can get started there. The UTR appears to be the same one that is given in the sample on their website (I am not surprised by this).Section 6 (Appendix D) of the Transaction.pdf document describes the "external test service" and gives the URL for working with that: https://test-transaction-engine.tax.service.gov.uk/submission (as opposed to the live version, https://transaction-engine.tax.service.gov.uk/submission, the difference seeming to be that it has test- on the front). Let's just see what happens if we curl our document to that:
$ curl -Hcontent-type:application/x-binary -d @$HOME/Projects/IgnoranceBlog/accounts/ct600/no-attach.xml https://test-transaction-engine.tax.service.gov.uk/submissionVery good. The user credentials failed validation. Just what I wanted.
<?xml version="1.0" encoding="UTF-8"?>
<GovTalkMessage xmlns="http://www.govtalk.gov.uk/CM/envelope">
<EnvelopeVersion>2.0</EnvelopeVersion>
<Header>
<MessageDetails>
<Class>UndefinedClass</Class>
<Qualifier>error</Qualifier>
<Function>submit</Function>
<TransactionID></TransactionID>
<CorrelationID></CorrelationID>
<ResponseEndPoint PollInterval="10">https://test-transaction-engine.tax.service.gov.uk/submission</ResponseEndPoint>
<GatewayTimestamp>2025-05-28T06:58:45.075</GatewayTimestamp>
</MessageDetails>
<SenderDetails/>
</Header>
<GovTalkDetails>
<Keys/>
<GovTalkErrors>
<Error>
<RaisedBy>Gateway</RaisedBy>
<Number>1046</Number>
<Type>fatal</Type>
<Text>Authentication Failure. The supplied user credentials failed validation for the requested service.</Text>
<Location/>
</Error>
</GovTalkErrors>
</GovTalkDetails>
<Body/>
</GovTalkMessage>
I'm going to describe that as a "failing test case". The only problem is that it isn't a failing automated test case. So can we do that?
At the moment, I see two challenges ahead of me:
- Work with the government system
- Build the actual XML files I want to send
A Failing Test
So I'm just going to do the minimum to duplicate the above as a failing test in Go. From there, I'll work to extract everything else. (Yes, I know this is not how I normally work here, but I feel a bit stuck at the moment.)package submission_test
import (
"fmt"
"io"
"log"
"net/http"
"os"
"strings"
"testing"
)
func TestTheMinimalFileWorks(t *testing.T) {
file, err := os.Open("../../../ct600/no-attach.xml")
if err != nil {
log.Fatalf("error reading xml file: %v", err)
}
cli := &http.Client{}
resp, err := cli.Post("https://test-transaction-engine.tax.service.gov.uk/submission", "application/x-binary", file)
if err != nil {
log.Fatalf("error posting xml file: %v", err)
}
body, err := io.ReadAll(resp.Body)
if err != nil {
log.Fatalf("error reading response: %v", err)
}
msg := string(body)
fmt.Printf("%s", msg)
if strings.Contains(msg, "GovTalkErrors") {
t.Fatalf("there was a GovTalkErrors block")
}
}
CT600_SUBMIT_FAILS:accounts/internal/ct600/submit/submission_test.go
And when I run this, it fails as I would expect to, but not before repeating the above errors.Note that I had to specifically look for that error block in the message; the actual communication was very polite and the service returns 200, not the 401 or 403 you might have expected.
Conclusion
I feel like I have been wandering around in the desert, but I now have a direction to go. I have managed to work through the various levels of documentation and have had a "successful" interaction with the test server, and even managed to automate that.At this point, there are any number of directions we could go with the next step. Come back next time to find out what actually happens!
No comments:
Post a Comment