After that digression, I'm now ready to pick up where I left off.
I've now submitted multiple entries to the [test] system. I believe they automatically prune this on a regular (daily?) basis, so I shouldn't be all that worried, but I do tend to think about these things. But, in any case, we will need to be able to monitor this sooner or later, so why not sooner?
List Current Submissions
So, now that we have a command line tool, I'm going to add a "--list" option to it that says "don't submit anything but list all of the current submissions". As I start to do this, I realize that the "main" function is getting too long, so I push off the arguments parsing into its own file.func main() {
conf, mode, err := config.ParseArguments(os.Args[1:])
if err != nil {
fmt.Printf("error parsing arguments: %v\n", err)
return
}
switch mode {
case config.SUBMIT_MODE:
err := submission.Submit(conf)
if err != nil {
fmt.Printf("submission failed: %v\n", err)
return
}
case config.LIST_MODE:
panic("unimplemented")
default:
log.Fatalf("there is no handler for mode %s\n", mode)
}
}
CT600_ARGS:accounts/cmd/ct600/main.go
package config
import (
"fmt"
)
const SUBMIT_MODE = "--submit"
const LIST_MODE = "--list"
func ParseArguments(args []string) (*Config, string, error) {
mode := ""
conf := MakeBlankConfig()
for _, f := range args {
switch f {
case LIST_MODE:
fallthrough
case SUBMIT_MODE:
if mode != "" {
return nil, "", fmt.Errorf("cannot specify %s and %s", mode, f)
}
mode = f
default:
err := IncludeConfig(conf, f)
if err != nil {
return nil, "", fmt.Errorf("failed to read config %s: %v", f, err)
}
}
}
if mode == "" {
mode = SUBMIT_MODE // default is submit
}
return conf, mode, nil
}
CT600_ARGS:accounts/internal/ct600/config/args.go
Going back to the transaction guide, I can see in section 3.9 that there is a "DATA_REQUEST" command. This is what I want to use for "--list". This is not that different from submitting, so I'm going to put it in the same package, but call the function List.func main() {
conf, mode, err := config.ParseArguments(os.Args[1:])
if err != nil {
fmt.Printf("error parsing arguments: %v\n", err)
return
}
switch mode {
case config.SUBMIT_MODE:
err := submission.Submit(conf)
if err != nil {
fmt.Printf("submission failed: %v\n", err)
return
}
case config.LIST_MODE:
err := submission.List(conf)
if err != nil {
fmt.Printf("submission failed: %v\n", err)
return
}
default:
log.Fatalf("there is no handler for mode %s\n", mode)
}
}
CT600_LIST:accounts/cmd/ct600/main.go
Obviously I want to share as much code as possible, so I've duplicated the minimal amount of submit.go and called it list.go:package submission
import (
"github.com/gmmapowell/ignorance/accounts/internal/ct600/config"
"github.com/gmmapowell/ignorance/accounts/internal/ct600/govtalk"
)
func List(conf *config.Config) error {
pollOptions := &govtalk.EnvelopeOptions{Qualifier: "request", Function: "list", SendCorrelationID: true, IncludeSender: true}
send, err := Generate(conf, pollOptions)
if err != nil {
return err
}
return transmit(send)
}
CT600_LIST:accounts/internal/ct600/submission/list.go
And introduced an options class to say how I would like the GovTalk message to encode itself:package govtalk
type EnvelopeOptions struct {
Qualifier string
Function string
SendCorrelationID bool
CorrelationID string
IncludeSender bool
IncludeKeys bool
IncludeBody bool
}
CT600_LIST:accounts/internal/ct600/govtalk/options.go
The idea here is that the format of the messages is very, very similar and what we want to do is tweak and configure them to look the way the individual messages need to look. There are more options here than we are currently using, for the simple reason that I started off down the bunnyhole of trying to implement the POLL command before realizing that was for something else.And this leads to lots of "if-then" code like this:
func (gtm *GovTalkMessage) AsXML() any {
env := ElementWithText("EnvelopeVersion", "2.0")
var corrId *SimpleElement
if gtm.opts.SendCorrelationID {
corrId = ElementWithText("CorrelationID", gtm.opts.CorrelationID)
}
msgDetails := ElementWithNesting(
"MessageDetails",
ElementWithText("Class", "HMRC-CT-CT600"),
ElementWithText("Qualifier", gtm.opts.Qualifier),
ElementWithText("Function", gtm.opts.Function),
corrId,
ElementWithText("Transformation", "XML"),
ElementWithText("GatewayTest", "1"),
)
CT600_LIST:accounts/internal/ct600/govtalk/govtalk.go
And finally, I went back and fixed up the implementation of submit.go to specify the appropriate options:package submission
import (
"github.com/gmmapowell/ignorance/accounts/internal/ct600/config"
"github.com/gmmapowell/ignorance/accounts/internal/ct600/govtalk"
)
func Submit(conf *config.Config) error {
submitOptions := &govtalk.EnvelopeOptions{Qualifier: "request", Function: "submit", IncludeSender: true, IncludeKeys: true, IncludeBody: true}
send, err := Generate(conf, submitOptions)
if err != nil {
return err
}
return transmit(send)
}
CT600_LIST:accounts/internal/ct600/submission/submit.go
And then I get output back like this:<GovTalkMessage>That's the information I want, but it's all a bit ugly, so let's turn it into a normal list.
...
<Body>
<StatusReport>
<SenderID/>
<StartTimeStamp></StartTimeStamp>
<EndTimeStamp></EndTimeStamp>
<StatusRecord>
<TimeStamp>08/07/2025 11:41:01</TimeStamp>
<CorrelationID>ED9FFE0EE5344728998DB2F7662E1046</CorrelationID>
<TransactionID/>
<Status>SUBMISSION_ERROR</Status>
</StatusRecord>
<StatusRecord>
<TimeStamp>08/07/2025 11:39:02</TimeStamp>
<CorrelationID>48195E6313C647528497B3C92070BF0F</CorrelationID>
<TransactionID/>
<Status>SUBMISSION_ERROR</Status>
</StatusRecord>
...
</StatusReport>
</Body>
</GovTalkMessage>
My first thought was to call Unmarshal, but it turns out with the way I have encoded my XML, Unmarshal cannot figure out how to turn the XML back into structs. That being the case, I need to look for an alternative. Digging through the documentation, it seems that there is an xml.Decoder class which I can use to scan the XML and analyze it. It is the work of five minutes (well, closer to half an hour, but you know what I mean!) to produce a list.
func List(conf *config.Config) error {
pollOptions := &govtalk.EnvelopeOptions{Qualifier: "request", Function: "list", SendCorrelationID: true, IncludeSender: true}
send, err := Generate(conf, pollOptions)
if err != nil {
return err
}
msg, err := transmit(send)
if err != nil {
return err
}
var summary []*Record
decoder := xml.NewDecoder(bytes.NewReader(msg))
var r *Record
var data string
for tok, err := decoder.Token(); err == nil; tok, err = decoder.Token() {
switch tok := tok.(type) {
case xml.StartElement:
if tok.Name.Local == "StatusRecord" {
r = &Record{}
summary = append(summary, r)
}
case xml.EndElement:
if tok.Name.Local == "StatusRecord" {
r = nil
} else if r != nil {
switch tok.Name.Local {
case "TimeStamp":
log.Printf("parsing time %s", data)
r.time, err = time.Parse("02/01/2006 15:04:05", data)
if err != nil {
log.Printf("failed to parse %s as a mm/dd/yyyy hh:mm:ss date", data)
}
case "CorrelationID":
r.corrId = data
case "Status":
r.status = data
}
}
case xml.CharData:
data = string(tok)
default:
log.Printf("%T", tok)
}
}
slices.SortFunc(summary, func(a, b *Record) int {
ct := a.time.Compare(b.time)
if ct != 0 {
return ct
}
return strings.Compare(a.corrId, b.corrId)
})
for _, r := range summary {
log.Printf("have %v %s %s\n", r.time.Format("2006-01-02 15:04:05"), r.corrId, r.status)
}
return nil
}
CT600_NEAT_LIST:accounts/internal/ct600/submission/list.go
The main loop moves through all the tokens provided by the decoder. At the StartElement of a StatusRecord, a new record is created and appended to the summary list. Each text element is captured temporarily in the variable data. And then when we reach the EndElement of the fields TimeStamp, CorrelationID and Status we transfer that into the record. When we reach the end of the StatusRecord we set the current record pointer, r, back to nil.Once we have collected all the records into summary, we can sort them and then print out all the responses in a list, one per line.
2025/07/08 18:17:26 have 2025-07-08 17:09:01 42C7C64B260440AB9E4B3C0FE6914888 SUBMISSION_ERRORNow that I can see the list clearly, it is obvious that a lot of those submissions are not mine - I wasn't even at my computer at 1709. I had previously suspected that the (test) credentials that the SDST had given me were not unique to me, but I think this confirms it. I was going to go through and delete all the submissions, but this now seems a bad idea, since they are other people's.
2025/07/08 18:17:26 have 2025-07-08 17:09:01 44BCEFBC20AB4E9D893A10C734E19337 SUBMISSION_ERROR
2025/07/08 18:17:26 have 2025-07-08 17:09:02 C770B3E731EF447AA62EB156EE2D709D SUBMISSION_ERROR
2025/07/08 18:17:26 have 2025-07-08 17:11:01 D76E609476344BD4842A535FBDD3D714 SUBMISSION_ERROR
But it does make me realize that I need to capture the CorrelationID that is generated in response to my own submissions so that I can see what is going on inside the system - and delete them selectively if I want.
I can use what amounts to the same code (only much simpler) in Submit:
func Submit(conf *config.Config) error {
submitOptions := &govtalk.EnvelopeOptions{Qualifier: "request", Function: "submit", IncludeSender: true, IncludeKeys: true, IncludeBody: true}
send, err := Generate(conf, submitOptions)
if err != nil {
return err
}
msg, err := transmit(send)
if err != nil {
return err
}
decoder := xml.NewDecoder(bytes.NewReader(msg))
var data string
for tok, err := decoder.Token(); err == nil; tok, err = decoder.Token() {
switch tok := tok.(type) {
case xml.StartElement:
case xml.EndElement:
if tok.Name.Local == "CorrelationID" {
log.Printf("CorrelationID: %s\n", data)
}
case xml.CharData:
data = string(tok)
}
}
return nil
}
CT600_CAPTURE_CORRID:accounts/internal/ct600/submission/submit.go
This now produces:CorrelationID: F1349E7293D34B4A92BBA52BC991BCACAnd we can go back to the list and see the following:
2025/07/08 18:39:02 have 2025-07-08 17:38:45 F1349E7293D34B4A92BBA52BC991BCAC SUBMISSION_ERRORIf I needed any further proof that other people are using the same account, I can see that there are two other submissions just after mine.
Also, I can now see that the submission is failing, although I don't receive an error immediately in response to my submit; I suspect that is because it is valid but not correct. The next thing to do is probably to dig in and see what I can get back by using the CorrelationID to do a poll, but that will have to wait until tomorrow.
Conclusions
This did not go at all where I was expecting.I can now see a list of submissions, but it's not a list of my submissions, so it isn't really much use to anybody. I did, however, manage to capture the CorrelationID from my submissions, so, although I will have to do more manual work than I had hoped, I can at least make some progress.
So, tomorrow I will come back to that and see if my intuition is correct: the submission is failing because it has a valid envelope but there isn't a letter inside. The moment that the CT introspector opens the envelope, it immediately notices this and complains.
No comments:
Post a Comment