Building out a Pipeline

OK, we've understood what we're dealing with, so let's get started.

I have worked with Google Docs and Sheets in the past (I am, in fact, writing this blog post as a Google Doc: it reaches you through a series of "automated" transformations) but always in Java.

Currently, I'm doing a lot in Go, and there doesn't seem any reason not to do this that way. Reading Google Sheets from Go should be supported since both come from Google, right?

As usual with Go, I am going to try and keep the cmd portion down to a minimum. So all that does is read the configuration and then pass that to the AccountsPipeline function which sets up a pipeline to read the accounts, generate a GnuCash file, create accounts from that, and then submit them.

Configuration

Since I don't want to hardcode lots of things into my code, but I also don't want 10,000 arguments, I am going to have a configuration file whose name is provided as the main argument to the program. It will define all the paths and addresses and private keys that I need. I suspect I may end up with a couple more arguments, such as the range of dates to include in the accounts. But this is good enough for now.

Since GnuCash files are in XML and I will need to be using XML to interact with that, I'm tempted to put my configuration file in XML. But I realize that even though it's functionally equivalent, I don't hate myself that much and I'll use JSON instead.

For obvious reasons, I'm not going to share either my accounts files or my configuration file here. Most of the details will become obvious as we play the game, and I will probably share excerpts of things where it is less obvious and I am not sharing private data. All the other code is where you would expect it.

Let's start by defining our initial configuration options in Go:

package config

type Configuration struct {
    Spreadsheet string
    Output      string
}

GNUCASH_CONFIG:accounts/internal/gnucash/config/config.go

We can easily write code to read that using the JSON Serializer:

package config

import (
    "encoding/json"
    "log"
    "os"
)

func ReadConfig(file string) (*Configuration, error) {
    log.Printf("reading %s\n", file)
    bs, err := os.ReadFile(file)
    if err != nil {
        return nil, err
    }
    ret := Configuration{}
    json.Unmarshal(bs, &ret)
    log.Printf("read %v\n", ret)
    return &ret, nil
}

GNUCASH_CONFIG:accounts/internal/gnucash/config/reader.go

And we can define a main program:

package main

import (
    "fmt"
    "os"

    "github.com/gmmapowell/ignorance/accounts/internal/gnucash/config"
)

func main() {
    _, err := config.ReadConfig(os.Args[1])
    if err != nil {
        fmt.Printf("failed to read config: %v\n", err)
    }
}

GNUCASH_CONFIG:accounts/cmd/accounts/main.go

Setting up a Pipeline

As you know, I'm a big fan of TDA (tell-don't-ask) programming, and this is no different. The general approach is to set up a pipeline of operations where each one takes what arguments it needs, along with a processor for "the next step". I'm going to put all of this in a function that I can call from my cmd file after reading the configuration. There are three basic steps:

• Read the spreadsheet;
• Convert into Gnucash format;
• Write to a .gnucash file.

Not that hard. The cmd file now looks like this:

package main

import (
    "fmt"
    "os"

    "github.com/gmmapowell/ignorance/accounts/internal/gnucash/config"
    "github.com/gmmapowell/ignorance/accounts/internal/gnucash/pipeline"
)

func main() {
    conf, err := config.ReadConfig(os.Args[1])
    if err == nil {
        pipeline.AccountsPipeline(conf)
    } else {
        fmt.Printf("failed to read config: %v\n", err)
    }
}

GNUCASH_OUTLINE_PIPELINE:accounts/cmd/accounts/main.go

And the pipeline is fairly simple too:

package pipeline

import (
    "github.com/gmmapowell/ignorance/accounts/internal/gnucash/accounts"
    "github.com/gmmapowell/ignorance/accounts/internal/gnucash/config"
    "github.com/gmmapowell/ignorance/accounts/internal/gnucash/sheets"
    "github.com/gmmapowell/ignorance/accounts/internal/gnucash/writer"
)

func AccountsPipeline(conf *config.Configuration) {
    writer := writer.MakeWriter(conf.Output)
    accts := accounts.MakeAccounts(writer)
    sheets.ReadSpreadsheet(conf.Spreadsheet, accts)
}

GNUCASH_OUTLINE_PIPELINE:accounts/internal/gnucash/pipeline/accounts.go

And for now, I am just going to stub out the rest of the things in packages under internal/gnucash.