[Jodavaho.io]

This is the first in a series of posts describing how I’m keeping data about my self, environment, work, and goals. You can read the whole series here

As part of my journalling habit, I used to keep excellent metrics about myself. From the year 2014 to about 2019, I recorded almost everything that I felt was important. This included workouts, spending, and time spent in a variety of areas, such as grad study. I logged each “good” awake hour. This allowed me to generate reports like the following:

Quick sidebar: You can imagine two ways to get into this. First, you set some goals, then you set up a way to track progress. Complicated goals implies Complicated tracking / reporting. Second way: You just start tracking and make tweaks along the way when you notice too much time or money spent on something. I took the second track back then, and want to again, as otherwise it’s analysis paralysis all the way down.

I’d rather measure and maximize effort (delta-progress / delta-t) than attempt to reach an arbitrarily designed goal.

Let’s break down what I want to track:

1. Finances
2. Time spent with Family, improving health, or on engineering-type tasks
3. Accomplishments vs goals

Let’s just go over finances first, the rest deserve their own posts.

Finances, current system

For the last 1.5 years, I’ve been using ledger to run financial reports. Long story short, it’s a pain to keep the input files synchronized with our actual expenses. It requires manual duplication removal, manual classification, and manual report running. It is nice at quickly querying multiple reports or data sets, but I found that we really only needed an answer for one question: Where did our money go this month?

To get data from financial instutions, I log in, download a custom-date-range csv file, and then run a bash script to convert the csv file to one readable by ledger, then I go transaction-by-transaction:

• Is this transaction already in the ledger?
• Is the classification for this transaction correct? If not, type “Expenses:Blah” to classify it.
• Is the source/ destination account sync’d? If it’s a transfer, I need to go over to the other account’s ledger, and make sure it’s not double counted.

I found:

• This is error prone. The double-entry accounting method made it great to know that I had captured all transactions (sum=0), but that wasn’t my primary concern. Instead it created a lot of double-counting, since account A would credit account B with a transfer, but account B would have an incoming transaction from A, and debit A for that amount.
• This is tedious. By my current count, we have around 1800 transactions in the ledger for the last 4 months, 100% of which I had to input, classify, and check by hand.
• It’s incomprehensible to my wife. The ASCII-column output is wonderful in a hacker sense, but makes no sense to my wife and harms her strong sense of supervisory role. (insert laugh track)

Finances, proposed system

Step 1: Simplify the report generation

Let’s simplify. Ledger is used to parse incoming data and generate interactive reports. The input format is remarkably human readable (look at the example at ledger-cli.org, which tells me the target audience is a somewhat-non-programmer bunch. I am not that audience.

Given that I’m already downloading csv files, let’s just write a jupyter notebook, and use something like viola to render a website that we can view.

Step 2: Simplify the classification of transactions

We’re habitual, most transactions are recurring (Utilities, subscriptions, other bills), or from a few major outlets (Target, Amazon, etc). So, we can get an 80/20 solution by automatically adding a classification column with a few simple rules. The remaining 20% can be tracked as an “other” expense group, and the largest or most frequent charges in that group can easily be plotted with, say, above reporting.

Later, I’d like to teach a neural network to do this form me.

Step 3: Automate the retrieval of data from financial institutions

This is the most complicated step. I’d like a weekly-fetched set of transactions, either for the whole year so I can run the whole batch at once, or to fetch the week’s transactions and append to an ongoing data store.

I’ve looked into ofxtools (readthedocs.org/ofxtools), but some of my financial institutions are either blocking bot access, or it was too difficult for me to configure. This is a solved problem: Plaid will allow developers to fetch transactions, for a modest per-user fee.

Step 4: Preserve ledger output

This is simple, since we import the transaction data, we can just print it out in a ledger-ready format for later use.

Put together

If successful, we’ll have some cron jobs on a tiny computer somewhere in the house:

1. Fetch year’s financial data from our handful of accounts
2. Go through and classify most of them using hand-crafted rules, and group the rest for manual inspection
3. Render a report with a ton of fancy interactive graphs, and a month-to-month and year-to-year status report.
4. Push that to an internally-accessible website.
5. Archive the data in an additional ledger-compatible format

I’ll update with how well we’re progressing soon.

You can read the whole series here

ChatGPT Summary:

The author discusses their intention to track various aspects of their life, including finances, family time, health improvement, and engineering tasks. They highlight their current system of using ledger-cli for financial reports but find it cumbersome. They propose a simplified system involving Jupyter notebooks, automated classification of transactions, data retrieval from financial institutions, and generating interactive reports. The goal is to create a user-friendly and efficient method for tracking and analyzing personal metrics.