Tyrel's Blog

Code, Flying, Tech, Automation

Dec 16, 2022

Advent of Code 2022 + End of Year Updates

Advent of Code this year is kicking my butt so I haven't been doing any tech blogging really lately. If you want to follow my progress, I think I might be done as of day 15 - This one seems to be a traveling salesman/knapsack problem related. Here's my repo: https://gitea.tyrel.dev/tyrel/advent-of-code/src/branch/main/2022/python.

I'm not on the computer that runs it, but I've been spending a lot of time playing with Apple's System7 in the BasiliskII emulator. Might have some fun projects with that coming up, but wanted to do some more learning before I start anything. So I have been going through a course on 6052 Assembly programming for the NES, and I'm about 73% done with that, it's really great!

It's By Gustavo Pezzi at Pikuma, if "oldschool" programming floats your boat then I definitely recommend it. It's all programming through making roms with CC65/CA65 assembler, and using FCEUX to see your results, super neat.

I've been picking up some more Go work at work. My current team is sort of disbanding so I'm going to be moving away from doing just Python. It's been a year since I've done Go stuff, since I left Tidelift, so I'm really rusty.

Speaking of Rust, I was trying to do Advent of code in Rust also, and made it TWO whole days in Rust. It's still on my bucket of stuff to learn, but my free time seems to be running out lately, and I have a lot of things on my plate to get done.

 · · ·  python  adventofcode  6502  assembly  rust  go

Oct 17, 2022

Comparing Go GORM and SQLX

Django ORM - My History

I'm not the best SQL developer, I know it's one of my weak points. My history is I did php/mysql from the early 2000s until college. In college I didn't really focus on the Database courses, the class selection didn't have many database course. The one Data Warehousing course I had available, I missed out on because I was in England doing a study abroad program that semester. My first job out of college was a Python/Django company - and that directed my next eight years of work.

Django, if you are unaware, is a MVC framework that ships with a really great ORM. You can do about 95% of your database queries automatically by using the ORM.

entry, created = Entry.objects.get_or_create(headline="blah blah blah")
q = Entry.objects.filter(headline__startswith="What")
q = q.filter(pub_date__lte=datetime.date.today())
q = q.exclude(body_text__icontains="food")

Above are some samples from the DjangoDocs. But enough about Django.

My Requirements

Recently at my job I was given a little bit of leeway on a project. My team is sort of dissolving and merging in with another team who already does Go. My Go history is building a CLI tool for the two last years of my previous job. I had never directly interacted with a database from Go yet. I wanted to spin up a REST API (I chose Go+Gin for that based on forty five seconds of Googling) and talk to a database.


Being that I come from the Django (and a few years of ActiveRecord) land, I reached immediately for an ORM, I chose GORM. If you want to skip directly to the source, check out https://gitea.tyrel.dev/tyrel/go-webservice-gin. Full design disclosure: I followed a couple of blog posts in order to develop this, so it is in the form explictly decided upon by the logrocket blog post and may not be the most efficient way to organize the module.

In order to instantiate a model definition, it's pretty easy. What I did is make a new package called models and inside made a file for my Album.

type Album struct {
      ID     string  `json:"id" gorm:"primary_key"`
      Title  string  `json:"title"`
      Artist string  `json:"artist"`
      Price  float64 `json:"price"`

This tracks with how I would do the same for any other kind of struct in Go, so this wasn't too difficult to do. What was kind of annoying was that I had to also make some structs for Creating the album and Updating the Album, this felt like duplicated effort that might have been better served with some composition.

I would have structured the controllers differently, but that may be a Gin thing and how it takes points to functions, vs pointers to receivers on a struct. Not specific to GORM. Each of the controller functions were bound to a gin.Context pointer, rather than receivers on an AlbumController struct.

The FindAlbum controller was simple:

func FindAlbum(c *gin.Context) {
      var album models.Album
      if err := models.DB.Where("id = ?", c.Param("id")).First(&album).Error; err != nil {
              c.JSON(http.StatusBadRequest, gin.H{"error": "Record not found!"})
      c.JSON(http.StatusOK, gin.H{"data": album})

Which will take in a /:id path parameter, and the GORM part of this is the third line there.

models.DB.Where("id = ?", c.Param("id")).First(&album).Error

To run a select, you chain a Where on the DB (which is the connection here) and it will build up your query. If you want to do joins, this is where you would chain .Joins etc... You then pass in your album variable to bind the result to the struct, and if there's no errors, you continue on with the bound variable. Error handling is standard Go logic, if err != nil etc and then pass that into your API of choice (Gin here) error handler.

This was really easy to set up, and if you want to get a slice back you just use DB.Find instead, and bind to a slice of those structs.

var albums []models.Album


SQLX is a bit different, as it's not an ORM, it's extensions in Go to query with SQL, but still a good pattern for abstracting away your SQL to some dark corner of the app and not inline everywhere. For this I didn't follow someone's blog post — I had a grasp on how to use Gin pretty okay by now and essentially copied someone elses repo with my existing model. gin-sqlx-crud.

This one set up a bit wider of a structure, with deeper nested packages. Inside my internal folder there's controllers, forms, models/sql, and server. I'll only bother describing the models package here, as thats the SQLX part of it.

In the models/album.go file, there's your standard struct here, but this time its bound to db not json, I didn't look too deep yet but I presume that also forces the columns to set the json name.

type Album struct {
  ID     int64   `db:"id"`
  Title  string  `db:"title"`
  Artist string  `db:"artist"`
  Price  float64 `db:"price"`

An interface to make a service, and a receiver are made for applying the CreateAlbum form (in another package) which sets the form name and json name in it.

func (a *Album) ApplyForm(form *forms.CreateAlbum) {
  a.ID = *form.ID
  a.Title = *form.Title
  a.Artist = *form.Artist
  a.Price = *form.Price

So there's the receiver action I wanted at least!

Nested inside the models/sql/album.go file and package, is all of the Receiver code for the service. I'll just comment the smallest one, as that gets my point across. Here is where the main part of GORM/SQLX differ - raw SQL shows up.

func (s *AlbumService) GetAll() (*[]models2.Album, error) {
      q := `SELECT * FROM albums;`

      var output []models2.Album
      err := s.conn.Select(&output, q)
      // Replace the SQL error with our own error type.
      if err == sql.ErrNoRows {
              return nil, models2.ErrNotFound
      } else if err != nil {
              return nil, err
      } else {
              return &output, nil

This will return a slice of Albums - but if you notice on the second line, you have to write your own queries. A little bit more in control of how things happen, with a SELECT * ... vs the gorm DB.Find style.

To me this feels more like using pymysql, in fact its a very similar process. (SEE NOTE BELOW) You use the service.connection.Get and pass in what you want the output bound to, the string query, and any parameters. This feels kind of backwards to me - I'd much rather have the order be: query, bound, parameters, but thats what they decided for their order.


Overall, both were pretty easy to set up for one model. Given the choice I would look at who the source code is written for. If you're someone who knows a lot of SQL, then SQLX is fine. If you like abstractions, and more of a "Code as Query" style, then GORM is probably the best of these two options.

I will point out that GORM does more than just "query and insert" there is migration, logging, locking, dry run mode, and more. If you want to have a full fledged system, that might be a little heavy, then GORM is the right choice.

SQLX is great if what you care about is marshalling, and a very quick integration into any existing codebase.


I sent this blog post to my friend Andrey and he mentioned that I was incorrect with my comparision of sqlx to pymysql. To put it in a python metaphor, "sqlx is like using urllib3, gorm is like using something that generates a bunch of requests code for you. Using pymysql is like using tcp to do a REST request." Sqlx is more akin to SqlAlchemy core vs using SqlAlchemy orm. Sqlx is just some slight extensions over database/sql. As the sort of equivalent to pymysql in Go is database/sql/driver from the stdlib.

 · · ·  go  sql  python  gorm  sqlx

Nov 11, 2021

Postmortem of a fun couple bugs

Story at my previous job:

Tieg: Hey Tyrel, I can't run invoke sign 5555, can you help with this?

This is How my night started last night at 10pm. My coworker Tieg did some work on our CLI project and was trying to release the latest version. We use invoke to run our code signing and deployment scripts, so I thought it was just a quick "oh maybe I screwed up some python!" fix. It wasn't.

I spent from 10:30 until 1:30am this morning going through and looking into why Tieg wasn't able to sign the code. The first thing I did was re-run the build on CircleCI, which had the same error, so hey! at least it was reproducible. The problem was that in our Makefile scripts we run tidelift version > tidelift-cli.version and then upload that to our deployment directories, but this was failing for some reason. We let clients download this file to see what the latest version is and then our CLI tool has the ability to selfupdate (except on homebrew) to pull this latest version if you're outdated.

Once I knew what was failing, I was able to use CircleCI's ssh commands and log in, and see what happened, but I was getting some other errors. I was seeing some problems with dbus-launch so I promptly (mistakenly) yelled to the void on twitter about dubs-launch. Well would you know it, I may have mentioned before, but I work with Havoc Pennington.

Havoc Pennington: fortunately I wrote dbus-launch so may be able to tell you something, unfortunately it was like 15 years ago

Pumped about this new revelation, I started looking at our keychain dependency, because I thought the issue was there as that's the only thing that uses dbus on Linux. Then we decided (Havoc Pointed it out) that it was a red herring, and maybe the problem was elsewhere. I at least learned a bit about dbus and what it does, but not enough to really talk about it to any detail.

Would you know it, the problem was elsewhere. Tieg was running dtruss and saw that one time it was checking his /etc/hosts file when it was failing, and another time it was NOT, which was passing. Then pointed out a 50ms lookup to our download.tidelift.com host.

Tieg then found Issue 49517 this issue where someone mentions that Go 1.17.3 was failing them for net/http calls, but not the right way.

It turns out, that it wasn't the keyring stuff, it wasn't the technically the version calls that failed. What was happening is every command starts with a check to https://download.tidelift.com/cli/tidelift-cli.version which we then compare to the current running version, if it's different and outdated, we then say "you can run selfupdate!". What fails is that call to download.tidelift.com, because of compiling with go1.17.3 and a context canceled due to stream cleanup I guess?

Okay so we need to downgrade to Go 1.17.2 to fix this. Last night in my trying, I noticed that our CircleCI config was using circle/golang:1.16 as its docker image, which has been superseded by cimg/go:1.16.x style of images. But I ran into some problems with that while upgrading to cimg/go:1.17.x. The problem was due to the image having different permissions, so I couldn't write to the same directories that when Mike wrote our config.yml file, worked properly.

Tieg and I did a paired zoom chat and finished this up by cutting out all the testing/scanning stuff in our config files, and just getting down to the Build and Deploy steps. Found ANOTHER bug that Build seems to run as the circleci user, but Deploy was running as root. So in the build working_directory setting, using a ~/go/tidelift/cli path, worked. But when we restored the saved cache to Deploy, it still put it in /home/circle/go/tidelift/cli, but then the working_directory of ~/go/tidelift/cli was relative to /root/. What a nightmare!

All tildes expanded to /home/circleci/go/tidelift/cli set, Makefile hacks undone, (removing windows+darwin+arm64 builds from your scripts during testing makes things A LOT faster!) and PR Merged, we were ready to roll.

I merged the PR, we cut a new version of TideliftCLI 1.2.5, updated the changelog and signed sealed delivered a new version which uses Go 1.17.2, writes the proper tidelift-cli.version file in deployment steps, and we were ready to ROCK!

That was fun day. Now it's time to write some rspec tests.

 · · ·  Go  dbus  bugs