New Year's eve eve, my main portable computer crashed. Rebooting to Safe mode, I could mount this MacBook's hard drive long enough to SCP the files over the network to my server, but I had to start that over twice because it fell asleep. I don't have access to rsync in the "Network Recovery Mode" it seems - maybe I should look to see if next time I can install things, it's moot now.
I spent all January 1st evening working on learning how Nix works. Of course, I started with Nix on macOS (intel at least) so I had to also learn how nix-darwin works. I have my dotfiles set up to use Nix now, rather than an INSTALL.sh file that just sets a bunch of symlinks.
I played around for a litle bit with different structures, but what I ended up with by the end of the weekend was two bash scripts (still working on makefile, env vars are being funky) one for each operating system rebuild-macos.sh and rebuild-ubuntu.sh. For now I'm only Nixifying one macOS system and two Ubuntu boxes. Avoiding it on my work m1 Mac laptop, as I don't want to have to deal with managing synthetic.conf and mount points on a work managed computer. No idea how JAMF and Nix will fight.
My filetree currently looks like (trimmed out a host and a bunch of files in home/)
Under hosts/ as you can see, I have a brew.nix file in my macbook pro's folder. This is how I install anything in homebrew. In my flake.nix for my macos folder I am using home-manager, nix-darwin, and nixpkgs. I provide this brew.nix to my darwinConfigurations and it will install anything I put in my brew nixfile.
I also have a _common directory in my hosts, this is things that are to be installed on EVERY machine. Things such as bat, wget, fzf, fish, etc. along with common symlinks and xdg-config links. My nvim and fish configs are installed and managed this way. Rather than need to maintain a neovim config for every different system, in the nix way, I can just manage it all in _common/programs.nix.
This is not "The Standard Way" to organize things, if you want more inspiration, I took a lot from my friend Andrey's Nixfiles. I was also chatting with him a bunch during this, so I was able to get three systems up and configured in a few days. After the first ubuntu box was configured, it was super easy to manage my others.
My home/ directory is where I store my config files. My ssh public keys, my gpg public keys, my ~/.<dotfiles> and my ~/.config/<files>. This doesn't really need any explaination, but as an added benefit is I also decided to LUA-ify my nvim configs the same weekend. But that's a story for another time.
I am at this time choosing not to do NixOS - and relying on Ubuntu for managing my OS. I peeked into Andrey's files, and I really don't want to have to manage a full system configuration, drivers, etc. with Nix. Maybe for the future - when my Lenovo X1 Carbon dies and I need to reinstall that though.
I have a tiny project Enclosure box that I dremeled a hole for the GPIO pins in the cover and I then sandwich the Blinkt onto the Pi Zero with another dremeled hole running to the micro usb power, and that's it for hardware.
For software, I installed the python packages for Pimoroni and Blinkt, which came with a lovely set of sample projects. I deleted everything except the mqtt.py file, which I then put my Mosquitto server settings.
I then added a new service in systemd to control the mqtt server
Pleased with the results, and testing by sending some messages over mqtt that changed the color, I then dove into Node-RED
Node-Red
This is my first project using Node-RED, so I'm sure I could optimize better, but I have two entry points, one is from running HomeAssistant app on my mac, which gets me sensor data for my webcam, and the other is the aforementioned Zoom Presence plugin I created. These are Events:State nodes.
When either of these are True, they call first my ceiling light to turn on, which next will then add a msg.payload of
as one string. This leads to a Split, which will in turn, emit a new MQTT message for each line (I split on \n) and turn on all 8 LEDs as red. This is inefficient because I am still using the sample code for the blinkt which requires you to address each LED individually, my next phase I will remove the pin requirement and just have it send a color for all of them at once, one line.
When either of the sensors states are False, I then flow into a Time Range node, in which I check if it's between 9-5 or not. If it is, then I turn all the LEDs Green, and if it's outside 9-5 I just turn the LEDs off. I do not turn OFF the overhead light, in case it was already on. I don't care about the state enough.
I also intentionally trigger at the Office Hours node, which will inherently turn the Green on at 9:01am, and off at 5:01pm. As well as turn on Red for any long standing meeting times I have.
I'm not the best SQL developer, I know it's one of my weak points.
My history is I did php/mysql from the early 2000s until college.
In college I didn't really focus on the Database courses, the class selection didn't have many database course.
The one Data Warehousing course I had available, I missed out on because I was in England doing a study abroad program that semester.
My first job out of college was a Python/Django company - and that directed my next eight years of work.
Django, if you are unaware, is a MVC framework that ships with a really great ORM.
You can do about 95% of your database queries automatically by using the ORM.
Above are some samples from the DjangoDocs.
But enough about Django.
My Requirements
Recently at my job I was given a little bit of leeway on a project.
My team is sort of dissolving and merging in with another team who already does Go.
My Go history is building a CLI tool for the two last years of my previous job.
I had never directly interacted with a database from Go yet.
I wanted to spin up a REST API (I chose Go+Gin for that based on forty five seconds of Googling) and talk to a database.
GORM
Being that I come from the Django (and a few years of ActiveRecord) land, I reached immediately for an ORM, I chose GORM.
If you want to skip directly to the source, check out https://gitea.tyrel.dev/tyrel/go-webservice-gin.
Full design disclosure: I followed a couple of blog posts in order to develop this, so it is in the form explictly decided upon by the logrocket blog post and may not be the most efficient way to organize the module.
In order to instantiate a model definition, it's pretty easy.
What I did is make a new package called models and inside made a file for my Album.
This tracks with how I would do the same for any other kind of struct in Go, so this wasn't too difficult to do.
What was kind of annoying was that I had to also make some structs for Creating the album and Updating the Album, this felt like duplicated effort that might have been better served with some composition.
I would have structured the controllers differently, but that may be a Gin thing and how it takes points to functions, vs pointers to receivers on a struct.
Not specific to GORM.
Each of the controller functions were bound to a gin.Context pointer, rather than receivers on an AlbumController struct.
The FindAlbum controller was simple:
funcFindAlbum(c*gin.Context){varalbummodels.Albumiferr:=models.DB.Where("id = ?",c.Param("id")).First(&album).Error;err!=nil{c.JSON(http.StatusBadRequest,gin.H{"error":"Record not found!"})}c.JSON(http.StatusOK,gin.H{"data":album})}
Which will take in a /:id path parameter, and the GORM part of this is the third line there.
To run a select, you chain a Where on the DB (which is the connection here) and it will build up your query.
If you want to do joins, this is where you would chain .Joins etc...
You then pass in your album variable to bind the result to the struct, and if there's no errors, you continue on with the bound variable.
Error handling is standard Go logic, if err != nil etc and then pass that into your API of choice (Gin here) error handler.
This was really easy to set up, and if you want to get a slice back you just use DB.Find instead, and bind to a slice of those structs.
varalbums[]models.Albummodels.DB.Find(&albums)
SQLX
SQLX is a bit different, as it's not an ORM, it's extensions in Go to query with SQL, but still a good pattern for abstracting away your SQL to some dark corner of the app and not inline everywhere.
For this I didn't follow someone's blog post — I had a grasp on how to use Gin pretty okay by now and essentially copied someone elses repo with my existing model.
gin-sqlx-crud.
This one set up a bit wider of a structure, with deeper nested packages.
Inside my internal folder there's controllers, forms, models/sql, and server.
I'll only bother describing the models package here, as thats the SQLX part of it.
In the models/album.go file, there's your standard struct here, but this time its bound to db not json, I didn't look too deep yet but I presume that also forces the columns to set the json name.
An interface to make a service, and a receiver are made for applying the CreateAlbum form (in another package) which sets the form name and json name in it.
Nested inside the models/sql/album.go file and package, is all of the Receiver code for the service.
I'll just comment the smallest one, as that gets my point across.
Here is where the main part of GORM/SQLX differ - raw SQL shows up.
func(s*AlbumService)GetAll()(*[]models2.Album,error){q:=`SELECT * FROM albums;`varoutput[]models2.Albumerr:=s.conn.Select(&output,q)// Replace the SQL error with our own error type.iferr==sql.ErrNoRows{returnnil,models2.ErrNotFound}elseiferr!=nil{returnnil,err}else{return&output,nil}}
This will return a slice of Albums - but if you notice on the second line, you have to write your own queries.
A little bit more in control of how things happen, with a SELECT * ... vs the gorm DB.Find style.
To me this feels more like using pymysql, in fact its a very similar process.
(SEE NOTE BELOW)
You use the service.connection.Get and pass in what you want the output bound to, the string query, and any parameters.
This feels kind of backwards to me - I'd much rather have the order be: query, bound, parameters, but thats what they decided for their order.
Conclusion
Overall, both were pretty easy to set up for one model.
Given the choice I would look at who the source code is written for.
If you're someone who knows a lot of SQL, then SQLX is fine.
If you like abstractions, and more of a "Code as Query" style, then GORM is probably the best of these two options.
I will point out that GORM does more than just "query and insert" there is migration, logging, locking, dry run mode, and more.
If you want to have a full fledged system, that might be a little heavy, then GORM is the right choice.
SQLX is great if what you care about is marshalling, and a very quick integration into any existing codebase.
I sent this blog post to my friend Andrey and he mentioned that I was incorrect with my comparision of sqlx to pymysql.
To put it in a python metaphor, "sqlx is like using urllib3, gorm is like using something that generates a bunch of requests code for you. Using pymysql is like using tcp to do a REST request."
Sqlx is more akin to SqlAlchemy core vs using SqlAlchemy orm.
Sqlx is just some slight extensions over database/sql.
As the sort of equivalent to pymysql in Go is database/sql/driver from the stdlib.
This was written in 2017, but I found a copy again, I wanted to post it again.
The Story
For a few months last year, I lived around the block from work. I would sometimes stop in on the weekends and pick up stuff I forgot at my desk. One day the power went out at my apartment and I figure I would check in at work and see if there were any problems. I messaged our Lab Safety Manager on slack to say "hey the power went out, and I am at the office. Is there anything you'd like me to check?". He said he hadn't even gotten the alarm email/pages yet, so if I would check out in the lab and send him a picture of the CO2 tanks to make sure that nothing with the power outage compromised those. Once I had procured access to the BL2 lab on my building badge, I made my way out back and took a lovely picture of the tanks, everything was fine.
The following week, in my one on one meeting with my manager, I mentioned what happened and she and I discussed the event. It clearly isn't sustainable sending someone in any time there was a power outage if we didn't need to, but the lab equipment doesn't have any monitoring ports.
Operation Lab Cam was born. I decided to put together a prototype of a Raspberry Pi 3 with a camera module and play around with getting a way to monitor the display on the tanks. After a few months of not touching the project, I dug into it in a downtime day again. The result is now we have an automated camera box that will take a picture once a minute and display it on an auto refreshing web page. There are many professional products out there that do exactly this, but I wanted something that has the ability to be upgraded in the future.
Summary of the Technical Details
Currently the entire process is managed by one bash script, which is a little clunky, but it's livable. The implementation of the script goes a little like:
Take a picture to a temporary location.
Add a graphical time stamp.
Copy that image to both the currently served image, and a timestamped filename backup.
The web page that serves the image is just a simple web page that shows the image, and refreshes once every thirty seconds.
The Gritty Technical Details
The program I'm using to take pictures is the raspistill program. If I had written my script to just call raspistill every time I wanted a picture taken, it would have potentially taken a lot longer to save the images. This happens because it needs to meter the image every time, which adds up. The solution is Signal mode and turning raspistill into a daemon. If you enable signal mode, any time you send a SIGUSR1 to the process, the backgrounded process will then take the image.
Instead if setting up a service with systemd, I have a small bash script. At the beginning, I run a ps aux and check if raspistill is running, if it's not, I start up a new instance of raspistill with the appropriate options and background it. The next time this script runs, it will detect that raspistill is running and be almost a no-op.
After this, I send a SIGUSR1 (kill -10) to take the picture which is then saved, un-timestamped. Next up I call imagemagick's convert on this image, I crop out the center (so I couldn't use raspistill's "-a 12" option) because all I care about is a 500x700 pixel region.
This is then copied to the image that is served by the web page, and also backed up in a directory that nginx will listen to.
Tieg: Hey Tyrel, I can't run invoke sign 5555, can you help with this?
This is How my night started last night at 10pm. My coworker Tieg did some work on our CLI project and was trying to release the latest version. We use invoke to run our code signing and deployment scripts, so I thought it was just a quick "oh maybe I screwed up some python!" fix. It wasn't.
I spent from 10:30 until 1:30am this morning going through and looking into why Tieg wasn't able to sign the code. The first thing I did was re-run the build on CircleCI, which had the same error, so hey! at least it was reproducible. The problem was that in our Makefile scripts we run tidelift version > tidelift-cli.version and then upload that to our deployment directories, but this was failing for some reason. We let clients download this file to see what the latest version is and then our CLI tool has the ability to selfupdate (except on homebrew) to pull this latest version if you're outdated.
Once I knew what was failing, I was able to use CircleCI's ssh commands and log in, and see what happened, but I was getting some other errors. I was seeing some problems with dbus-launch so I promptly (mistakenly) yelled to the void on twitter about dubs-launch. Well would you know it, I may have mentioned before, but I work with Havoc Pennington.
Havoc Pennington: fortunately I wrote dbus-launch so may be able to tell you something, unfortunately it was like 15 years ago
Pumped about this new revelation, I started looking at our keychain dependency, because I thought the issue was there as that's the only thing that uses dbus on Linux. Then we decided (Havoc Pointed it out) that it was a red herring, and maybe the problem was elsewhere. I at least learned a bit about dbus and what it does, but not enough to really talk about it to any detail.
Would you know it, the problem was elsewhere. Tieg was running dtruss and saw that one time it was checking his /etc/hosts file when it was failing, and another time it was NOT, which was passing. Then pointed out a 50ms lookup to our download.tidelift.com host.
Tieg then found Issue 49517 this issue where someone mentions that Go 1.17.3 was failing them for net/http calls, but not the right way.
It turns out, that it wasn't the keyring stuff, it wasn't the technically the version calls that failed. What was happening is every command starts with a check to https://download.tidelift.com/cli/tidelift-cli.version which we then compare to the current running version, if it's different and outdated, we then say "you can run selfupdate!". What fails is that call to download.tidelift.com, because of compiling with go1.17.3 and a context canceled due to stream cleanup I guess?
Okay so we need to downgrade to Go 1.17.2 to fix this. Last night in my trying, I noticed that our CircleCI config was using circle/golang:1.16 as its docker image, which has been superseded by cimg/go:1.16.x style of images. But I ran into some problems with that while upgrading to cimg/go:1.17.x. The problem was due to the image having different permissions, so I couldn't write to the same directories that when Mike wrote our config.yml file, worked properly.
Tieg and I did a paired zoom chat and finished this up by cutting out all the testing/scanning stuff in our config files, and just getting down to the Build and Deploy steps. Found ANOTHER bug that Build seems to run as the circleci user, but Deploy was running as root. So in the build working_directory setting, using a ~/go/tidelift/cli path, worked. But when we restored the saved cache to Deploy, it still put it in /home/circle/go/tidelift/cli, but then the working_directory of ~/go/tidelift/cli was relative to /root/. What a nightmare!
All tildes expanded to /home/circleci/go/tidelift/cli set, Makefile hacks undone, (removing windows+darwin+arm64 builds from your scripts during testing makes things A LOT faster!) and PR Merged, we were ready to roll.
I merged the PR, we cut a new version of TideliftCLI 1.2.5, updated the changelog and signed sealed delivered a new version which uses Go 1.17.2, writes the proper tidelift-cli.version file in deployment steps, and we were ready to ROCK!
That was fun day. Now it's time to write some rspec tests.
I never intended this to be a full fleshed CLI tool comparable to the likes of the real GitHub CLI. This was simply a way to refresh myself and have fun. I have accomplished this, and am now calling this "Feature Complete". You can play around with it yourself from the repository on gitlab.
HTTPX.LINKS
Today I learned some fun things with httpx mainly. The main thing I focused on today was figuring out pagination. The GitHub API uses the link header which I had never seen before.
The format of the header is a url, and then a relationship of that url. Lets take my friend Andrey's repos for example:
sample header value from Getting Andrey's repositories list and checking pagination
The link header there has two items split by a comma, each with two fields split by a semicolon, the first of which is a URL inside angle brackets... and AGHH this is going to be annoying to parse! Luckily httpx responses have this handled and client.get(...).links returns a lovely dictionary of the proper data.
With a response.links.get('next') check, you can get the url from response.links['next']['url']. So much nicer than writing some regular expressions.
TESTING
With that accomplished, I then added pytest-cov to my requirements.in and was able to leverage some coverage checks. I was about 30% with the latest changes (much higher than anticipated!) so I knew what I wanted to focus on next. The API seemed the easiest to test first again, so I changed around how I loaded my fixtures and made it pass in a name and open that file instead. In real code I would not have the function in both my test files, I would refactor it, but again, this is just a refresher, I'm lazy.
I decided earlier that I also wanted to catch HTTP 403 errors as I ran into a rate limit issue. Which, I assure you dear reader, was a thousand percent intentional so I would know what happens. Yeah, we'll go with that.
Py.Test has a context manager called pytest.raises and I was able to just with pytest.raises(httpx.HttpStatusError) and check that raise really easily.
The next bits of testing for the API were around the pagination, I faked two responses and needed to update my link header, checking the cases where there was NO link, was multiple pages, and with my shortcut return - in case the response was an object not a list. Pretty straight forward.
The GHub file tests were kind of annoying, I'm leveraging rich.table.Table so I haven't been able to find a nice "this will make a string for you" without just using rich's print function. I decided the easiest check was to see if the Table.Columns.Cells matched what I wanted, which felt a little off but it's fine.
The way I generated the table is by making a generator in a pretty ugly way and having a bunch of repo['column'],repo['column'] responses, rather than doing a dict comprehension and narrowing the keys down. If I ever come back to this, I MIGHT reassess that with a {k:v for k,v in repos if k in SELECTED_KEYS} and then yield a dictionary, but it's not worth the effort.
Overall I'd say this project was fun. It gave me a glimpse back into the Python world, and an excuse to write a couple blog posts. My next project is to get a Django site up and running again, so I can figure out how to debug my django-dbfilestorage.
Closing Thoughts
If I had to do this again, I would probably have tried some test driven development. I've tried in the past, but I don't work on a lot of greenfield projects. I tend to be the kind of engineer who jumps HEAD FIRST into code and then tests are an after thought.
I also kind of want to rewrite this in Go and Rust, two other languages I've been fond of lately, just to see how they'd compare in fun. I haven't done any API calls with Rust yet, only made a little Roguelike by following Herbert Wolverson's Hands-On-Rust book. The Tidelift CLI is all Go and a bazillion API calls (okay like ten) so that wouldn't be too hard to use like SPF13's Cobra CLI library and make a quick tool that way.
One fun thing I learned while moving things over to GitLab is that my user Tyrel is a super early adopter. I was in the first 36,000 people! I showed a screenshot of my user ID to my friend Sunanda at GitLab and we had fun finding that out.
It's no lie that I love terminals. I wish I could live on a terminal and never really need to see a GUI application again.
Last night I migrated a lot of my old code from one GitLab account to another (tyrelsouza to tyrel) in an effort to clean up some of my usernames spread across the world. While doing that I noticed my django-dbfilestorage Python module that has been sitting and rotting for three years. I played around a little bit in order to port it to Python 3.9, but I ran into some base64 errors. I tried a little bit but it was late and I couldn't figure it out. My resolve is that I have been away from Python for too long so the little things - that I knew and love - had fallen away. I mentioned this to my friend Alex and he said "make a barebones github cli (readonly?) with issue viewer, and stats display". I've embarked on a journey to refresh my Python well enough to repair DBFS.
I knew I wanted to use httpx as my network client library, it's new, fast, and I have a couple friends who work on it. I started with a barebones requirements.in file, tossed in invoke, pytest, and black. From there I used pip-compile to generate my requirements.txt - (a tip I picked up recently while adding Pip-Compile support to the Tidelift CLI) and I was good to go.
The docs for the GitHub API are pretty easy to read, so I knew all I really needed to do was set my Accept header to be Version3 and I could view the schema. With the schema saved to a .json file I then wrote a GHub class to pull this data down using httpx.client.Client.get, super simple! The only two endpoints I care about right now are the user and repos endpoints, so I made two get_ functions for each. After a little bit of work - which I won't bore you with the super intricate details - I have a functional cli.py file. For now, the only interaction is a propmt from rich for a username, and then you get a fancy table (also from rich) of the first page of results of repos, stargazer/watchers/forks counts, and a description.
Prompting for the username and showing my table of repositories.
It was a fun evening of learning what's changed in Python3 since I last touched it, especially as I've spent the majority of my career in Python2.7. Type annotations are super awesome. I'll probably pick it up again once I get some more free time later in the week. It's also nice blog fodder! I already have a million things I want to do next - pagination, caching, some more interaction.
Showing py.test running
I know the tool I'm writing is nothing special, especially with their own cli now, but I'm not looking at reinventing the wheel!
Check out the code so far on my GitLab (heh, ironic it's there).
When I worked at Propel Marketing, we used to outsource static websites to a third party vendor, and then host them on our server. It was our job as developers to pull down the finished website zip file from the vendor, check it to make sure they used the proper domain name, (they didn't a lot of the time,) and make sure it actually looks nice. If these few criteria were met, we could launch the site.
Part of this process was SCPing the directory to our sites server. The sites server was where we had Apache running with every custom static site as a vhost. We would put the website in /var/www/vhosts/domain.name.here/ and then create the proper files in sites-available and sites-enabled (more on this in another entry). After that the next step was to run a checkconfig and restart Apache.
Here's where it all went wrong one day. If I can recall correctly, my boss was on vacation so he had me doing a bit of extra work and launching a few more sites than I usually do. Not only that, but we also had a deadline of the end of the month which was either the next day, or the day after. I figure I'll just setup all mine for two days, and then have some extra time the next day for other things to work on. So I started launching my sites. After each one, I would add the domain it was supposed to be at into my /etc/hosts file and make sure it worked.
I was probably half way done with my sites, and suddenly I ran into one that didn't work. I checked another one to see if maybe it was just my network being stupid and not liking my hosts file, but no, that wasn't the problem. Suddenly, EVERY SITE stopped working on this server. Panicking, I delete the symlink in sites-enabled and restart Apache. Everything works again. I then proceed to put that site aside, maybe something in the php files breaks the server, who knows, but I have other sites I can launch.
I setup the next site and the same thing happens again, no sites work. Okay, now it's time to freak out and call our sysadmin. He didn't answer his phone, so I call my boss JB. I tell him the problem and he says he will reach out to the sysadmin and see what the problem is, all the while I'm telling JB "It's not broken broken, it just doesn't work, it's not my fault" etc etc. A couple hours later, our sysadmin emails us back and says he was able to fix the problem.
It turns out, there's a hard limit to the number of files your system can have open per user, and this was set to 1000 for the www-data user. The site I launched was coincidentally the 500th site on that server, each of them having an access.log and an error.log. These two files apparently constantly open on each site for apache to log to. He was able to change www-data's ulimit to a lot higher, (I don't recall now what it was) and that gave a lot more leeway in how many sites the sites server could host.
When I worked at Propel Marketing, my dev team used to have presentations on things they loved. I love the Python debugger. It's very useful and I believe a proper understanding of how to use a debugger, will make you a better programmer. Here is a presentation on the debugger I made for my team. https://prezi.com/cdc4uyn4ghih/python-debugger/
I had a friend complain that he had to keep adding his ssh key to his ssh-agent every time he rebooted. I have a really easy bit of shell code you can put into your .bashrc or your .zshrc file:
This will check every time your bash or zsh rc file is sourced for your key in the currently added keys, and if it's not, it will add it.
This has the benefit of not requiring you to put in your password every time you connect to a remote machine if you password your ssh keys (which you should).
