Tyrel's Blog

Notes

Small note: If you run this, you will actually get a list of tuples, so you can map those back to a list, if you need to update them for any reason. I just wanted square brackets in my examples.

# This is just messy looking, so I didn't mention it until now
>>> list(map(list, zip(*matrix[::-1])))

As I mentioned, due to using zip this will work with non-square examples as well.

>>> matrix = [
... [1,2,3,4,5,6,7,8,9],
... [9,8,7,6,5,4,3,2,1],
... ]
>>> print(list(zip(*matrix[::-1])))
[(9, 1),
 (8, 2),
 (7, 3),
 (6, 4),
 (5, 5),
 (4, 6),
 (3, 7),
 (2, 8),
 (1, 9)]

NOTES

This post is ported over from my wiki, so the format isn't as storytelling as a blog post could be, but I wanted it here.

Bill of Materials

• Raspberry Pi Zero W H (WiFi + Headers)
• BlinkT LED Strip GPIO

Home Assistant Parts

[Unit]
Description=Meeting Indicator

[Service]
Type=simple
ExecStart=/usr/bin/python2 /home/pi/mqtt.py
WorkingDirectory=/home/pi/Pimoroni/blinkt/examples
Restart=always
RestartSec=2

[Install]
WantedBy=sysinit.target

rgb,0,255,0,0
rgb,1,255,0,0
rgb,2,255,0,0
rgb,3,255,0,0
rgb,4,255,0,0
rgb,5,255,0,0
rgb,6,255,0,0
rgb,7,255,0,0

Images

Django ORM - My History

I'm not the best SQL developer, I know it's one of my weak points. My history is I did php/mysql from the early 2000s until college. In college I didn't really focus on the Database courses, the class selection didn't have many database course. The one Data Warehousing course I had available, I missed out on because I was in England doing a study abroad program that semester. My first job out of college was a Python/Django company - and that directed my next eight years of work.

Django, if you are unaware, is a MVC framework that ships with a really great ORM. You can do about 95% of your database queries automatically by using the ORM.

entry, created = Entry.objects.get_or_create(headline="blah blah blah")

q = Entry.objects.filter(headline__startswith="What")
q = q.filter(pub_date__lte=datetime.date.today())
q = q.exclude(body_text__icontains="food")

Above are some samples from the DjangoDocs. But enough about Django.

My Requirements

Recently at my job I was given a little bit of leeway on a project. My team is sort of dissolving and merging in with another team who already does Go. My Go history is building a CLI tool for the two last years of my previous job. I had never directly interacted with a database from Go yet. I wanted to spin up a REST API (I chose Go+Gin for that based on forty five seconds of Googling) and talk to a database.

GORM

Being that I come from the Django (and a few years of ActiveRecord) land, I reached immediately for an ORM, I chose GORM. If you want to skip directly to the source, check out https://gitea.tyrel.dev/tyrel/go-webservice-gin. Full design disclosure: I followed a couple of blog posts in order to develop this, so it is in the form explictly decided upon by the logrocket blog post and may not be the most efficient way to organize the module.

In order to instantiate a model definition, it's pretty easy. What I did is make a new package called models and inside made a file for my Album.

type Album struct {
      ID     string  `json:"id" gorm:"primary_key"`
      Title  string  `json:"title"`
      Artist string  `json:"artist"`
      Price  float64 `json:"price"`
}

This tracks with how I would do the same for any other kind of struct in Go, so this wasn't too difficult to do. What was kind of annoying was that I had to also make some structs for Creating the album and Updating the Album, this felt like duplicated effort that might have been better served with some composition.

I would have structured the controllers differently, but that may be a Gin thing and how it takes points to functions, vs pointers to receivers on a struct. Not specific to GORM. Each of the controller functions were bound to a gin.Context pointer, rather than receivers on an AlbumController struct.

The FindAlbum controller was simple:

func FindAlbum(c *gin.Context) {
      var album models.Album
      if err := models.DB.Where("id = ?", c.Param("id")).First(&album).Error; err != nil {
              c.JSON(http.StatusBadRequest, gin.H{"error": "Record not found!"})
      }
      c.JSON(http.StatusOK, gin.H{"data": album})
}

Which will take in a /:id path parameter, and the GORM part of this is the third line there.

models.DB.Where("id = ?", c.Param("id")).First(&album).Error

To run a select, you chain a Where on the DB (which is the connection here) and it will build up your query. If you want to do joins, this is where you would chain .Joins etc... You then pass in your album variable to bind the result to the struct, and if there's no errors, you continue on with the bound variable. Error handling is standard Go logic, if err != nil etc and then pass that into your API of choice (Gin here) error handler.

This was really easy to set up, and if you want to get a slice back you just use DB.Find instead, and bind to a slice of those structs.

var albums []models.Album
models.DB.Find(&albums)

SQLX

SQLX is a bit different, as it's not an ORM, it's extensions in Go to query with SQL, but still a good pattern for abstracting away your SQL to some dark corner of the app and not inline everywhere. For this I didn't follow someone's blog post — I had a grasp on how to use Gin pretty okay by now and essentially copied someone elses repo with my existing model. gin-sqlx-crud.

This one set up a bit wider of a structure, with deeper nested packages. Inside my internal folder there's controllers, forms, models/sql, and server. I'll only bother describing the models package here, as thats the SQLX part of it.

In the models/album.go file, there's your standard struct here, but this time its bound to db not json, I didn't look too deep yet but I presume that also forces the columns to set the json name.

type Album struct {
  ID     int64   `db:"id"`
  Title  string  `db:"title"`
  Artist string  `db:"artist"`
  Price  float64 `db:"price"`
}

An interface to make a service, and a receiver are made for applying the CreateAlbum form (in another package) which sets the form name and json name in it.

func (a *Album) ApplyForm(form *forms.CreateAlbum) {
  a.ID = *form.ID
  a.Title = *form.Title
  a.Artist = *form.Artist
  a.Price = *form.Price
}

So there's the receiver action I wanted at least!

Nested inside the models/sql/album.go file and package, is all of the Receiver code for the service. I'll just comment the smallest one, as that gets my point across. Here is where the main part of GORM/SQLX differ - raw SQL shows up.

func (s *AlbumService) GetAll() (*[]models2.Album, error) {
      q := `SELECT * FROM albums;`

      var output []models2.Album
      err := s.conn.Select(&output, q)
      // Replace the SQL error with our own error type.
      if err == sql.ErrNoRows {
              return nil, models2.ErrNotFound
      } else if err != nil {
              return nil, err
      } else {
              return &output, nil
      }
}

This will return a slice of Albums - but if you notice on the second line, you have to write your own queries. A little bit more in control of how things happen, with a SELECT * ... vs the gorm DB.Find style.

To me this feels more like using pymysql, in fact its a very similar process. (SEE NOTE BELOW) You use the service.connection.Get and pass in what you want the output bound to, the string query, and any parameters. This feels kind of backwards to me - I'd much rather have the order be: query, bound, parameters, but thats what they decided for their order.

Conclusion

Overall, both were pretty easy to set up for one model. Given the choice I would look at who the source code is written for. If you're someone who knows a lot of SQL, then SQLX is fine. If you like abstractions, and more of a "Code as Query" style, then GORM is probably the best of these two options.

I will point out that GORM does more than just "query and insert" there is migration, logging, locking, dry run mode, and more. If you want to have a full fledged system, that might be a little heavy, then GORM is the right choice.

SQLX is great if what you care about is marshalling, and a very quick integration into any existing codebase.

Repositories

• Go, Gin, Gorm
• Go, Gin, sqlx

Notes

I sent this blog post to my friend Andrey and he mentioned that I was incorrect with my comparision of sqlx to pymysql. To put it in a python metaphor, "sqlx is like using urllib3, gorm is like using something that generates a bunch of requests code for you. Using pymysql is like using tcp to do a REST request." Sqlx is more akin to SqlAlchemy core vs using SqlAlchemy orm. Sqlx is just some slight extensions over database/sql. As the sort of equivalent to pymysql in Go is database/sql/driver from the stdlib.

Parsing Epubs

Recently I've become frustrated with the experience of reading books on my Kindle Paperwhite. The swipe features, really bother me. I really like MoonReader on Android, but reading on my phone isn't always pleasing. This lead me to look into other hardware. I've been eyeing the BOOX company a while ago, but definitely considering some of their new offerings some time. Until the time I can afford the money to splurge on a new ebook reader, I've decided to start a new project, making my own ebook reader tools!

I'm starting with EPUBs, as this is one of the easiest to work with. At its core, an EPUB is a zip file with the .epub extension instead of .epub with many individual XHTML file chapters inside it. You can read more of how they're structured yourself over at FILEFORMAT.

The tool I've chosen for reading EPUBs is the Python library ebooklib. This seemed to be a nice lightweight library for reading EPUBs. I also used DearPyGUI for showing this to the screen, because I figured why not, I like GUI libraries.

My first task was to find an EPUB file, so I downloaded one from my calibre server. I convert all my ebook files to .epub and .mobi on my calibre server so I can access them anywhere I can read my OPDS feed. I chose Throne of Glass (abbreviating to TOG.epub for rest of post). Loading I launched Python, and ran

>>> from ebooklib import epub
>>> print(book := epub.read_epub("TOG.epub")

This returned me a <ebooklib.epub.EpubBook object...> , seeing I had an EpubBook I ran a dir(book) and found the properties available to me

['add_author', 'add_item', 'add_metadata', 'add_prefix',
 'bindings', 'direction', 'get_item_with_href', 'get_item_with_id',
 'get_items', 'get_items_of_media_type', 'get_items_of_type',
 'get_metadata', 'get_template', 'guide',
 'items', 'language', 'metadata', 'namespaces', 'pages', 'prefixes',
 'reset', 'set_cover', 'set_direction', 'set_identifier', 'set_language',
 'set_template', 'set_title', 'set_unique_metadata', 'spine',
 'templates', 'title', 'toc', 'uid', 'version']

Of note, the get_item_with_X entries caught my eye, as well as spine. For my file, book.spine looks like it gave me a bunch of tuples of ID and a "yes" string of which I had no Idea what was. I then noticed I had a toc property, assuming that was a Table of Contents, I printed that out and saw a bunch of epub.Link objects. This looks like something I could use.

I will note, at this time I was thinking that this wasn't the direction I wanted to take this project. I really wanted to learn how to parse these things myself, unzip, parse XML, or HTML, etc., but I realized I needed to see someone else's work to even know what is going on. With this "defeat for the evening" admitted, I figured hey, why not at least make SOMETHING, right?" I decided to carry on.

Seeing I was on at least some track, I opened up PyCharm and made a new Project. First I setup a class called Epub, made a couple of functions for setting things up and ended up with

class Epub:
    def __init__(self, book_path: str) -> None:
        self.contents: ebooklib.epub.EpubBook = epub.read_epub(book_path)
        self.title: str = self.contents.title
        self.toc: List[ebooklib.epub.Link] = self.contents.toc

I then setup a parse_chapters file, where I loop through the TOC. Here I went to the definition of Link and saw I was able to get a href and a title, I decided my object for chapters would be a dictionary (I'll move to a DataClass later) with title and content. I remembered from earlier I had a get_item_by_href so I stored the itext from the TOC's href: self.contents.get_item_with_href(link.href).get_content(). This would later prove to be a bad decision when I opened "The Fold.epub" and realized that a TOC could have a tuple of Section and Link, not just Links. I ended up storing the item itself, and doing a double loop in the parse_chapters function to loop if it's a tuple.

def parse_chapters(self) -> None:
    idx = 0
    for _item in self.toc:
        if isinstance(_item, tuple):  # In case is section tuple(section, [link, ...])
            for link in _item[1]:
                self._parse_link(idx, link)
                idx += 1
        else:
            self._parse_link(idx, _item)
            idx += 1

_parse_link simply makes that dictionary of title and item I mentioned earlier, with a new index as I introduced buttons in the DearPyGUI at this time as well.

def _parse_link(self, idx, link) -> None:
    title = link.title
    self.chapters.append(dict(
        index=idx,
        title=title,
        item=self.contents.get_item_with_href(link.href)
    ))

That's really all there is to make an MVP of an EPUB parser. You can use BeautifulSoup to parse the HTML from the get_body_contents() calls on items, to make more readable text if you want, but depending on your front end, the HTML may be what you want.

In my implementation my Epub class keeps track of the currently selected chapter, so this loads from all chapters and sets the current_text variable.

def load_view(self) -> None:
    item = self.chapters[self.current_index]['item']
    soup = BeautifulSoup(item.get_body_content(), "html.parser")
    text = [para.get_text() for para in soup.find_all("p")]
    self.current_text = "\n".join(text)

I don't believe any of this code will be useful to anyone outside of my research for now, but it's my first step into writing an EPUB parser myself.

The DearPyGUI steps are out of scope of this blog post, but here is my final ebook Reader which is super inefficient!

I figure the Dedication page is not as copywrited as the rest of the book, so it's fair play showing that much. Sarah J Maas, if you have any issues, I can find another book for my screenshots.

HTTPX.LINKS

Today I learned some fun things with httpx mainly. The main thing I focused on today was figuring out pagination. The GitHub API uses the link header which I had never seen before.

The format of the header is a url, and then a relationship of that url. Lets take my friend Andrey's repos for example:

The link header there has two items split by a comma, each with two fields split by a semicolon, the first of which is a URL inside angle brackets... and AGHH this is going to be annoying to parse! Luckily httpx responses have this handled and client.get(...).links returns a lovely dictionary of the proper data.

With a response.links.get('next') check, you can get the url from response.links['next']['url']. So much nicer than writing some regular expressions.

TESTING

With that accomplished, I then added pytest-cov to my requirements.in and was able to leverage some coverage checks. I was about 30% with the latest changes (much higher than anticipated!) so I knew what I wanted to focus on next. The API seemed the easiest to test first again, so I changed around how I loaded my fixtures and made it pass in a name and open that file instead. In real code I would not have the function in both my test files, I would refactor it, but again, this is just a refresher, I'm lazy.

I decided earlier that I also wanted to catch HTTP 403 errors as I ran into a rate limit issue. Which, I assure you dear reader, was a thousand percent intentional so I would know what happens. Yeah, we'll go with that.

Py.Test has a context manager called pytest.raises and I was able to just with pytest.raises(httpx.HttpStatusError) and check that raise really easily.

The next bits of testing for the API were around the pagination, I faked two responses and needed to update my link header, checking the cases where there was NO link, was multiple pages, and with my shortcut return - in case the response was an object not a list. Pretty straight forward.

The GHub file tests were kind of annoying, I'm leveraging rich.table.Table so I haven't been able to find a nice "this will make a string for you" without just using rich's print function. I decided the easiest check was to see if the Table.Columns.Cells matched what I wanted, which felt a little off but it's fine.

The way I generated the table is by making a generator in a pretty ugly way and having a bunch of repo['column'], repo['column'] responses, rather than doing a dict comprehension and narrowing the keys down. If I ever come back to this, I MIGHT reassess that with a {k:v for k,v in repos if k in SELECTED_KEYS} and then yield a dictionary, but it's not worth the effort.

Overall I'd say this project was fun. It gave me a glimpse back into the Python world, and an excuse to write a couple blog posts. My next project is to get a Django site up and running again, so I can figure out how to debug my django-dbfilestorage.

Closing Thoughts

If I had to do this again, I would probably have tried some test driven development. I've tried in the past, but I don't work on a lot of greenfield projects. I tend to be the kind of engineer who jumps HEAD FIRST into code and then tests are an after thought.

I also kind of want to rewrite this in Go and Rust, two other languages I've been fond of lately, just to see how they'd compare in fun. I haven't done any API calls with Rust yet, only made a little Roguelike by following Herbert Wolverson's Hands-On-Rust book. The Tidelift CLI is all Go and a bazillion API calls (okay like ten) so that wouldn't be too hard to use like SPF13's Cobra CLI library and make a quick tool that way.

One fun thing I learned while moving things over to GitLab is that my user Tyrel is a super early adopter. I was in the first 36,000 people! I showed a screenshot of my user ID to my friend Sunanda at GitLab and we had fun finding that out.