Nov 13, 2013
Recently I have been diving into using signals with Django, which of course are pretty neat.
I am working on a website for work which in the most basicexplanation, is a task management site. Recently I have added in the ability to subscribe to tasks and get emails, I did this by connecting to the post_save signal. I only email out when a task is changed, not created (of course, no one would be subscribed to it). This worked flawlessly and "emails" out to anyone who is subscribed. I say that in quotes, because I haven't actually hooked it up to a real SMTP server, and only use
python -m smtpd -n -c DebuggingServer localhost:1025
which will output any emails to stdout. But I digress… A problem arose when I was working on ordering tasks.
I store an integer in the "ordering" column, which any authenticated user can drag the row to a new location and that will reorder the task. I did this after I setup the emailing signal, so I didn't think about an email being sent out for EVERY task being changed.
I tried a lot of different things, and was debating some that would be a bit messy. Among those ideas were trying to store the past values in another table, but that would get expensive fast. The reason I tried this was because I wanted to see if the ordering was the only thing that changed, and if that was the case, not send an email. I eventually found a thread on StackOverflow that says to use update on the queryset to not trigger the signal.
You can do this by doing something like this:
from app.models import ModelName
def reorder(request):
new_order = request.POST.get('new_order', None)
pk = request.POST.get('modelname_pk', None)
if new_order:
ModelName.objects.filter(pk=pk).update(ordering=new_order)
I am not sure if this is the proper way save changes and not trigger a post_save signal, but this is the way that worked for me so I figured I would document this.
Aug 06, 2013
My boss tasked me with getting the load time of 90 seconds(HOLY CARP!) on one page down. First thing I did was install the Django Debug Toolbar to see what was really happening.
There are currently 2,000 users in the database, the way our model is setup is that a UserProfile can have other UserProfiles attached to it in one of three M2M relations, which in the Django Admin would cause 2,000 queries PER M2M field. This is very expensive as obviously you don't want 10,000 queries that each take 0.3ms to take place.
The solution, after a day and a half of research is to override the formfield_for_manytomany method in the Admin class for our UserProfile object.
Our solution is to prefetch for any M2M that are related to the current Model.
def formfield_for_manytomany(self, db_field, request, **kwargs):
if db_field.__class__.__name__ == "ManyToManyField" and \
db_field.rel.to.__name__ == self.model.__name__:
kwargs['queryset'] = db_field.rel.to.objects.prefetch_related("user")
return super(UserProfileInline, self).formfield_for_manytomany(
db_field, request, **kwargs)
This goes inside our admin class UserProfileInline(admin.StackedInline). Simple clean and easy to drop into another ModelAdmin with minimal changes.
Other things I pondered was to set all our M2M's as raw_id_fields, then using Select2 or Chosen, query our UserProfiles when the related users were being selected. This would take a lot of load off the initial page load, but is more of a bandaid rather than a real fix.
I tried to override the Admin class's def queryset(self, request): but this was not affecting anything.
Jul 02, 2013
I have a friend who is interested in becoming a Python developer. He has some Python experience with CodeAcademy, but he of course wants to take this a step further and develop on his own computer. I figure I'd give him a few pointers, and I know this has been rehashed a million times, but what the hell, why not blog on it again.
There are a few important things to learn besides the actual language itself. The first I am going to discuss is has to deal with installing packages, then followed up closely with Python's path trickery. Finally I'm going to wrap up by discussing some software related to development, that could be used for any language, but I use daily in my work as a Python Software Engineer. Let's get started.
PIP
Python is a wonderful language, but how useful would it be if you had to rewrite everything by hand? Not useful at all. That's why the lovely pip developers were born. PIP (executable pip) is a package manager written for Python. It's very simple to use, and in my opinion is way better than easy_install. To use pip you need to know at a minimum three commands.
pip install
This command does exactly what it says on the box. It queries PyPI (Python Package Index) and downloads the latest version of the package on the server. It then installs it to your site-packages.
pip uninstall
This deletes all files associated with the package supplied. 100% simple.
pip freeze
This shows what packages are installed on your system and what versions. If you supply ‐‐local it will show what packages are installed in your current environment.
These three commands will get you started with package management, there are more commands you can find by looking through the help documents.
Virtualenv
If you notice I mentioned a current environment in my previous pip freeze explanation, here is why. Python has a default place that it looks when you reference a package. This is generally in something like /usr/lib/python2.7/site-packages/ or C:\Python27\lib. There is a set of scripts called virtualenv that creates an environment where you run it with a complete copy of your Python executable, and a blank (unless you copy them over) site-packages directory. You can then install any packages there activate the virtual environment. When activated you use those specific versions, no matter the version of what is installed on your system.
Let's show an example of the first time use of virtualenv:
$ sudo pip install virtualenv # Only time you might need sudo, try without first.
$ virtualenv myenv # Create the virtual environment
$ source myenv/bin/activate # Activate the virtual environment
(myenv)$ python -c "import MYPACKAGE; print MYPACKAGE"
Notice how it says your package is not in /usr/lib/python2.7/site-packages/ ? That's because you're using virtualenv to tell your copied python to use that library instead. There are many reasons you would want to use a virtual environment. The most frequent reason is to preserve version numbers of installed packages between a production and a development environment. Another reason virtualenv is useful if you do not have the power to install packages on your system, you can create a virtual environment and install them there.
Virtualenvwrapper
After you create a virtual environment, you just run``source bin/activate`` and it will activate the virtual environment. This can get tedious knowing exactly where your virtual environments are all the time, so some developers wrote some awesome scripts to fix that problem. This is called``virtualenvwrapper`` and once you use it once, you will always want to use it more. What it does is that it has you create a hidden directory in your home directory, set that to an environment variable and references that directory as the basis for your virtual environments. The installation of this is pretty easy, you can``pip install virtualenvwrapper`` if you want, or download the package and compile by hand.
Once installed correctly, you can run the command mkvirtualenv envname to create a virtual environment. You can then run``workon envname`` from anywhere, and it will activate that environment. For example, you could be at``/var/www/vhosts/www.mysite.com/django/`` and run``workon envname`` and it would activate the environment from there. This isn't a required package (none of them are really…) as I went a couple years without using``virtualenvwrapper``, but it is very useful and now I use it every day. Some tips I use with my setup of``virtualenvwrapper`` is that I use the postactivate scripts to automatically try to change into the proper project directory of my environment. This also means I usually name my``virtualenv`` after my project name for easy memory. It makes no sense to have a project called "cash_register" but the``virtualenv`` be called "fez". This is how I change to the right project after activating my virtualenv. This goes in $WORKON_HOME/postactivate
#!/bin/bash
# This hook is run after every virtualenv is activated.
# if its a work or a personal project (Example)
proj_name=$(echo $VIRTUAL_ENV|awk -F'/' '{print $NF}')
if [[ -e "/Users/tsouza/PersonalProjects/$proj_name" ]]
then
cd ~/PersonalProjects/$proj_name
else
cd ~/WorkProjects/$proj_name
fi
This about wraps up part one of this two part blog series. Next time I will discuss how to use Git and how to configure SublimeText2 and Aptana Studio for use with Python. Stay tuned!
May 25, 2012
Today I was tasked with installing CFEngine3 on CentOS-5.7 (A little outdated). When installing CFEngine-3.3.1 I kept getting an error that I couldn't find libtokyocabinet.so.9. I had to set my prefix to /usr/ because the location that tokyocabinet was installing my libraries to was not being read by CFEngine's make script.
To do this (I am using tokyocabinet 1.4.47)
wget http://fallabs.com/tokyocabinet/tokyocabinet-1.4.47.tar.gz
tar -xzvf tokyocabinet-1.4.47.tar.gz
cd tokyocabinet-1.4.47/
./configure --prefix=/usr/
make
sudo make install
Then I was able to ./configure && make && make install cfengine3 without any problems.
So my overall script looked something like this:
wget http://fallabs.com/tokyocabinet/tokyocabinet-1.4.47.tar.gz
tar -xzvf tokyocabinet-1.4.47.tar.gz
cd tokyocabinet-1.4.47/
./configure --prefix=/usr/
make
sudo make install
wget http://cfengine.com/source-code/download?file=cfengine-3.3.1.tar.gz
tar -xzvf cfengine-3.3.1.tar.gz
cd cfengine-3.3.1
./configure
make
sudo make install
sudo cp /var/cfengine/bin/cf-* /usr/bin/
May 25, 2012
I met this random kid online on IRC a year and a half ago (I believe it was December 19th, 2010). His name was HarryD. We got talking and one time he mentioned that he was going to hike Mount Monadnock. That is near me so we got talking and he said he lived near North Hampton, MA. That was cool that I met some random kid who lived near me. He was only 17 at the time.
Eventually we met up because my new girlfriend lived near NoHo in Holyoke. One day I went down to see her and met up with him. He was wearing all brown, so I joked that he was a UPS delivery guy. We hung out for a bit, I think I met his friend Sam that day. Then I left and went home. For the next year we would hang occasionally, usually I would go down and visit NoHo because it was easy for him to Bike to and he worked there. We went bowling, once or twice, and he came up for this super bowl to watch Star Wars instead because screw sports!
I think that was the last time I saw him.
On this Saturday, May 19th, 2012, he was riding his bicycle in North Hampton and collided with a car, sending him flying off and smacking his head on the edge of the curb. There are some other details which I am not 100% sure about. He then was in the hospital until Tuesday May 22, 2012 at which time he passed away.
We used to talk every day, either on IRC, AIM or gTalk. His account on IRC (no_numbers_here) is still active because his VPS for http://harry.is is still paid for and online until the next billing period. Its so sad seeing his name online and not being able to send a "hey dood" or some other random greeting, then start talking about computers, python, bikes, etc.
Harry was an avid cyclist. I am reading stories that even if he had to go thirty feet, he would hop on his bike to get there. He got me interested in cycling as well. He was going to sell me a bike, but the I was talking to a friend and he gave me on, so I never bought it. Which as it turns out was good as he wanted to give that to his new girlfriend.
I was planning on hanging out with him next weekend, as he was busy with something this weekend that I can't remember. I wanted to take him target shooting, and wanted to eventually be in enough shape to go hiking with him. None of this ever came to fruition.
Harry Delmolino, we may not have been as close as we could have been, but you definitely made a difference in my life for the better and I never got to thank you for this.
Edit:
I went to the Calling Hours today and I was only there maybe a half hour, but there were so many people there. It's amazing that a man so young has touched so many lives of so many people, and had accomplished so much. People might say "Oh we was just a kid, he was only 18″ but if they looked at the accomplishments of this young man, they would realize how grown up he was.
I think his mother and father might eventually delete his Facebook, so I went back and took a screenshot of one of our last conversations so I can remember. Everything he said usually warranted a laugh.
May 07, 2012
For the past three months I have been upgrading and rewriting version 2 of my software for Hypertherm. I am under a contract for my father's company. His company is developing a machine to test how well air flows (laminar flow) through a plasma cutting torch head, and how much air leaks out over a certain time (delta pressure loss).
This has been a nice adventure. I am talking to the tester over serial, reading in a hand scanner (barcodes and acts as a keyboard easy), talking to a DYMO printer and using a database.
The serial communication was pretty straightforward. I started a new thread and listen for serial all the time. The tricky part with that was that because it was on another thread, I needed a delegate to talk to my UI when I did things like change the picture from blank to a big red X, or update a label.
The hand scanner wasn't even a factor that took longer than 10 minutes, I just pop up a dialog box asking for input.
The DYMO printer was the hardest part. This took me a month to figure out, I kept fighting with the printer. I could figure out how to print to the left roll, the ones we setup as as the passing labels, but I couldn't for the life of me figure out how to get it to print to the right label, using a custom label. I tried to load the labels into data and use a StreamWriter/StreamReader object to treat that as the label, but it kept printing one that had, for reasons unknown to me, been locked into the printer. I finally gave up on using the interface they provided and am writing the label to a temporary file. The file is in the user's %appdata% directory in a sub directory that it will not be mistakenly written to, so I feel safe doing it this way. Granted, the machine is a single purpose machine, once this program is installed it will only run this program day and night.
Once I got the printer working, I checked it in to github and realized it took me way longer than anticipated. I learned a lot about .NET development (by no means everything, or even most things, just a lot compared to what I did know before [nothing].)
Tonight while developing I decided to video some aspects of the Program.
The following four links are videos, showing parts of the program and machine in action.
- [Hosted on Qik - no longer available]
- [Hosted on Qik - no longer available]
- [Hosted on Qik - no longer available]
May 04, 2012
Last night I wrote the beginnings of my first NodeJS application. Is application even the correct word?
I've been meaning to try out the cool API by Twilio, which is used for SMS and VoiceCalling. I decided to design a system that will be two+ endpoints. One is the main server which will listen for UDP messages. When it receives the correct UDP message, configured in the config(konphyg) files, it will fire off a message to Twilio and send me a text message.
The next steps, which I should be getting to tonight, are to create the Arduino portion and the serial listener. The Arduino will have a button that will send a message over serial to another NodeJS listener. This will decide if the press was good enough, if it passes the debouncing filter, and then fire a message to the main Ganymede server.
This could be used as a little text message doorbell for when you have your music on too loud. I don't believe I will ever sell this, as it's just for me to get into NodeJS, but It would be fun to share with friends.
The source so far is located on my github at [DEADLINK].
I will write more as the project continues about the different technologies and comment on my choices in the source a little bit.
Mar 08, 2012
I realize I haven't updated in a while. I haven't had much free time recently as I've been working on a project for my father in C# after work hours. This is a great change from only working in Python and JavaScript recently. I'm making a program that will analyze test results from a plasma torch for a company called HyperTherm. My father built the physical machine, but the employees want something that they can better see the results of a passed torch unit, or a failed torch unit. This program has a bar code scanner that scans the tool used in the test and matches it up to the lot of torch parts. Another added feature is the ability to print a white label that says "UNIT PASSED" or a giant red label that says the unit failed and which of the 8 tests failed were.I had to learn how to use delegates, as my serial event listener is on a separate thread and I can't update labels, or parts of the User Interface without them. Still working on it, hopefully will wrap it up by Saint Patrick's day.
I recently found a cool command in BASH that I hadn't previously known. C-o will execute the current line, and then bring the following line up from BASH history. If you have a set of commands you want to execute again, rather than having to press up 20 times, hit enter, press up 19 times, hit enter, and so on… You can just hit up 20 times. Press C-o as many times as you need to.
For example:
$ touch a
$ touch b
$ touch c
# [up] [up] [up]
$ touch a [C-o]
$ touch b [C-o]
$ touch c [C-o]
As you can see there, all I had to do was go back to the $ touch a line, and hit control-o three times and it touched the files again!
Feb 17, 2012
I was invited to a Hackathon that one of our client's client was throwing. Being that I love programming and learning, I decided I would go.
The event was in Cambridge, MA. I arrive early, (my friend said there would be a lot more traffic than there was at that time of day) so I got a tour of office. It's situated in an old, what I believe to be, factory building. The coolest part of the office was that they had whiteboard paint on every wall surface, complete with markers of course.
The event started and people who were attending had tossed up ideas on the white board. A couple people wanted to integrate LinkedIN with HubSpot. Another person wanted to integrate Eventbrite with HubSpot, to get information to/from event goers after the event ends. I didn't like any of those ideas and my only experience with HubSpot is their Leads API, so I stuck to what I know.
I had an idea for an app the second I walked in the door, it was like magic. My main hassle was that HubSpot's Canvas integration REQUIRES HTTPS. Now, my web host is DreamHost and I am kind of cheap, so of course I don't have any way to host a HTTPS site immediately. A big part of me wanted to bite the bullet and order a secure server from DreamHost, or setup another linode, but I felt that I've been spending a lot of money lately and that I would figure out a way. Adrian, my contact at HubSpot, of who I am working with on the PPC project(more on that later), walked by and saved me.
He asked if I had ever used GoogleAppEngine. Of course I hadn't because I was under the belief that it cost money to use, but then I realized I was thinking of Amazon's EC2. I sign up for GAE and within an hour I have a HelloWorld site setup. The slow part was installing Python2.5 so I could use the same version that GAE used and not have to fix a lot of backwards compatibility errors between 2.5->2.7.
After I had a site up that could do HTTPS I dove into programming for my HubSpot app. The app I am doing for work graphs leads per day combined with Google AdWords data per day. I decided to do something different. My app is still a graph, as graphs are fun and easy to understand by everyone.
This app graphs a set of leads and shows how many leads happened in a given hour for the previous day. Given extra time I would have added an interface to specify the day to graph leads, but last night my time was severely limited by the fact that I had to setup my environment for GoogleAppsEngine.
Improvements I can and want to do to this app are database, faster processing, and being able to select a date. I almost wanted to break down and learn NodeJS for this, because from my understanding of the event driven nature of NodeJS would be a lot easier to load data over a longer period of time, than to just load it all at once and timeout with HubSpot's Jakarta Commons-HttpClient.
Feb 08, 2012
I am working with txStatsD and Graphite. I was having the hardest problem looking through the txStatsD code today finding how to graph something as an event, not a data point. I eventually went into every option on the graphite dashboard and found an option to make a bar.
This is the option that you must use when you want to mark events. For example we want to know "Server restarted", we would use this technique, as it doesn't make sense to aggregate "server restarted". Using nonzero as infinite is a good way to show an event took place.
