Oct 01, 2014
Recently I decided I wanted to learn Java again. I last programmed in Java when I was in College and that was the main language they taught in. I wouldn't say I was a great Java developer, although I completed every Java course well enough to get an A or better.
I want to relearn Java because for the past four years I have primarily focused on Python. While it is a great language, I feel I need a change from what I'm focusing on now with primarily web based programming.
I decided to refresh myself with Java and read a "Java for Python developers" guide, which was a great refresher. After that I sat around wondering what to program, inspiration wasn't coming quickly. I settled on a SSH Configuration Manager, which is something I've wanted for a while now.
This Configuration Manager will read in your ~/.ssh/config files, and show you what hosts you have in a GUI interface. The great part of it will be that you can also create new ssh configurations, without having to remember every little detail. There will be a lot of help tooltips, and pre-fills as well. I have a pretty basic idea of what I want it to look like. Ideally a list on the far left with +/- buttons to add a new Host, and to the right of that will be another hierarchy list of all the key groups you can change, with the most common (that I or people I talk to) being in a "General" or "Common" list. To the right of that will be the actual keys and values you change. I think I would like to be able to "favorite" keys that you use frequently. This way when you create a new host entry, you can quickly fill out your usual configurations be it only adding an IdentityFile and User. Another feature I thought of would be copying/templating, for example being able to create a new "work based server" configuration by just copying one you already have.
Some of the options will be a bit tricky, a couple of them are along the lines of allowing "yes", "no", "ask", or an integer, and I haven't figured out exactly how I want to manage that yet.
Currently I have a model that only has getters/setters and toString support, there's a lot of them so it's already a 1050 line file last I checked. Next time I work on this project I want to start with data validation and learning how to write tests in Java. I think learning good BDD or TDD habits while learning a "new" language would definitely benefit me.
Jun 21, 2014
A lot of times when I stop at someone's computer and help them in the terminal, I use a Readline command and people say "How the heck did you do that?"
Let me first backup and explain what Readline is. From the GNU Readline Documentation - "The GNU Readline library provides a set of functions for use by applications that allow users to edit command lines as they are typed in." By default, Readline is set up in Emacs mode, no you don't have to have an extra four fingers to use Readline, most of the commands are simple.
Here are a couple of the commands I use daily:
Movement
- To move to the beginning of a line, you press C-a
- To move to the end of a line you press C-e
Killing and Yanking
- To cut the rest of the line from where your cursor is, to the end, you press C-k
- To delete one word you press C-w
- To paste either of the two previous back you can press C-y
Miscellaneous
- To clear the screen and get to a fresh start, you can press C-l
- To end your session you can send a C-d (This will send an end of file character)
- To search for a command you typed recently, press C-r and start typing, it will search backwards. C-r again will search for an earlier match.
- The inverse of C-r is C-s, they function the same.
- To open your $EDITOR to edit the current shell command you wish to write, press C-x C-e
Finally, don't forget about C-c. While not specifically Readline, it's very useful because it sends the SIGINT signal to the program, which if just on the command line, will not execute the line you have type, and give you a new line with nothing on it. A nice clean start.
To find out a lot more, read the documentation at the Readline Commands Docs I even learned some things while writing this up, apparently pressing C-x $ will list off all the possible usernames. Good to know, and good to always keep learning.
Nov 13, 2013
Recently I have been diving into using signals with Django, which of course are pretty neat.
I am working on a website for work which in the most basicexplanation, is a task management site. Recently I have added in the ability to subscribe to tasks and get emails, I did this by connecting to the post_save signal. I only email out when a task is changed, not created (of course, no one would be subscribed to it). This worked flawlessly and "emails" out to anyone who is subscribed. I say that in quotes, because I haven't actually hooked it up to a real SMTP server, and only use
python -m smtpd -n -c DebuggingServer localhost:1025
which will output any emails to stdout. But I digress… A problem arose when I was working on ordering tasks.
I store an integer in the "ordering" column, which any authenticated user can drag the row to a new location and that will reorder the task. I did this after I setup the emailing signal, so I didn't think about an email being sent out for EVERY task being changed.
I tried a lot of different things, and was debating some that would be a bit messy. Among those ideas were trying to store the past values in another table, but that would get expensive fast. The reason I tried this was because I wanted to see if the ordering was the only thing that changed, and if that was the case, not send an email. I eventually found a thread on StackOverflow that says to use update on the queryset to not trigger the signal.
You can do this by doing something like this:
from app.models import ModelName
def reorder(request):
new_order = request.POST.get('new_order', None)
pk = request.POST.get('modelname_pk', None)
if new_order:
ModelName.objects.filter(pk=pk).update(ordering=new_order)
I am not sure if this is the proper way save changes and not trigger a post_save signal, but this is the way that worked for me so I figured I would document this.
Aug 06, 2013
My boss tasked me with getting the load time of 90 seconds(HOLY CARP!) on one page down. First thing I did was install the Django Debug Toolbar to see what was really happening.
There are currently 2,000 users in the database, the way our model is setup is that a UserProfile can have other UserProfiles attached to it in one of three M2M relations, which in the Django Admin would cause 2,000 queries PER M2M field. This is very expensive as obviously you don't want 10,000 queries that each take 0.3ms to take place.
The solution, after a day and a half of research is to override the formfield_for_manytomany method in the Admin class for our UserProfile object.
Our solution is to prefetch for any M2M that are related to the current Model.
def formfield_for_manytomany(self, db_field, request, **kwargs):
if db_field.__class__.__name__ == "ManyToManyField" and \
db_field.rel.to.__name__ == self.model.__name__:
kwargs['queryset'] = db_field.rel.to.objects.prefetch_related("user")
return super(UserProfileInline, self).formfield_for_manytomany(
db_field, request, **kwargs)
This goes inside our admin class UserProfileInline(admin.StackedInline). Simple clean and easy to drop into another ModelAdmin with minimal changes.
Other things I pondered was to set all our M2M's as raw_id_fields, then using Select2 or Chosen, query our UserProfiles when the related users were being selected. This would take a lot of load off the initial page load, but is more of a bandaid rather than a real fix.
I tried to override the Admin class's def queryset(self, request): but this was not affecting anything.
Jul 02, 2013
I have a friend who is interested in becoming a Python developer. He has some Python experience with CodeAcademy, but he of course wants to take this a step further and develop on his own computer. I figure I'd give him a few pointers, and I know this has been rehashed a million times, but what the hell, why not blog on it again.
There are a few important things to learn besides the actual language itself. The first I am going to discuss is has to deal with installing packages, then followed up closely with Python's path trickery. Finally I'm going to wrap up by discussing some software related to development, that could be used for any language, but I use daily in my work as a Python Software Engineer. Let's get started.
PIP
Python is a wonderful language, but how useful would it be if you had to rewrite everything by hand? Not useful at all. That's why the lovely pip developers were born. PIP (executable pip) is a package manager written for Python. It's very simple to use, and in my opinion is way better than easy_install. To use pip you need to know at a minimum three commands.
pip install
This command does exactly what it says on the box. It queries PyPI (Python Package Index) and downloads the latest version of the package on the server. It then installs it to your site-packages.
pip uninstall
This deletes all files associated with the package supplied. 100% simple.
pip freeze
This shows what packages are installed on your system and what versions. If you supply ‐‐local it will show what packages are installed in your current environment.
These three commands will get you started with package management, there are more commands you can find by looking through the help documents.
Virtualenv
If you notice I mentioned a current environment in my previous pip freeze explanation, here is why. Python has a default place that it looks when you reference a package. This is generally in something like /usr/lib/python2.7/site-packages/ or C:\Python27\lib. There is a set of scripts called virtualenv that creates an environment where you run it with a complete copy of your Python executable, and a blank (unless you copy them over) site-packages directory. You can then install any packages there activate the virtual environment. When activated you use those specific versions, no matter the version of what is installed on your system.
Let's show an example of the first time use of virtualenv:
$ sudo pip install virtualenv # Only time you might need sudo, try without first.
$ virtualenv myenv # Create the virtual environment
$ source myenv/bin/activate # Activate the virtual environment
(myenv)$ python -c "import MYPACKAGE; print MYPACKAGE"
Notice how it says your package is not in /usr/lib/python2.7/site-packages/ ? That's because you're using virtualenv to tell your copied python to use that library instead. There are many reasons you would want to use a virtual environment. The most frequent reason is to preserve version numbers of installed packages between a production and a development environment. Another reason virtualenv is useful if you do not have the power to install packages on your system, you can create a virtual environment and install them there.
Virtualenvwrapper
After you create a virtual environment, you just run``source bin/activate`` and it will activate the virtual environment. This can get tedious knowing exactly where your virtual environments are all the time, so some developers wrote some awesome scripts to fix that problem. This is called``virtualenvwrapper`` and once you use it once, you will always want to use it more. What it does is that it has you create a hidden directory in your home directory, set that to an environment variable and references that directory as the basis for your virtual environments. The installation of this is pretty easy, you can``pip install virtualenvwrapper`` if you want, or download the package and compile by hand.
Once installed correctly, you can run the command mkvirtualenv envname to create a virtual environment. You can then run``workon envname`` from anywhere, and it will activate that environment. For example, you could be at``/var/www/vhosts/www.mysite.com/django/`` and run``workon envname`` and it would activate the environment from there. This isn't a required package (none of them are really…) as I went a couple years without using``virtualenvwrapper``, but it is very useful and now I use it every day. Some tips I use with my setup of``virtualenvwrapper`` is that I use the postactivate scripts to automatically try to change into the proper project directory of my environment. This also means I usually name my``virtualenv`` after my project name for easy memory. It makes no sense to have a project called "cash_register" but the``virtualenv`` be called "fez". This is how I change to the right project after activating my virtualenv. This goes in $WORKON_HOME/postactivate
#!/bin/bash
# This hook is run after every virtualenv is activated.
# if its a work or a personal project (Example)
proj_name=$(echo $VIRTUAL_ENV|awk -F'/' '{print $NF}')
if [[ -e "/Users/tsouza/PersonalProjects/$proj_name" ]]
then
cd ~/PersonalProjects/$proj_name
else
cd ~/WorkProjects/$proj_name
fi
This about wraps up part one of this two part blog series. Next time I will discuss how to use Git and how to configure SublimeText2 and Aptana Studio for use with Python. Stay tuned!
May 04, 2012
Last night I wrote the beginnings of my first NodeJS application. Is application even the correct word?
I've been meaning to try out the cool API by Twilio, which is used for SMS and VoiceCalling. I decided to design a system that will be two+ endpoints. One is the main server which will listen for UDP messages. When it receives the correct UDP message, configured in the config(konphyg) files, it will fire off a message to Twilio and send me a text message.
The next steps, which I should be getting to tonight, are to create the Arduino portion and the serial listener. The Arduino will have a button that will send a message over serial to another NodeJS listener. This will decide if the press was good enough, if it passes the debouncing filter, and then fire a message to the main Ganymede server.
This could be used as a little text message doorbell for when you have your music on too loud. I don't believe I will ever sell this, as it's just for me to get into NodeJS, but It would be fun to share with friends.
The source so far is located on my github at [DEADLINK].
I will write more as the project continues about the different technologies and comment on my choices in the source a little bit.
Mar 08, 2012
I realize I haven't updated in a while. I haven't had much free time recently as I've been working on a project for my father in C# after work hours. This is a great change from only working in Python and JavaScript recently. I'm making a program that will analyze test results from a plasma torch for a company called HyperTherm. My father built the physical machine, but the employees want something that they can better see the results of a passed torch unit, or a failed torch unit. This program has a bar code scanner that scans the tool used in the test and matches it up to the lot of torch parts. Another added feature is the ability to print a white label that says "UNIT PASSED" or a giant red label that says the unit failed and which of the 8 tests failed were.I had to learn how to use delegates, as my serial event listener is on a separate thread and I can't update labels, or parts of the User Interface without them. Still working on it, hopefully will wrap it up by Saint Patrick's day.
I recently found a cool command in BASH that I hadn't previously known. C-o will execute the current line, and then bring the following line up from BASH history. If you have a set of commands you want to execute again, rather than having to press up 20 times, hit enter, press up 19 times, hit enter, and so on… You can just hit up 20 times. Press C-o as many times as you need to.
For example:
$ touch a
$ touch b
$ touch c
# [up] [up] [up]
$ touch a [C-o]
$ touch b [C-o]
$ touch c [C-o]
As you can see there, all I had to do was go back to the $ touch a line, and hit control-o three times and it touched the files again!
Feb 08, 2012
I am working with txStatsD and Graphite. I was having the hardest problem looking through the txStatsD code today finding how to graph something as an event, not a data point. I eventually went into every option on the graphite dashboard and found an option to make a bar.
This is the option that you must use when you want to mark events. For example we want to know "Server restarted", we would use this technique, as it doesn't make sense to aggregate "server restarted". Using nonzero as infinite is a good way to show an event took place.
Jan 13, 2012
Today we had a problem at work on a system.
Without getting into too much detail as to give away secrets behind the verbal NDA I am behind, I will just say that it had to do with a GPG public key of mine that was expired on a dev machine, accidentally propagating during install to a production machine.
This key had a sub key as well, so figuring out this was tricky.
To start, you can list your gpg keys like so:
This will list keys such as
pub 4096R/01A53981 2011-11-09 [expires: 2016-11-07]
uid Tyrel Anthony Souza (Five year key for email.)
sub 4096R/C482F56D 2011-11-09 [expires: 2016-11-07]
To make this not expire, (same steps to change expiration date to another time), you must first edit the key
$ gpg --edit-key 01A53981
You will then see a gpg prompt gpg>
Type "expire" in and you will be prompted for how long to change it to
Changing expiration time for the primary key.
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
You are then done setting the expiration on the primary key, if you have sub key, doing this is as easy as typing key 1 and repeating the expiration step.
To finish and wrap things up, type save and you are done.
Jan 05, 2012
For work I had to write a custom url model field. This model field when setting up accepts a default protocol, and a list of other protocols.
When checking the protocol, the url is split by "://". If the split has one or two parts, then the url is validly formed.
In the event of a single element split, there is no protocol specified. When there is no protocol, the url is prepended with the default protocol specified. If there is a protocol, it is checked to make sure it exists in a union of the default protocol and other protocols. If it is not, a ValidationError is raised letting the user know that the protocol is not accepted.
This can all be found at On my github [deadlink].
I have a couple ways I could have done this better and probably will. Improvements would be just one parameter called parameters in which it is checked if there is at least one element. Passing this, when there is no protocol specified, the first element is the default one.
This would be a little cleaner.
this example would allow for http, https, ssh, spdy and mailto, anything else would error out.
facebook_page = URLField(default_protocol="http", protocols=["https","ssh","spdy","mailto"])
The way I could improve this would be
facebook_page = URLField(protocols=["https","https","ssh","spdy","mailto"])
