Skip navigation

Tag Archives: python

Fixing MySQLdb encoding issues with Python

Are you running into encoding issues with MySQLdb on Python? Have you searched high and low only to be told to change your my.cnf settings and create tables with the proper encoding, yet it STILL doesn’t work? In that case, be sure to specify charset='utf8' in your MySQLdb.connect(...) along with the rest of the required parameters. This tiny, neglected setting is what’s actually used to encode unicode back for MySQL. Not configuration files, not table settings, just that.

Of course, you should have a table with a utf8 charset for it to work, but this is the setting that will prevent those latin-1 encoding issues when you swore you didn’t say latin-1 anywhere.

eDonkey2000 Hash in Python

As part of a filesystem watcher program I maintain as part of a larger site, I need to obtain the ed2k hash of some files. Believe me when I tell you I have searched high and low for an Edonkey2000 hash algorithm in python. I’ve seen massive libraries that scared the bejesus out of me, but nothing clean, concise, and without external dependencies. Enjoy the below. It’s worth noting it’s just as fast as the C version.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

import hashlib
 
def hash_file(self, file_path):
    """ Returns the ed2k hash of a given file. """
 
    md4 = hashlib.new('md4').copy
 
    def gen(f):
        while True:
            x = f.read(9728000)
            if x: yield x
            else: return
 
    def md4_hash(data):
        m = md4()
        m.update(data)
        return m
 
    with open(file_path, 'rb') as f:
        a = gen(f)
        hashes = [md4_hash(data).digest() for data in a]
        if len(hashes) == 1:
            return hashes[0].encode("hex")
        else: return md4_hash(reduce(lambda a,d: a + d, hashes, "")).hexdigest()

Django On-Demand Model Fields

I’m writing an Inventory system for work in my downtime, part to brush up my rusting Django skills, part because we really need one.

We started with a few requirements for what would make a good inventory system to meet our needs, and one of those was that it would be painless to add new types of items and have the ability to customize them. If you know much about Django models, you probably know enough to know that models are typically hard-baked. If you’re writing a blog system, you might create a model with fields for Title, Author, Date, Text. Well for us, what if I want Widget X to have a field for Serial Number, and Widget Y to have a field for Firmware?

Well, one option is to write a model for each item. This unfortunately breaks both requirements- new item types must be added by the coder (me), and any changes to item properties also have to go through me. I don’t want to have to run constant maintenance here, so no thanks, I’ll pass on that.

What I’ve come up with instead are two models. One called ItemType, which consists of 3 fields: name, display, and custom_fields. The first two are self-explanatory. The last field is typically shown like this:

firmware=DDWRT,Stock|model=Linksys,Cisco|mac_address

The second model is just called Item and contains common properties all items have: ID, Condition, Notes, etc; ItemType is a foreign key here.

A rather customized (but short and sweet!) ModelForm parses the custom_fields for the item type of a particular Item, and adds the fields (combobox or textfield depending on if you supplied choices), and the backend is managed by django-expando so we can even have these custom properties in the first place.

I ran into a stupid Django bug when coding this, which prevented runtime fields from displaying in the ModelAdmin. The patch I’ve documented and described, but I’ve really little hope upstream will notice or even fix it. But should you want to do this in the future and it doesn’t work (e.g., it was never fixed), that’s what you’ll need to do.

There’s also this small, nasty recursion error expando is spawning off for reasons I’ve yet to figure out. I’ll nail it eventually. (UPDATE: upstream django-expando has fixed this, hoo-rah!)

But regardless, the dynamic typing system for this works really well and I’m kind of surprised and happy it turned out to be a success. Should it get to be proper production-level code, I’ll look into opensourcing it. For now, just get a hold of me if you’ve any questions about how to make this solution work.

So, Passenger is actually pretty useful

I have to admit, I’d heard of Passenger/mod_rails before, but I only found out yesterday by chance that it also handles WSGI. Since most of any serious work I do is with Python, this caught my attention. Passenger turned out to be simple to setup, and braindead-simple to get running.

I’ve gone through several iterations of how to run my WSGI, from Apache mod_wsgi to Nginx mod_wsgi to Nginx proxy + CherryPy, and now back to Apache “mod_rails”. As an aside, Nginx mod_wsgi sounds like a good idea, but it isn’t. The author has written about this as well, and I encourage you to Google for more information.

One thing to remember when you setup your WSGI app with Passenger is that your passenger_wsgi.py file should be in the directory above your DocumentRoot. So if your docroot is /var/www/my.site/public, put the .py in /var/www/my.site/

I also run two redmines, so this simplifies that as well, since Passenger does rails primarily.

Goodbye, Nginx TCP proxying.