Fancier logging in Python (with module and method/function names)

As Python programmers most of us know the old good logging module in Python standard library. This is somewhat really good and useful library for debugging and logging the behavior of the program, but I found it somewhat lacking.

What I needed was to be able to log what function or class method issued the log.debug() and in what line. This was needed by me for asynchronous and multithreaded networking library project I’m working on.

The solution was to write my own custom logging.Logger class implementation that overrides it’s debug method with some fancy inspect module tricks. Works like charm! Try it by yourself.

import logging
import inspect

logging.basicConfig(level = logging.DEBUG)

class MyLogger(logging.Logger):
    
    def debug(self, msg, *args, **kwargs):
        method = inspect.stack()[1][3]
        frm = inspect.stack()[1][0]
        if 'self' in frm.f_locals:
            clsname = frm.f_locals['self'].__class__.__name__
            method = clsname + '.' + method
        if not method.startswith('<'):
            method += '()'
        msg = ':'.join((method, str(frm.f_lineno), msg))
        self.__class__.__bases__[0].debug(self, msg, *args, **kwargs)

logging.setLoggerClass(MyLogger)
getLogger = logging.getLogger

Voila! Now, after instancing this new MyLogger class, you can use it just like normal Logger class, except for the bonus of having module name, function or method names and line number in your log.

Here's a simple debug output from this class:


DEBUG:network.connpool:Connection.analyze():187:Found packet: [0, 1, 0, 42, 0]
DEBUG:network.connpool:Connection.analyze():187:Found packet: [0, 1, 1, 69, 0]
DEBUG:network.server:PacketHandler.handle_close():102:Closing connection id: 139697871065392
DEBUG:core.event:emit():153:Emitted net.conn "CONN_DROP"
DEBUG:core.event:EventManager.process():194:Sending event net.conn (CONN_DROP) to ConnectionPool
DEBUG:core.event:ConnectionPool.notify():86:ConnectionPool got event of type "net.conn"
DEBUG:network.connpool:ConnectionPool.remove():294:Closing connection id: 139697871065392
DEBUG:network.connpool:ConnectionPool.parse_packets():332:Removing connection id: 139697871065392

Posted in Programming, Python | Tagged | Leave a comment

Python GUI in Linux frame buffer

I was recently wondering about how can I display some graphics in Linux frame buffer with Python. It seems that if your text terminal is initialized in frame buffer mode (most of modern distributions do this), what you need is our old friend PyGame. It supports a wide selection of video drivers among which there is also some for utilizing frame buffer. If you want to play with or develop GUI applications in Python without X Window then here’s a simple example how.

PyGame checks for environment variable SDL_VIDEODRIVER that specifies which video driver PyGame should utilize for initializing display. This can come handy if you are running pure PyGame based application and want to use it if you don’t have or want X.org. Unfortunately this also means that you have to manually export SDL_VIDEODRIVER variable before running your application. There’s a little trick you can do for automatically determining whether you want to run your app in frame buffer.

When application is run within X.org context, the DISPLAY environment variable is passed to your application (in fact it’s set in the system context for the app, but that doesn’t matter). You can test if app was run from X Window system like this:

import os
disp_no = os.getenv('DISPLAY')
if disp_no:
    print "I'm running under X display = {0}".format(disp_no)

 

When we know if we have X Window running or not we can act accordingly. What we want to do in our app is to check for the X display first, the check if SDL_VIDEODRIVER is set and if not we want to set it automatically. This is how we do it:

driver = 'directfb'
if not os.getenv('SDL_VIDEODRIVER'):
    os.putenv('SDL_VIDEODRIVER', driver)

 

As PyGame documentation states there are different video drivers available under Linux: x11, dga, fbcon, directfb, ggi, vgl, svgalib, aalib. Obviously we are interested only in three: directfb, fbcon and svgalib. Those are the drivers for console frame buffers. Which should you choose depends on the Linux distribution you have. Ideally you want directfb, because it’s probably the most advanced and modern type of frame buffer library. You can either let your users choose the appropriate driver before you launch your primary app or you can test which is the most suitable. This is really simple:

drivers = ['directfb', 'fbcon', 'svgalib']

found = False
for driver in drivers:
    if not os.getenv('SDL_VIDEODRIVER'):
        os.putenv('SDL_VIDEODRIVER', driver)
    try:
        pygame.display.init()
    except pygame.error:
        print 'Driver: {0} failed.'.format(driver)
        continue
    found = True
    break

if not found:
   raise Exception('No suitable video driver found!')

 

This will automatically try all drivers in the list and stay with the first one that works. If none is found an Exception will be raised.

When we found appropriate driver we now have to initialize our screen and display mode. With frame buffer we have to determine it’s resolution before we can appropriately set mode. We can do this like this:

size = (pygame.display.Info().current_w, pygame.display.Info().current_h)
screen = pygame.display.set_mode(size, pygame.FULLSCREEN)

 

pygame.display.Info() is a special object that carries information about current display. We use current_w and current_h variables to set new size for PyGame display. If we didn’t do it, we would not be using the whole frame buffer resolution which would look awkward. I’m not sure if it’s possible to switch between frame buffer modes after fb console terminals were initialized.

After we’ve called pygame.display.set_mode() we can proceed with our application development as usually.

Posted in Game Programming, Linux, Programming, Python | Tagged , , , , , , , , | 21 Comments

Paster server encoding problems

If you have stumbled upon strange unicode encoding errors within Paste server like this one:

  File "/usr/lib/python2.6/site-packages/paste/httpserver.py", line 437, in handle_one_request
    self.wsgi_execute()
  File "/usr/lib/python2.6/site-packages/paste/httpserver.py", line 290, in wsgi_execute
    self.wsgi_write_chunk(chunk)
  File "/usr/lib/python2.6/site-packages/paste/httpserver.py", line 150, in wsgi_write_chunk
    self.wfile.write(chunk)
  File "/usr/lib64/python2.6/socket.py", line 292, in write
    data = str(data) # XXX Should really reject non-string non-buffers
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in position 1280: ordinal not in range(128)

 

You can easily fix those encoding bugs typing into config/environment.py (if you are using Pylons):

# Workaround stupid Paster encoding bugs
import sys
reload(sys)
sys.setdefaultencoding('utf-8')

 

just before def load_environment() function. Module reload is needed, because sys.setdefaultencoding() is sometimes not accessible from command-line. This will make your encoding problems within Paster go away.

Posted in Coding, Programming, Pylons, Python | Tagged , , , , , , | 2 Comments

Pylons, ContextObj attribute error, strict_c and fix

I’ve recently upgraded my Pylons to version 1.0 for the CMS project I’m working on and few things broke as a result. There were some changes in dependency packages as well, most notably changing of url_for to url which had to be replaced everywhere within the code.

However the most irritating problem was with attributes of template context, mainly because of their absence in the templates. Template context in previous versions of Pylons worked in a manner that whenever there was no attribute in the template context object “c” it would be created on the fly being an empty string. Pylons 1.0 changed this behavior so you had to explicitly create the attribute. Given the scale of the project I’m not willing to go through each template out of more than hundred and then check whether or not I forgot some attribute.

Googling a little bit gave answer that you had to insert this line into config/environment.py:

    config['pylons.strict_c'] = False

just before your template customization, but it didn’t work!

I’ve been looking for an answer to this problem for quite some time and found that if you change the above line from pylons.strict_c to:

    config['pylons.strict_tmpl_context'] = False

it will happily work. So there you are. 🙂

Posted in Coding, Programming, Pylons, Python | Tagged , , , , , | Leave a comment

How to get class method name in Python?

There are two ways to do this. You can either do:


def whoami():
    """Returns name of the method."""
    import sys
    return sys._getframe(1).f_code.co_name

def callersname():
    """Returns name of the caller."""
    import sys
    return sys._getframe(2).f_code.co_name

Or you can simply do:


import inspect
inspect.stack()[0][3]

This will give you method name. Pretty simple?

Posted in Coding, Programming, Python | Tagged , , | Leave a comment

PocketSpinx voice recognition with GStreamer and Mandriva

I’ve recently bumped upon an open source speech recognition software for Linux based on the CMU Sphinx project called PocketSphinx. Unfortunately standard Mandriva repositories doesn’t have this PocketSphinx library in the 64 bit arch version. So I’ve downloaded SphinxBase and PocketSphinx and compiled them myself, by the usual means of ./configure –prefix=/usr –libdir=/usr/lib64 && make && make install

It all went well, except for the little fact I have noticed when trying to use some examples on how to use PocketSphinx from Python (yes, there are bindings for it). Unfortunately, the compilation omitted GStreamer plugin for PocketSphinx which was needed. I’ve figured out that you have to install from URPMI:
urpmi lib64gstreamer-plugins-base0.10-devel
and then recompile PocketSphinx (remember to do “make clean” first, or else nothing will change after reconfiguring). The plugin compiled successfully and I can now use PocketSphinx from Python.

Posted in Mandriva, Python | Tagged , , , , | 1 Comment

Using SQLAlchemy-Migrate with Elixir model in Pylons

I’ve recently come across the database schema migration toolkit that builts on top of the awesome SQLAlchemy ORM library named SQLAlchemy-Migrate. Since I’m working mostly on Pylons projects with extensive schema written using SQL Elixir, I was wondering if I could make SQLAlchemy-Migrate to work with my Elixir models. I’ve managed to make it work perfectly. But before I tell you how, some quick review of the tools and libraries for those of you new to the world of SQLAlchemy.

SQLAlchemy is an Object-Relational Mapper, ORM in short, that maps Python object classes to database objects, i.e.: tables, records, foreign-keys and all the other related stuff. Since SQLAlchemy is an ORM, you don’t have to write sometimes complicated SQL code for querying database. Instead you are working with Python objects and methods. SQLAlchemy adds overhead to SQL query execution times performance, but this tradeoff most of the time is worth it. Trust me. And besides, SQLAlchemy makes your project database engine independent, at least in theory.

Elixir is a declarative-style layer on top of SQLAlchemy that utilizes so called ActiveRecord design pattern. It allows you to use some shortcut methods known mostly from Rails ORM. Elixir is a library that fully interfaces SQLAlchemy and wraps some of the methods and declarations into one Entity. SQLAlchemy as of version 0.6 has declarative layer extension, but it doesn’t have those ActiveRecord methods. Using Elixir is easier than bare SQLAlchemy, but I should warn you, that without understanding SQLAlchemy first you’d be really lost. Elixir is much harder to debug when you stumble upon some kind of non-trivial error, since it’s a wrapper on top of SQLAlchemy.

However Elixir, with some caution and understanding on what you’re doing, can be really satisfactory to work with. Currently I’m working on advanced content management system and I can assure you that it’s model is really complicated. Without Elixir I would be lost long time ago, but since it’s very clean and transparent, just a quick glance at a class based on Elixir Entity gives you complete picture of what’s going on. I very much like it and would recommend using it, especially because you can mix Elixir and SQLAlchemy without any problems provided you know what you’re doing. But enough of this.

So. SQLAlchemy-Migrate. What is it? As I’ve said at the beginning it’s a database schema versioning toolkit built on top of SQLAlchemy. It doesn’t support Elixir out of the box, but with a little trickery you can make it work. What is database schema versioning? Well. It’s a record of changes in database structure between some arbitrary “points in time” called versions. Let’s suppose you have a table in Elixir model defined like this:

class Page(Entity):
    id = Field(Integer, primary_key = True)
    title = Field(Unicode(100))
    content = Field(UnicodeText, required = True)

 

This is fairly basic Elixir entity that really works. That’s all. You just run setup_all() method from Elixir and you can use this entity in your code querying, updating, inserting and making changes in database. Let’s assume that you use this model and it behaves really well, but when you develop your application further, you find it rather limiting.

For example you wish you could add a new field, so the entity would automatically update creation date and time on inserting new Page record to database. You can alter your model’s Entity like this:

class Page(Entity):
    id = Field(Integer, primary_key = True)
    title = Field(Unicode(100))
    content = Field(UnicodeText, required = True)
    ctime = Field(DateTime, required = True, default = datetime.datetime.now())

 

It will work without any problems, but… Not on existing database. You would either have to issue SQL query directly on the database to bring it up to date with your defined model (i.e. add new column ctime) OR in case of Pylons run paster setup-app which will drop your tables and recreate database from scratch. This is acceptable for new project or development databases, but not on testing (to some degree) or especially production ones, since you would loose all your data.

Here comes SQLAlchemy-Migrate. It’s an extension and toolset built on and working with SQLAlchemy models. It’s intended for schema migration, which is exactly resolving our problem changing and keeping up to date our database table’s structure with our application’s model. Migrate would create repository of changes for your application and create additional table in the database called migrate_version. After creating the repository your database would be at version zero. Official documentation states that this version should represent empty database, in our case empty model.

I’ll digress here a little bit. I’m mostly agreeing with this approach, because it ensures that you’ll build your model from the scratch and then use Migrate to upgrade it to current or desired schema version. There are however some cases when I find it unacceptable and not worth the work. One such case is the CMS I’m currently working on. It’s almost three years worth of work, containing nearly hundred tables, numerous Entity mix-ins, very deep relations of ManyToOne, OneToMany, ManyToMany and other, each table and model class has multitude of methods, decorators and much much more. Rewriting this model according to official documentation guidelines would be really hard work giving virtually no benefits.

What I wanted was to have an established baseline database model that could be extended for this CMS and changes could be committed to the baseline version and further. I need such kind of arrangement, because current version of this application with baseline model is deployed in production for few clients, that I have an agreement with for upgrade, when new versions of the software would be available. This way I could write migration scripts and upgrade their system instances to the latest version, without all the tedious work on writing alternation and schema change scripts in pure SQL, dumping current data and readjusting it to new schema.

SQLAlchemy-Migrate allows me to do this, because each change is registered sequentially as new version of the schema. For each change I write a script with two methods: upgrade and downgrade. First one brings changes up to date and the second drops those changes if you would like to return to previous version of the database schema.

Basically setting a baseline model is a good idea. Official Migrate docs state that database should be empty, but they don’t restrict it such as version zero of migration repository must be empty. Probably it’s related to the problems that might occur when working in teams and you would need to quickly bootstrap new application instance, but I won’t digress here. You can find much information on this topic in the Internet.

The only problem might be that with baseline model must be always in current version of migrate repository. Using your application models IS NOT recommended. There are many problems that can arise from such arrangement and that’s why, according to the Zen Of Python: “Explicit is better than implicit” which means that you have to keep two similar models – one in your application and one in the repository.

Yes. That’s somehow doubling the work, but when you setup a baseline just like me, you can simply track only changes in your migration repository model, updating application model accordingly. Shouldn’t be much work and it’s a good tradeoff for a comfort of having tested and verified database structure upgrade path. And most of those problems that can arise for keeping in sync your two models could be mitigated to some degree by using Mercurial or some other version control system.

Phew. Let’s get back to our problem. How to make SQLAlchemy-Migrate work with Elixir? Let’s suppose that you’d be interested in adding migration facilities to your existing project. I’d assume that you installed sqlalchemy-migrate via pip or easy_install. If so, you’re ready to go.

This is how I’ve setup my migrate repository. First step is to create a migration repository. I have assumed that it will reside under my project’s root directory. The project is under VirtualEnv control and it’s main directory is a subdirectory within the VirtualEnv root. So from my main project directory I have issued:

migrate create migration "Project Repo"

 

What this basically does is to create a migration subdirectory in the main project’s directory that will hold the sqlalchemy-migrate repository named “Project Repo”. After successful repository creation you must put your database under control. If you’re using Pylons, check out development.ini option sqlalchemy.url, which will give you necessary credentials for your database. To put your database under migrate version control you have to issue:

migrate version_control postgres://login:password@host/database

 

This will initialize your database for the version control with migrate repository version zero. You also can create a support script which will hold the credentials for your database, so you wouldn’t have to pass those to the migrate script. It’s a convenience script which wraps around migrate. We’ll call it migrate.py and it can be automatically created by typing in this command:

migrate manage manage.py --repository=migration --url=postgres://login:password@host/database

This will create migrate.py script in your project’s root directory, for repository residing under migration subdirectory and with credentials specified by –url option. Note that I’m using PostgreSQL engine, but you can use whatever engine that SQLAlchemy supports. Note however that SQLite doesn’t support full ALTER statements in SQL, so you might not get results that you’ve expected when using sqlalchemy-migrate. Now you can use:

python manage.py

 

as a shortcut for your migration repository administration. You can issue:

python manage.py version

 

to check at what version your migration repo is at and you can also use:

python manage.py db_version

 

to see what version of the migration model your database is currently at.

Ok. So we’re finally getting to the juicy part. We have our migration repository setup and ready to use. Let’s establish that our baseline model would be the first version of Page Elixir Entity presented in this post (this without ctime field). We want to create a script that will allow us to create a script for updating our baseline model with this option. Let’s say that we would name this change “Add ctime DateTime field to Page”. We can issue:

python manage.py script “Add ctime DateTime field to Page”

This will create a script named migration/versions/001_Add_ctime_DateTime_field_to_Page.py in the migration/versions subdirectory. It will look like this:

from sqlalchemy import *
from migrate import *

def upgrade(migrate_engine):
    # Upgrade operations go here. Don't create your own engine; bind migrate_engine
    # to your metadata
    pass

def downgrade(migrate_engine):
    # Operations to reverse the above upgrade go here.
    pass

 

There are two functions: upgrade – which brings database up to date to current version of the script (in this case version 1) and downgrade – which will rollback all the changes created by this version’s changeset (given of course you write the method code).

Since we’re using Elixir and not pure SQLAlchemy, it needs additional setup for each change script. Also having baseline model as our version zero for the migration, we must make sure that we won’t introduce conflicts with our model. If you’re probably familiar with SQLAlchemy, you know that Metadata object instance holds database metadata including database engine bindings. As you can see upgrade and downgrade methods take migrate_engine as their parameter. This is the engine specified by –url option on the migrate script, in our case manage.py which will feed the engine automagically. You have to import additional modules in our script:

from sqlalchemy import *
from migrate import *
from elixir import *
import elixir

 

This will import Elixir and it’s namespace, so we can use Entity, Field and it’s definitions directly, but will also allow us to utilize elixir namespace. This would be needed for engine binding to Elixir’s metadata, since we don’t want to use pure SQLAlchemy metadata. We want to update Entity (and database table) Page accordingly. To do this we should declare Page class which inherits from Entity like this:


import datetime

class Page(Entity):
    using_options(tablename = 'page', autoload = True)
    ctime = Field(DateTime, required = True, default = datetime.datetime.now())

 

This sets up Elixir Entity named Page, which will take our baseline model and add additional field named ctime. This however doesn’t do anything, except loading our Page model from database and declaring additional table field. Take a look at using_options. It has two parameters:

  • tablename – which holds a value of page; and
  • autoload – which is set to True.

What those options do? Option tablename is specifying table name in the database this model will resolve to. This is important, because sometimes when using ManyToMany, Elixir creates distinct names. Second autoload set to True, states that Elixir should load table definition from the database in a way our previously defined fields would be available to us, namely: id, content and title. Defining ctime would cause addition, not substitution of the table fields and in our model.

Note: You could also forget about “autoload = True” in the using_options if you had specified at the top of the script, just right after imports a directive:

options_defaults["autosetup"] = True

 

But since, we want to have control over our model, I didn’t use it. If you think it’s safe (not many relations), go ahead and use it. Just know what you’re doing. This Entity basically creates a model foundation for upgrade and downgrade functions. So let’s get into them. First we’ll take an upgrade:

def upgrade(migrate_engine):
    # Upgrade operations go here. Don't create your own engine; bind migrate_engine
    # to your metadata
    elixir.metadata.bind = migrate_engine

    Page.table.c.ctime.create()

 

Wait a second, you would say. Everywhere you’ve looked tutorials bound metadata to global metadata variable which was instantiated as metadata = MetaData(), even those describing Elixir usage with migrate. We’ll. Unfortunately they are all wrong.You have to bind your database engine to Elixir metadata and not the pure SQLAlchemy metadata. This way we don’t need any global MetaData instance and we will be sure that there are no problems with engine_dialect not found errors. That’s basically the magic trick. You could also bind metadata directly to the Elixir Entity table, but that’s too much fuss. If you need that kind of granularity, you probably wouldn’t be reading this tutorial after all. 😉

To actually create a new column for the table, we have to actually point to the table’s column definition and call .create() method on it, so that migrate would create this column with SQL’s ALTER statement. Just remember that SQLite has some limitations. PostgreSQL and MySQL should be perfectly fine. Ok. This is pretty straightforward. Without delay, we’ll do something analogous with downgrade function:

def downgrade(migrate_engine):
    # Operations to reverse the above upgrade go here.
    elixir.metadata.bind = migrate_engine

    Page.table.c.ctime.drop()

 

This causes the column definition to be dropped from database. You can now test our repository change script typing at your command prompt:

python manage.py test

And you should see:

Upgrading...
done
Downgrading...
done
Success

This will confirm that migration upgrade to version 1 and downgrade to version 0 is working fine. I could end just here, because that’s almost all you need to know, but unfortunately migrate is a little bitchy with Elixir, when using imported model Entities for using in relations. And with Elixir relations also. I have eaten my teeth on it, but I’ve finally managed to get it up and running perfectly.

Let’s suppose that we want to create another Entity named User, which for clarity would be defined as:

class User(Entity):
    id = Field(Integer, primary_key = True)
    login = Field(Unicode(30), required = True, unique = True)

 

and it would reside somewhere else, like project_directory.model.entities subdirectory. We want to create a ManyToOne Elixir relation in our Page model with the field name of created_by. Migrate needs a little trickery for this to work. So, let’s suppose that instead of ctime field, we want this created_by relation. We must edit our Page entity accordingly:


from project.model.entities import User

class Page(Entity):
    using_options(tablename = 'page', autoload = True)
    created_by = ManyToOne('User')

 

This will initialize our model instance of Page with relation to User entity. However, migrate does not know anything about ManyToOne relation, so we would have to fallback to our old, good SQLAlchemy within upgrade and downgrade functions. Since Elixir uses Integer fields for ManyToOne relations with the name constructed as: <RELATION_FIELD_NAME>_<PRIMARY_KEY_FIELD>, given RELATION_FIELD_NAME = created_by, and PRIMARY_KEY_FIELD = id, we would end up with created_by_id field of Integer type. We have to create a relation manually within the upgrade function like this:

def upgrade(migrate_engine):
    # Upgrade operations go here. Don't create your own engine; bind migrate_engine
    # to your metadata
    elixir.metadata.bind = migrate_engine

    clmn = Column('created_by_id', Integer)
    clmn.create(Page.table)

    cbid_fk = ForeignKeyConstraint([Page.table.c.created_by_id], [User.table.c.id])
    cbid_fk.create()

    Index('ix_page_created_by_id', Page.table.c.created_by_id).create()

 

This will add created_by_id Integer field that will be used for relation. Note that we have to define this relation manually. cbid_fk variable would create ForeignKey Constraint on this Integer field which relates to User.id field and adds it to database table definition. Index will create index on this field to speed up lookup. This is how a full ManyToOne relation from Elixir looks under SQLAlchemy. Determining names for ForeignKey and Index should be pretty straightforward for you.

By analogy we define downgrade function. This time however we only need to drop this Integer field. Migrate will take care of removing ForeignKey and Index for us. That’s how this function looks like:

def downgrade(migrate_engine):
    # Operations to reverse the above upgrade go here.
    elixir.metadata.bind = migrate_engine

    Page.table.c.created_by_id.drop()

 

That’s how you can use Elixir with SQLAlchemy-Migrate using also relations. Perhaps this should be designed better on behalf of Migrate, but this package is pretty awesome and I commend it’s developers with a really useful and easy to use package, that’s really well documented.

Feel free to post comments if you have suggestions for improvement or perhaps better way to force Migrate to work in accord with Elixir. That’s all folks for now. Hopefully you found this useful. Cheers.

Posted in Elixir, Programming, Pylons, Python, SQLAlchemy | 13 Comments

How to retrieve current module path in Python?

If you ever wondered how current path for module can be retrieved reliably in Python, there is a relatively simple way to do this. Each module has a __file__ variable that can be used for it.

The answer is pretty simple.


import os
path = os.path.abspath(__file__)
dir_path = os.path.dirname(path)

And that’s all. If you put this in your module, dir_path variable represents current directory of the module. That’s pretty cool thing, because it doesn’t matter what is the directory the program was called from, dir_path will always represent proper path for each module. And one of the benefits for using this simple construct is an ability to modify sys.path.

What for you might ask. Well, let’s assume that you have your own project that spans multiple subdirectories. However, you have also supporting scripts or programs which are used only from command-line and de facto they are not part of your primary application. They can be used for example for database migration, sanitizing, fixing, cleanup, backups and all other administrative purposes you can imagine, which are not part of your application. If your main project has it’s own virtual environment like e.g. using Django or Pylons, it’s hard to use module import namespace from within the subdirectory, such as the subdirectory was your import root. You always have to add primary project namespace and that somehow defies the purpose of supporting scripts, which are used for the primary application, but are not part of it in the strict sense.

If you have your project virtual environment set up and primary application resides under for example /myawesomeproject/ and your supporting scripts reside under /myawesomeproject/scripts, and some model modules for supporting scripts reside under /myawesomeproject/scripts/lib then if you’re importing modules from this lib subdirectory you’d have to do:


from myawesomeproject.scripts.lib import somemodule

But when you want this script to be used as a standalone, without reliance on the virtual environment of the primary application, you would certainly want to use:


from lib import somemodule

And here’s where the dir_path variable comes to exactly serve this purpose. When executing your supporting script, you can do:


PYTHONPATH=$PYTHONPATH:. python myscript.py

But then, it can be a problem for your users, because they have to remember they should force Python system path change. However with dir_path you can put this code into the beginning of your main script’s module to automatically set correct system path, so the imports would work without any problems:


import sys, os
path = os.path.abspath(__file__)
dir_path = os.path.dirname(path)
sys.path.append(dir_path)

And that’s all. After putting this code at the very beginning of your main module for the script, you can proceed with other imports (they will treat main module path as import root) and your script’s program code.

PS. Just remember to put empty __init__.py files to each subdirectory you would use as a collection for your imports for this to work correctly.

Posted in Coding, Programming, Python | Tagged | Leave a comment

Slow DNS requests in Linux

So, I have finally nailed it. It seems that my last post regarding this topic was only a partial solution and in some cases didn’t solve the problem of IPv6 DNS requests at all. It seems that for example OpenSSH is compiled to try AAAA record before A as default in many distributions. The problem is that GLibC resolver is sending A and AAAA DNS request at once. And even if it gets reply to A it still waits until timeout for AAAA if you don’t have any IPv6 routers present on the network you’re connected to.

The solution is simple. Newer versions of GLibC (2.10 and later) resolver has option to try one request at a time. Edit your options in /etc/resolv.conf (or if you’re using resolvconf probably with DHCP client edit /etc/resolvconf/resolv.conf.d/tail) and add a line like this:

options single-request

And that’s all. You’re done. (In case of using resolvconf you should reload it with /etc/init.d/resolvconf reload).

Posted in Internet, Network | Leave a comment

Streaming YouTube music videos as MP3 with Python

I’ve been talking with my friend about the idea of standalone YouTube music player in Linux, since we are using YouTube to listen to music very often. There is great wealth of good quality tunes published there and easy access to them, just few clicks away makes YouTube a nice candidate for streaming music. My friend suggested that there should be a plugin for Amarok or any other player in Linux which allows you to enqueue some tracks from YouTube and let you play the music, without the burden of running browser.

I said that’s really not a problem since we are Python programmers and quickly thought I might try to quickly hack a proof of concept. After just under an hour, I’ve come up with a quick, dirty hack that allows you to stream music videos from YouTube as MP3.

So what I’ve done is write a really simple and basic script that would conform to the Unix philosophy – one program, one function – under Python of course. The script takes an argument with YouTube video ID and then streams it back to standard out (stdout). Here’s the script:

— cut —

# -*- coding: utf-8 -*-
import urllib
import sys

class AppURLopener(urllib.FancyURLopener):
  version = "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101230 Mandriva Linux/1.9.2.13-0.2mdv2010.2 (2010.2) Firefox/3.6.13"

if len(sys.argv) < 2:
  print 'Usage: {prog} URL'.format(prog = sys.argv[0])
  sys.exit(1)

video_id = sys.argv[1]

opener = AppURLopener()
fp = opener.open('http://www.youtube.com/get_video_info?video_id={vid}'.format(vid = video_id))
data = fp.read()
fp.close()

if data.startswith('status=fail'):
  print 'Error: Video not found!'
  sys.exit(2)

vid_list = []
tmp_list = urllib.unquote(urllib.unquote(data)).split('|')
for fmt_chk in tmp_list:
  if len(fmt_chk) == 0:
    continue
  if not fmt_chk.startswith('http://'):
    continue
  vid_list.append(fmt_chk)

# FIXME: Format choice
link = vid_list[0]

fp = opener.open(link)
data = fp.read(1024)
while data:
  sys.stdout.write(data)
  data = fp.read(1024)
fp.close()

— cut —

This script streams the raw video to stdout. It doesn’t have any error corrections and is really simple. If YouTube decides to change the description file it won’t work anymore. Nevertheless you can stream your video to stdout and under Linux you can take care of that stream with available tools.

So I have used FFMPEG transcoder that would rip out on the fly the audio stream, transcode it to MP3 that you can write or listen to if you want.

How can you do this? Pretty simple, given you’ve got necessary decoders installed (mainly FAAC, FAAD and others used in conjunction with libavcodec). If you want to listen to mp3 stream here’s how you do it:

python script.py YOUTUBE_VIDEO_ID | ffmpeg -i - -ab 192k -vn -acodec libmp3lame -f mp3 - | mpg123 -

Where YOUTUBE_VIDEO_ID is the identificator of YouTube video. What FFMPEG options mean:

  • “-i -” – that means the input is stdin (standard input)
  • “-ab 192k” – means you want to transcode to 192kbit/s mp3 stream
  • “-vn” – you don’t want to stream any video
  • “-acodec libmp3lame” – use LAME MP3 encoder for the trancoding process
  • “-f mp3” – output format should be MP3
  • “-” – at the end, means that output is to stdout (standard out)

Due to pipelining you use three programs which make all of this possible. First the streaming script I have written, then the transcoder (ffmpeg) to mp3 stream, then the player (in this case mpg123). If you want to download YouTube video music and save it as MP3 file you do:

python script.py YOUTUBE_VIDEO_ID > stream.flv; ffmpeg -i stream.flv -ab 192k -vn -acodec libmp3lame -f mp3 OUTFILE.MP3

Where you have to substitute YOUTUBE_VIDEO_ID with YouTube video identificator and OUTFILE.MP3 with name of your new MP3 file.

That’s all! Hope you find it useful.

Posted in Coding, Linux, Programming, Python | Tagged , , , , | 5 Comments