:: cleverdevil ::

Buy My Expertise and Save Lives!

I’m proud to report that my employer, ShootQ is sponsoring the ThirstRelief Mentor Auction to benefit ThirstRelief International. Bidding starts tonight, Thursday, January 29 at 10 p.m. EST. If you have any interest in photography, I’d encourage you to participate in the auction, as there are some great opportunities available to interact with some of the best in the world.

One of the auctions available to bid on is a 90-minute mentor session with the ShootQ development team, including yours truly. While we may not be as glitzy as some of the other folks up for auction, I’d like to think that we could still fetch a nice sum. So join in the cause, and start bidding tonight!

Its a joy to be a part of a business that puts such an emphasis on making a difference in the world, and I am excited to be able to chip in a bit myself.

Metaclasses Demystified [5]

This article originally appeared in the July 2008 issue of Python Magazine and was adapted from Jonathan LaCour’s CLEVERtrain professional Python training services.

The word “metaclass” often strikes fear into the hearts of Python programmers. This article explores metaclasses in an approachable way, in order to take the mystery out of metaclasses, and enable you to leverage metaprogramming techniques to solve real problems.

There are few things in the Python world that are so misunderstood and feared as metaclasses. In fact, when I recently was training a group of seasoned programmers about metaclasses, I asked the room what sprung to mind when they heard the word “metaclass.” The students shouted out their feelings in chorus â “confusing,” “magical,” and worst of all, “difficult.”

Well, as I told my students that day, when it comes to metaclasses, I’d recommend that we called upon the wise words of Douglas Adams in his famed book “The Hitchhiker’s Guide to the Galaxy.”

“Don’t panic!”

Once you take a closer look at how object-oriented programming works in Python, you’ll quickly discover that not only are metaclasses easy, they’re also extremely useful. Here, we’ll do our best to take all of the mystery out of metaclasses, and show you that there’s nothing to be afraid of!

Revisiting Classes

Most Python programmers are familiar with creating classes, and might even create or modify their own classes several times a week in the course of their work. Many of us were even taught about classes and object-oriented programming in the course of our education. But, when was the last time any of us really thought about classes in depth since we were first introduced to the concept? I’d wager that most of us take classes for granted, and don’t really think about what they are providing for us, and more importantly, how they provide it to us.

Understanding metaclasses can be greatly simplified by taking another look at Python classes, from a slightly different perspective. In order to do this, lets pose a simple question. What exactly does a class do?

My first instinct when attempting to answer this question is to rely on my education, which unearths big, fancy, computer-sciency words like “encapsulation” and “abstraction.” Its tempting to allow this to cloud our thinking, so lets do our best to think at a much simpler level when considering this question. So, I ask again â what does a class do?

Fundamentally, a class is used to construct objects called instances. This is essentially all a class does â creates instances. The process of creating instances using a class is called instantiation. For example, given a class Person, we can instantiate it to create an “instance” of that Person class.

    class Person(object):
        pass

    jonathan = Person()

This instance is related back to its class in that the class is what constructs the instance. What do we mean by “construct”? Well, when a class is instantiated, it constructs an instance by providing it with its “namespace.” We all know from Tim Peters’ “The Zen of Python” that namespaces are a “honking great idea,” and classes are a great example of why! The attributes in the namespace of a class are used to define the namespace of its instances, thus providing those instances with behavior and state. Lets enhance our Person class, to see how this works.

    >>> class Person(object):
    ...    greeting = 'Hello'
    ...    
    ...    def greet(self, who):
    ...        print self.greeting, who

    >>> jonathan = Person()
    >>> jonathan.greet('Readers')
    'Hello Readers'

The Person class now has two attributes in its namespace â greeting and greet. When we instantiate our Person class, our instance is given both state, in the form of the greeting attribute, and behavior, in the form of the greet method. In this way, a class is of critical and primary importance in defining how its instances will behave.

So, what does a class do? Lets summarize. In a nutshell, a class is an object with constructs instances through a process called instantiation, therefore defining the namespace, behavior, and state of that instance.

Defining Metaclasses

Now that we’ve established clearly what a class is, we can confidently pose the question â what exactly is a metaclass? When sifting through documentation, the most common answer you’ll find to that question is short, but often difficult to unpack:

A metaclass is the class of a class.

While this answer is certainly correct and concise, and tells us what a metaclass is, it still doesn’t tell us what a metaclass does. Lets look at an alternate answer to the question:

A metaclass is a class whose instances are classes.

I will concede that this answer is just as much of a mouthful as its counterpart, but it has the benefit of giving us a clue as to what a metaclass does! In short, a metaclass constructs classes. We also know that a metaclass is just another class, and we’ve just spent some time outlining what a class does. As a result, we should be able to build upon what we’ve learned about classes earlier to determine a bit more about metaclasses.

Recall that a class helps define the behavior and state of its instances. Metaclasses provide the same basic capability to classes, giving you the ability to change the way a class will behave at runtime. This technique is commonly referred to as “metaprogramming.”

Having a definition of metaclasses and metaprogramming is useful, but lets take a deeper look by investigating an existing metaclass.

The `type` Metaclass

In Python, classes which inherit from object are called “new-style classes.” All such new-style classes have a default metaclass called type. We can prove that type is the metaclass of object through simple introspection in the Python interpreter. Recall, a metaclass is the class of a class, so what is the class of object:

    >>> print object.__class__
    <type 'type'>

Just as we expected! The metaclass of object is type, and thus all classes which inherit from object will be provided with this metaclass by default. Classes that do not inherit from object are called “old-style classes” and will disappear in Python 3.0. While old-style classes also support metaclasses and metaprogramming, we’ll focus on new-style classes in this article for the sake of simplicity.

Typically, classes are defined using the class statement in Python, as we saw in our earlier Person example. However, we have just learned that metaclasses create classes when they are instantiated. This means that we should be able to define a class by instantiating the type metaclass manually.

Lets define our original Person class again, but this time, lets do it without using the class statement by instantiating the type metaclass:

    >>> def greet(self, who):
    ...    print self.greeting, who

    >>> Person = type(
    ...    'Person', 
    ...    (object,),
    ...    {'greet': greet, 'greeting': 'Hello'}
    ... )
    >>>
    >>> jonathan = Person()
    >>> jonathan.greet('Readers')
    'Hello Readers'

This method of creating classes is equivalent to using the class statement, and reveals quite a bit about how metaclasses work. The constructor for a metaclass expects very specific arguments:

The first argument is the name of the class.
The second argument is a tuple of the base classes for the class.
The last argument is a dictionary representing the namespace of the class. This dictionary contains all of the attributes that would typically appear within the body of a class statement.

Now, we’ve seen a metaclass in action, and we know how to instantiate them to create classes. Armed with this knowledge of the default type metaclass, we can now tackle the much more interesting problem of creating our own metaclasses.

Defining Metaclasses

Defining metaclasses in Python is as simple as creating a class that inherits from the built-in type metaclass. The constructor for our metaclass will take the same arguments as the constructor for the type metaclass:

The class name.
A tuple of the class bases.
A dictionary representing the namespace of the class.

Constructors typically perform some action on their instances, so lets make our constructor set a flag on the instance that we can inspect to make sure that our metaclass is being used.

    class MyFirstMeta(type):
        def __init__(cls, name, bases, ns):
            cls.uses_my_metaclass = True

One important thing to note here is that the first argument to the constructor of a class is typically called self, as it refers to the instance being constructed. It is conventional to name the first argument of a metaclass constructor cls as the metaclass instance is actually a class.

Now that we’ve defined our metaclass, we know that we can construct a new class called MyClass simply by instantiating the metaclass:

    >>> MyClass = MyFirstMeta('MyClass', (object,), {})
    >>> print MyClass.uses_my_metaclass
    True

This very simple metaclass has given us our first glimpse into the power of metaclasses. Within our metaclass constructor, we gave our class some state in the form of a boolean attribute uses_my_metaclass. Metaclasses have the power to add, remove, or modify any attribute of the class being constructed. Metaclasses will frequently add or replace methods on their instances, based upon the data in the namespace of the class. Many Python object-relational mappers use metaclasses to transform the attributes of a class into database table definitions, for example.

While you can certainly construct classes by manually instantiating custom metaclasses, it is much more convenient to use the class statment to create your classes. Python allows you to define the metaclass for a class by using the special __metaclass__ attribute in your class statement:

    class MyClass(object):
        __metaclass__ = MyFirstMeta

This is the preferred method of attaching metaclasses to your classes. An important thing to note about this syntax is that while you are not manually instantiating the metaclass when defining your classes this way, the Python interpeter will instantiate the metaclass. The metaclass instantiation will occur immediately after the class statement has been fully executed. As a result, bugs in metaclasses often are triggered during imports. In a way, the class statement is simply syntactic sugar for instantiating metaclasses!

Mystery Metaclass Methods

Before we move onto some practical examples of metaclasses, lets investigate metaclass definition a bit further. Our Person class defines a method called greet within its namespace. Instances of Person will thus have an “instance method” called greet. We know what happens when we define methods on a class, but what happens if we define a method on a metaclass?

Methods defined on a class become instance methods. Since instances of metaclasses are classes, methods defined on metaclasses become _class methods_. Lets take a look at this in practice:

    >>> class MysteryMeta(type):
    >>> ... def mystery_method(cls):
    >>> ...     return 'I am a class method!'
    >>> 
    >>> class Mystery(object):
            __metaclass__ = MysteryMeta
    >>> 
    >>> print Mystery.mystery_method()
    I am a class method!

This revelation often surprises people, but is a logical outcome of the fact that metaclasses are simply classes which produce classes. Many metaclasses utilize this capability of defining class methods, but it is often preferable to define such class methods on a base class, which is easier to document and understand.

We’ve now established what metaclasses are, how they work, and how we can define our own by inheriting from type. But, what about practical use cases for metaclasses? Lets take a look at several examples of how we might come to use metaprogramming in practice.

Example: The Enforcer

When I am teaching Python to programmers with strong backgrounds in “bondage and discipline” languages like Java, I frequently hear the same complaints about Python. One such complaint is that class definitions are “loose” and you cannot enforce the type of variables by declaring them up-front. In order to illustrate how to use metaclasses, lets define a little library that will address this particular complaint.

Our goal is to create a base class called Enforcer that will enforce the types of attributes on its subclasses. Lets say we wanted to enforce that the name attribte of our Person class was a string, and that the age attribute was an integer. Attempting to set attribtues with the wrong type should trigger a TypeError to be raised, just like the Java compiler would catch such errors.

    class Person(Enforcer):
        name = Field(str)
        age = Field(int)

The first thing we need to do is define the Field class, which we will use to hold onto the type of the variable we’re attempting to restrict. Lets also give it the ability to validate whether or not a value is of the right type for that particular attribute:

    class Field(object):
        def __init__(self, ftype):
            self.ftype = ftype

        def is_valid(self, value):
            return isinstance(value, self.ftype)

Now that we have our Field class, we need to create a metaclass that will look at the namespace of our class, searching for Field definitions, and then storing them in a dictionary for later use. Recall that the last argument to the constructor of a metaclass is a dictionary containing the namespace of the class. We can loop through this namespace to find Field definitions:

    class EnforcerMeta(type):
        def __init__(cls, name, bases, ns):
            # store the field definitions on the class as a dictionary 
            # mapping the field name to the Field instance.
            cls._fields = {}

            # loop through the namespace looking for Field instances
            for key, value in ns.items():
                if isinstance(value, Field):
                    cls._fields[key] = value

Our metaclass first attaches a _fields dictionary to the class itself. This data structure is where we will store Field definitions for later use. We then loop through the items in the namespace looking for Field instances, and finally we store them in our _fields dictionary.

Next up is the Enforcer base class itself. The Enforcer base class first needs to attach the EnforcerMeta metaclass we’ve just defined. This is a very common way for libraries to distribute their metaclasses, by defining a base class to inherit from, rather than requiring users to even know that a metaclass is being used, or how to attach the metaclass to their classes.

The second thing the Enforcer base class needs to do is to override the __setattr__ method. This is a special method on Python classes that allows you to override the default attribute setting behavior on your Python objects. The __setattr__ method takes in the name of the attribute being set, and the value being set.

    class Enforcer(object):
        # attach the metaclass
        __metaclass__ = EnforcerMeta

        def __setattr__(self, key, value):
            if key in self._fields:
                if not self._fields[key].is_valid(value):
                    raise TypeError('Invalid type for field!')
            super(Enforcer, self).__setattr__(key, value)

Our Enforcer class first attaches the metaclass. Then, it overrides the __setattr__ method so that it can watch for field assignments. First, we check to see if the attribute being set is one of our defined fields. Then, we ask the field definition if the value that is being passed is valid for the field definition. If it is not a valid type for the field, we raise a TypeError.

The last line of the Enforcer class is extremely important. This line is instructing the Python interpreter to call the __setattr__ implementation on the appropriate superclass definition, in this case object. Without this line, attribute setting would fail on all Enforcer subclasses.

Lets try out our new creation from the Python interactive prompt:

    >>> class Person(Enforcer):
    ...    name = Field(str)
    ...    age = Field(int)
    ... 
    >>> jonathan = Person()
    >>> jonathan.name = 3
    TypeError: Invalid type
    >>> jonathan.age = 'Old'
    TypeError: Invalid type
    >>> jonathan.name = 'Jonathan'
    >>> jonathan.age = 28

Our metaclass has completely changed the way that our class behaves, transforming it into a more rigid class definition that keeps our Java-loving friends happy. While such restrictive enforcement is atypical in Python, it certainly shows how powerful metaclasses can be.

This example also illustrates a common pattern for metaclasses, in which a class describes how it wants to behave in a “declarative” way without actually writing any code to implement that behavior. The metaclass subsequently takes that metadata, and uses it to “reprogram” the class. This is the essence of what metaprogramming brings to the table, and is the most popular use-case for metaclasses.

Example: Auto Decorator

Since being given a special syntax in Python 2.4, decorators have become a very commonly used feature. A decorator is essentially a wrapper around a function or method. In versions of Python prior to 2.4, decorators were applied by manually replacing the method definition with a “decorated” version of the method. Starting in Python 2.4, decorators could be applied using the now popular @decorator syntax.

Sometimes, I find myself in the situation where I need to decorate all of the methods of a class with the same decorator. Lets create a metaclass that will simplify this process by automatically applying a decorator that is declared in the namespace of the class:

    import inspect

    class AutoDecorateMeta(type):
        def __init__(cls, name, bases, ns):
            deco = ns.get('decorator', lambda f: f)
            for key, value in ns.items():
                # skip the decorator and constructor
                if key in ('decorator', '__init__'): continue

                # skip objects in the namespace that aren't methods
                if not inspect.isfunction(value): continue

                # apply the decorator
                setattr(cls, key, deco(value))

Our metaclass first looks up an attribute named decorator in the namespace of the class. This is the decorator that we will apply to every method of the class. Note that if such a decorator cannot be found in the namespace of the class, we define a fake decorator that will simply return the existing method definition.

Next, we loop through the namespace of the class looking for methods. If we encounter the decorator or the constructor, we skip them, as we don’t want to decorate our constructor or the decorator itself. We then make use of the inspect module to determine if the value is a method, and if it is, we replace the method definition with a decorated version of that method.

Lets put our metaclass to the test by creating a class that defines several properties. Typically, we’d have to decorate every method with the @property decorator, which could add a lot of verbosity to our class, depending on the number of methods we’d have to decorate.

    class Person(object):
        __metaclass__ = AutoDecorateMeta
        decorator = property

        def __init__(self, first, middle, last):
            self.first = first
            self.middle = middle
            self.last = last

        def name(self):
            return '%s %s' % (self.first, self.last)

        def full_name(self):
            return '%s %s %s' % (self.first, self.middle, self.last)

        def initials(self):
            return '%s%s%s' % (self.first[0], self.middle[0], self.last[0])

Now that we’ve created our class, we should be able to try it out from our Python interactive prompt to see if all of our methods were transformed into properties:

    
    >>> mlk = Person('Martin', 'Luther', 'King')
    >>> print mlk.name
    Martin King
    >>> print mlk.full_name
    Martin Luther King
    >>> print mlk.initials
    MLK

Not only is this example useful in practice, it also illustrates another use case for metaclasses and metaprogramming â automating repetitive tasks that occur in class definition. This metaclass could be used to wrap every method of a class in a transaction, to add debugging hooks, or even to synchronize all of the methods using a thread lock.

Cautionary Notes

Metaclasses are fantastically cool, and as we’ve seen, they can be pretty useful. The rise of object-relational mappers and web frameworks have put metaclasses into use by an increasing number of Python users, and has increased the visibility of metaclasses substantially. That being said, metaclasses are a feature of Python that must be used carefully. Because metaclasses do their work at class definition time, bugs in your metaclasses can result in errors that are triggered at import time. In addition, metaclasses are often hidden from the programmer behind a base class, which can cause confusion.

Applied judiciously, metaclasses can be a great tool for solving problems, and give Python programmers the ability to take advantage of metaprogramming techniques in their own code. I hope that this article has shed some light on what metaclasses are, and has taken some of the mystery out of metaclasses.

Metaclasses in July Python Magazine [1]

While I do not advertise it directly on this site, in addition to my “day job” I also provide Python training services for companies and individuals that want a comprehensive introduction to the Python programming language on a compressed schedule. One of the most difficult topics to condense and explain to a group of new Python programmers is metaclasses. Over the last year or so I’ve spent a lot of time refining my teaching methods to make this complex topic approachable.

A few months ago, I presented a version of my slides on metaclasses at a PyATL meeting. Following the meeting, I was approached by Doug Hellmann about the possibility of writing an article on metaclasses for Python Magazine. Well, I am pleased to say that my article, entitled Metaclasses Demystified appears in the July 2008 issue of Python Magazine.

This is my first professionally published technical article, and I am really excited that my work will be read by many Python programmers. Depending on feedback from readers, expect to see more contributions to Python Magazine from me in the future. It was a lot of fun crafting the article!

MobileMe Disappointment [1]

When I first heard about Apple’s new MobileMe service, I was excited by the prospect of being able to keep all of my devices in sync instantly, with no waiting. I was so excited, that I purchased a boxed version of .Mac so that I’d be one of the first to get my hands on the service when it debuted.

One of the main reasons I purchased MobileMe was because of Apple’s beautiful marketing site describing the features of MobileMe. The site states:

MobileMe stores all your email, contacts, and calendars in the cloud and pushes them down to your iPhone, iPod touch, Mac, and PC. When you make a change on one device, the cloud updates the others. Push happens automatically, instantly, and continuously. You don’t have to wait for it or remember to do anything — such as docking your iPhone and syncing manually — to stay up to date.

As it turns out, this is not only misleading, but is essentially a bold-faced lie. While the “push” does occur from your iPhone to the cloud and vice-versa, your Mac only receives and sends updates once every 15 minutes, even if you instruct it to sync “Automatically” in the MobileMe preferences pane on your Mac. Honestly, I expected better from Apple.

In addition to this very irritating issue, I have several other problems with the service, which is billed as “Exchange for the rest of us,” that greatly diminish its usefulness.

I would love to utilize the push email service to manage my personal email. However, I have a personal domain where all of my email is delivered. In addition, I subscribe to many mailing lists, and receive several hundred emails a day. All of my email is neatly organized into folders automatically, using server-side mail filters through the excellent procmail utility.

MobileMe email has two massive shortcomings that make it useless for my situation:

You cannot set up MobileMe to handle email for a personal domain, unless you set up your existing mail server to forward email onto MobileMe. I could live without this one, if it were not for the second problem.
While rules that you set up in Apple mail “sync” to MobileMe, they are not applied server side. In order for your rules to be applied, you must have a version of Apple Mail running on some computer connected to the internet to apply those rules client-side.

MobileMe is a very compelling service if you just read the marketing, but in the end, in falls extremely short of my expectations from a company like Apple. I have sent my complaints onto MobileMe customer support, and am awaiting a response. Hopefully, they’ll at least acknowledge the misleading marketing.

iPhone 3G is Cheaper* [11]

I purchased a first-generation iPhone the day that it came out. Its been a wonderful purchase from the start, in spite of the fact that it cost me $599. Sure, thats a lot of money, but I always felt like I was getting my money’s worth out of the hardware. The first iPhone was the best iPod, best mobile browsing device, and best phone. When Apple reduced the price to $399, I thought that this was an absolute steal.

When I first heard rumors that the iPhone 3G was going to cost less than the current iPhone, I had guessed that this was pure rumor and speculation. There is no way that Apple would sell the iPhone for less than $399, as it would force them to restructure the pricing for their iPod line. By the same token, I couldn’t fathom Apple allowing ATT to subsidize the phone, because this would mean that ATT would be making this subsidy back up by somehow gouging the customer in service fees. Apple did an excellent job of standing up for the consumer with the first iPhone, by negotiating a low-price data plan and allowing for at-home activation through iTunes.

Needless to say, I was shocked at the price point that was announced during the keynote. Only $199? And no mention of contracts, subsidies, or rate increases? Was this too good to be true?

As it turns out, yes.

Currently, I pay ATT $20/month for 200 text messages and unlimited data over their EDGE network. ATT has announced that the new iPhone 3G data plan will cost $30/month, and will include no text messages. ATT charges $5/month for 200 text messages, which means that the iPhone data plan is effectively being increased by $15/month, representing a 75% increase over the existing data plan.

Over the course of the two year contract, an iPhone 3G will cost you a full $360 more than a first-generation iPhone would. This means that with the same ATT service plans, the $199 8GB iPhone 3G will actually end up costing you $559, where an 8 GB iPhone 1.0 will cost you only $399, representing a savings of $160! The iPhone 3G isn’t cheaper at all, it is in fact far more expensive.

Compounding this is the fact that if I upgraded, I’d likely give my current iPhone to my wife, and have to pay an additional $20/month for her data plan. By my math, getting a new iPhone 3G would cost me a minimum of $1039 over two years. Yikes!

So, I’ll be passing on the new “cheaper” iPhone, and keeping my iPhone 1.0. I’ll be able to run all the great new third-party applications, and will have every feature that my iPhone 3G toting friends will have, apart from 3G speed and GPS. As much as I hate the EDGE network, and would like to have GPS, those features simply aren’t worth over a thousand dollars to me. Here’s hoping that Apple and ATT come up with something more compelling and consumer-friendly, rather than misleading consumers.

:: cleverdevil ::

Buy My Expertise and Save Lives!

Metaclasses Demystified [5]

Revisiting Classes

Defining Metaclasses

The type Metaclass

Defining Metaclasses

Mystery Metaclass Methods

Example: The Enforcer

Example: Auto Decorator

Cautionary Notes

Metaclasses in July Python Magazine [1]

MobileMe Disappointment [1]

iPhone 3G is Cheaper* [11]

The `type` Metaclass