Python Metaclasses – A blueprint for types

Everything in Python is an object. In order to create an object, a blueprint of the object must first be declared. Like most object-oriented languages, blueprints are declared in the form of a class. In order to create an object, one must first call the class’s constructor. For example:

class Person:
    def __init__(self, first_name, last_name):
        self.first_name = first_name
        self.last_name = last_name

person = Person('Aaron', 'Boman')

Classes are also known as types as they depict the type of an object. In the above example, when instantiating the Person class, it returns a new object of type Person. You can see what type an object is in Python using the type() function.

>>> type(person)
<class Person>

Python also has the ability to programatically generate a class. To create the same Personclass programatically, use the second signature to the type() function:

# create the init method
def person_initializer(self, first_name, last_name):
    self.first_name = first_name
    self.last_name = last_name

# create the class
Person = type('Person', (), {'__init__': person_initializer})

person = Person('Aaron', 'Boman')
print type(person)
#  outputs: <class 'Person'>

The above will create a class which will behave just like declaring a class with the class keyword. The second signature for type() is type(name, bases, attributes) where

  • name: is the name of the type (or class) to create
  • bases: is a tuple of classes to inherit from
  • attributes: is a dictionary of class-level attributes. Note how __init__ is being declared as a class-level attribute.

So, the type() function serves two purposes:

  • to determine what type an object is
  • to create a new type

Metaclasses

Since everything in Python is an object, that means that classes are objects too. That definitely blew my mind the first time I had heard it. If classes are objects, that must mean that type is a class? Yes, that’s absolutely correct. And, since type is a class, you can inherit from that class. What is a class that inherits from type? Why, it’s a blueprint for types, of course!

A class that creates types (or classes) is called a metaclass. To create one, simply declare a new class that inherits from type:

class MyMetaClass(type):
    pass

The new metaclass can then be used just like type():

>>> MyClass = MyMetaClass('MyClass', (), {})
>>> my_instance = MyClass()
>>> type(my_instance)
<class MyClass>

It is also possible to instruct a class declared with the class keyword to use a particular metaclass in order to construct that class. 

class MyClass(metaclass=MyMetaClass):
    pass

Now, when the MyClass class is constructed, it will use MyMetaClass to construct it. This is absolutely pointless unless the metaclass extends the type class in some way. This is where the fun begins.

Extending the type class

If you declare a __new__ method on a metaclass, it controls how the class is constructed. The signature for __new__ is 
__new__(mcs, name, bases, attrs)where mcs is the metaclass instance and the remaining three parameters are exactly the same as the type() function. One use for overriding the __new__() method is to dynamically generate methods for the class.

def functon_factory(method_name, something_to_return):
    def created(self):
        return something_to_return
    created.__name__ = method_name
    return created

class MyMetaClass(type):
    def __new__(mcs, name, bases, attrs):
        for name, message in (('method1', 'Hello'), ('method2', 'How are you?')):
            attrs[name] = functon_factory(name, message)
        # python 3 use of super() here
        return super().__new__(mcs, name, bases, attrs)

Then, the metaclass can be used to construct a new class. The new class will automatically have two methods: method1 and method2.

class MyClass(metaclass=MyMetaClass):
    pass
# ...
>>> mc = MyClass()
>>> mc.method1()
'Hello'
>>> mc.method2()
'How are you?'

Note: you must return a newly created object from the __new__() method! You will always want to call super() in some way, which will return a new class instance. Then, you will want to return that instance before the method ends.

A real world use of __new__ in metaclasses

If you’re familiar with Django at all, Django allows you to specify a model of a table in a database by inheriting a class named Model. The Model class expects that you specify the fields for the table by setting class-level attributes. The name of the attribute becomes the name of the field in the table. The value is expected an instance of one of the Field types. The field type describes how the database should store the field value. For example,

class Employee(Model):
    first_name = CharField(max_length=255)
    last_name = CharField(max_length=255)

This will create an employee table in a database with two varchar fields that hold an employee’s first and last name. The Employeeclass itself can be instantiated, however, it doesn’t behave like a normal class

>>> e1 = Employee(first_name='Aaron', last_name='Boman')
>>> e1.first_name
'Aaron'

Without specifying an __init__ method, I am able to instantiate an Employee instance with the first_name and last_name keyword arguments. How is this done? Metaclasses of course! By overriding the __new__ method on the ModelMeta class, Django is able to first see that there are two fields on the employee table in the database, and second, see how to create the constructor for the Employee class. Neat, huh?

Wanna see the implementation? Have a look at Django’s source.

Other uses of a metaclass

There have been a few times where I want to create a singleton instance of an object. A singleton instance is like memoization for the constructor: whenever the constructor is called with the same arguments, I want the same instance to be returned. This is a piece of cake with metaclasses.

class PersonMeta(type):
    def __init__(cls, first_name, last_name):
        super().__init__(*args, **kwargs)
        cls._instance_map = {}

    def __call__(cls, first_name, last_name):
        key = (first_name, last_name)
        if key not in cls._instance_map:
            new_instance = super().__call__(first_name, last_name)
            cls._instance_map[key] = new_instance
        return cls._instance_map[key]

class Person(metaclass=PersonMeta):
    def __init__(self, first_name, last_name):
        self.first_name = first_name
        self.last_name = last_name

This then allows me to do something like

>>> person1 = Person('Aaron', 'Boman')
>>> person2 = Person('Aaron', 'Boman')
>>> person1 is person2
True

Definining the __init__ method instructs how to initialize the class. In this case, I’m creating a class-level variable, _instance_map; a dictionary that maps the constructor’s parameters to an instance that was constructed with those parameters. Then, by defining the __call__ method, I’m actually intercepting the call to the class’s constructor. In it, I check to see if an object has been instantiated with the values passed to the constructor. If it has, it returns the already existing instance. Otherwise, it continues on to the original constructor, creates the new instance, then updates the cache to point to the new instance. Finally, the new instance is returned.

Note: I could have used *args and **kwargsfor the constructor’s argument and made this a broad-use, but in order to use the dictionary from kwargs, it will first have to be converted to a sorted tuple of key value pairs and this may not be desirable as it could potentially be a performance hit. Thus, I elected to hard-code the constructor parameters in the above example.

Another use for a metaclass is to create a classproperty. If you haven’t ever heard of the property() function, it is a function that allows you to declare a method on a class as a descriptor. Basically, this is just a fancy way of saying that you can treat a method like an attribute. For example:

class Person:
    def __init__(self, first_name, last_name):
        self.first_name = first_name
        self.last_name = last_name

    @property
    def full_name(self):
        return '{} {}'.format(self.first_name, self.last_name)

>>> person = Person('Aaron', 'Boman')
>>> # note how this isn't a function call!
>>> person.full_name
'Aaron Boman'

Sometimes it is desirable to create a class-level property for similar reasons. Python doesn’t support this by default, but it can be done with metaclasses:

class ToolsetConfigMeta(type):
    def __init__(cls, *args, **kwargs):
        # declare the default values here in case a base
        # class doesn't declare them as full_toolset below
        # will throw an error if the attributes are not declared.
        cls.toolset = ''
        cls.version = ''
        super().__init__(*args, **kwargs)

    @property
    def full_toolset(cls):
        return '{}-{}'.format(cls.toolset, cls.version)


class ToolsetConfig(metaclass=ToolsetConfigMeta):
    toolset = 'msvc'
    version = '12.0'


>>> ToolsetConfig.full_toolset
'msvc-12.0'

Simply declare the attributes in the __init__(don’t forget to call super()!), then declare the method decorated with the property function, and that’s it! Computed class properties just like object properties.

Warnings

Metaclasses are really neat. They provide a way to create custom types purely in Python. However, with all of this great power comes great responsibility. Much to my dismay, there aren’t too many real-world uses for them. As always, code should be readable and understandable. Metaclasses throw a level of indirection in understanding code that most Python programmers are not familiar with and therefore reduces the readability of the code. A general rule of thumb for using metaclasses should be: Does it make the end-user API substantially easier to use? In the case of the Django ORM API, I would say that’s a definite yes.

Leave a Reply