ProgrammingBits

More On Function Decorators

In my previous post, I presented a couple of examples on how to use function decorators in Python. Those examples were illustrative, yet fairly restricted. First of all, they assumed that the function being decorated only receives one positional argument. What can I do if I want to decorate a function that takes two or more positional arguments, or one or more keyword arguments? Secondly, they didn't allow the decorator to be configured in any special way. So, how do I send input arguments to the decorator itself so that I can vary its behavior?

I will now elaborate in a slightly more complex example that will give us a better insight on how to define much more general decorators, without any of the restrictions I just mentioned.

NOTE: All Python code examples presented here are based in Python 3.0. Full source code: more_function_decorators.py

Suppose we want a decorator function that is able to "swallow" one or more kinds of exceptions that might be raised during the execution of the decorated function¹. If a particular exception is actually produced, we want to be able to specify a default value to be returned instead of allowing the exception to propagate through the execution stack and possibly causing the program to terminate. A client of our decorator would basically be able to avoid the hassle of writing an explicit try statement.

To demonstrate how this could work, let's assume we have a function like this one:

def divide(dividend=0, divisor=1):
    return dividend / divisor

When calling this function, there are at least two possible exceptions that might get raised: ZeroDivisionError and TypeError. The first one is produced when the divisor parameter is zero. The second exception occurs when any of the two arguments sent are not of a numerical type, when more than two arguments are actually sent, or when you try to use a nonexistent keyword argument. Some examples:

>>> divide(1, 2)
0.5

>>> divide(1, 0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in divide
ZeroDivisionError: int division or modulo by zero

>>> divide("hello")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in divide
TypeError: unsupported operand type(s) for /: 'str' 
and 'int' 

>>> divide(whatever=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: divide() got an unexpected keyword 
argument 'whatever'

Our swallow decorator will receive two keyword arguments:

exceptions: a single exception class or a tuple containing several exception classes. These represent the exceptions that should be swallowed. Any other exception will be propagated as usual. Defaults to BaseException (the root of Python's exception hierarchy) if it's not explicitly provided.
default: the value to return if one of the specified exceptions is raised. Defaults to None when not provided.

Let's look at three usage examples.

Example 1: If a division by zero is attempted, return zero:

@swallow(exceptions=ZeroDivisionError, default=0)
def divide(dividend=0, divisor=1):
    return dividend / divisor

Example 2: If a ZeroDivisionError or a TypeError is raised, return zero:

@swallow(
    exceptions=(ZeroDivisionError, TypeError), 
    default=0)
def divide(dividend=0, divisor=1):
    return dividend / divisor

Example 3: Chain two swallow decorators, so that each specific exception has its own default value:

@swallow(exceptions=TypeError, default='Huh?')
@swallow(exceptions=ZeroDivisionError, default=0)
def divide(dividend=0, divisor=1):
    return dividend / divisor

For this last example, this is how we could now use our decorated divide function:

>>> divide(1, 2)
0.5
>>> divide(1, 0)
0
>>> divide("hello")
'Huh?'

In order to implement the swallow decorator, we must take a better look on how the @ syntax works. After the @ sign, you must actually specify an expression that when evaluated produces a callable object². When this callable object is effectively called, it receives as its only argument the function that is to be decorated, and it returns that same function or some new callable thing.

The expression after the @ sign is commonly just the name of a function (the decorator function), but it can also be: 1) a new instance of a class that contains an implementation of the __call__ method; or 2) a call to some other function. Both these options allow us to send additional information to the decorator by specifying it via input parameters.

A first implementation using classes that define the __call__ method could be as follows:

class swallow:
    
    class helper:
        
        def __init__(self, outer, fun):
            self.outer = outer
            self.fun = fun            
            
        def __call__(self, *args, **kwargs):
            try:
                return self.fun(*args, **kwargs)
            except self.outer.exceptions:
                return self.outer.default
    
    def __init__(self, 
                 default=None, 
                 exceptions=BaseException):
        self.default = default
        self.exceptions = exceptions
        
    def __call__(self, fun):                            
        return swallow.helper(self, fun)

Let's try to understand how this works. This code:

@swallow(exceptions=ZeroDivisionError, default=0)
def divide(dividend=0, divisor=1):
    return dividend / divisor

is basically equivalent to this one:

def divide(dividend=0, divisor=1):
    return dividend / divisor     
divide = swallow(exceptions=ZeroDivisionError, 
                 default=0).__call__(divide)

In the last statement of the above code, an instance of the swallow class is created and initialized according to our needs. Then, the __call__ method is invoked on that very same instance, which in turn creates and returns a new instance of the swallow.helper nested class. The final effect is that the divide variable refers to an instance of this specific class.

The swallow.helper class implements a __call__ method that accepts any number of positional and keyword arguments³. Thus, instances of this class can effectively decorate any function that takes whatever arguments it needs. The __call__ method itself contains a try statement that does all the work: it invokes the decorated function and sends back the returned value, unless any of the specified exceptions get caught, in which case it returns the specified default value. Note that all the values that are required to do the job are conveniently stored and shared using the instance variables of the swallow and swallow.helper classes.

A second implementation of the swallow decorator can be coded using only function definitions (and their corresponding lexical closures). Although it's considerably shorter, it might be a little bit more difficult to understand at first:

def swallow(default=None, exceptions=BaseException):
    def helper1(fun):
        def helper2(*args, **kwargs):
            try:
                return fun(*args, **kwargs)
            except exceptions:
                return default
        return helper2
    return helper1

As can be observed, the swallow function takes the two input parameters that allow us to configure the decorator. It returns the nested function helper1, which is just what the @ syntax expects. The helper1 function will be immediately called with the function being decorated as its only argument, returning function helper2 as its result. This means that if we are decorating a function called f, variable f will end up holding a reference to the helper2 function. So now, whenever f gets invoked, helper2 will be called and the try statement will do its job exactly as described before.

Notes

¹ Swallowing exceptions can be convenient under certain circumstances, but you should avoid using this technique indiscriminately. Specifically, it can make debugging code very hard.

² Callable objects contain a special attribute named __call__. If x is a callable object, then the syntax:

x(arg1, arg2, arg3)

is equivalent to:

x.__call__(arg1, arg2, arg3)

Callable objects include user-defined functions, built-in functions, methods of built-in objects, class objects, methods of class instances, and instances of classes that define or inherit their own __call__ method.

³ If your not familiar with the *args and **kwargs notation, check the Python tutorial for more details.

Function Decorators

The Python programming language has an interesting syntactic feature called a decorator. Let's use an example in order to explain how and why you would want to use a Python decorator.

NOTE: All Python code examples presented here are based in Python 3.0. Full source code: function_decorators.py

Suppose we want to compute the Fibonacci numbers using a recursive function¹. The following function definition does the trick:

def fib(n):
    if n in (0, 1):
        return n
    else:
        return fib(n - 1) + fib(n - 2)

We can now test the function with different input values:

>>> fib(0)
0
>>> fib(1)
1
>>> fib(4)
3
>>> fib(10)
55
>>> fib(30)
832040
>>> fib(35)
9227465

If you've actually typed these examples in a Python shell, you've probably noticed that obtaining a result takes longer for bigger inputs. In order to make it run faster, we can use a technique called memoization. A memoized function stores in a cache the results corresponding to some set of specific inputs. Later calls, with previously computed inputs, return the results stored in the cache, thus avoiding their recalculation. This means that the primary cost of a call with certain parameters is taken care of during the first call made to the function with those same parameters.

Instead of modifiying the fib function directly, it's better to write a reusable memoizing function:

def memoize(f):
    cache = {}
    def helper(x):
        if x not in cache:            
            cache[x] = f(x)
        return cache[x]
    return helper

The memoize function wraps another function, called helper, which is going to provide additional functionality to the function received through parameter f. The helper function is actually a lexical closure, that is, a function object that remembers the variable bindings that were in its scope when it was created. In this case, helper remembers variables f and cache, which happen to hold a function object and a dictionary, respectively. The last statement in memoize just returns to its caller the helper lexical closure.

Type the following code, and you'll notice right away a speed increase when calling fib. The speedup can be appreciated even in the very first call because the memoization immediately boosts all the recursive calls inside fib's definition.

>>> fib = memoize(fib)
>>> fib(50)
12586269025

The assignment in the first line can be read as: the fib function is being decorated by the memoize function. Because this is such a common programming idiom in Python, there's a special decorator syntax to simplify its use. This syntax was originally inspired by Java's annotations: just above the definition of the function that is to be decorated, place an at sign (@) followed by the name of the decorator function.

In other words, this Python syntax:

@some_decorator
def some_function():
    # function body...

is equivalent to:

def some_function():
    # function body...
some_function = some_decorator(some_function)

Most should agree that the @ notation is more readable and less error prone.

In order to use the Python decorator with our Fibonacci + memoization example, we would have to rewrite as follows:

def memoize(f):
    # Same code as before...
 
@memoize 
def fib(n):
    # Same code as before...

The memoize function shows the general structure that a typical function decorator should have: it receives a function f as its input parameter, and it returns another function that attaches some additional responsibilities to f.

Another interesting thing about Python decorators is that you can chain two or more together. Let's extend our example in order to incorporate a tracing decorator that will display a message before the decorated function gets called and also when it returns.

def memoize(f):
    # Same code as before...
 
def trace(f):
    def helper(x):
        call_str = "{0}({1})".format(f.__name__, x)
        print("Calling {0} ...".format(call_str))
        result = f(x)
        print("... returning from {0} = {1}".format(
              call_str, result))
        return result
    return helper
 
@memoize
@trace
def fib(n):
    # Same code as before...

Notice the use @memoize and @trace just before the definition of fib. Now, when invoking fib, first the memoize decorator gets called, then the trace decorator, and finally the original fib code. Check this example:

>>> fib(5)
Calling fib(5) ...
Calling fib(4) ...
Calling fib(3) ...
Calling fib(2) ...
Calling fib(1) ...
... returning from fib(1) = 1
Calling fib(0) ...
... returning from fib(0) = 0
... returning from fib(2) = 1
... returning from fib(3) = 2
... returning from fib(4) = 3
... returning from fib(5) = 5
5

Python has three built-in functions that are intended to be used as function decorators:

classmethod: used to indicate that the decorated method is a class method, similar to those in Smalltalk or Ruby.
staticmethod: used to indicate that the decorated method is a static method, like those in C++, C# or Java.
property: used to decorate methods that will be used to get, set and delete object properties.

It's also worth noting that Python 3.0 not only allows you to decorate functions, but also complete classes. Hopefully, I'll take some time to write about these specific kinds of decorators in a a future post.

Notes

¹ Yes, I know that using recursion to implement the Fibonacci sequence is very inefficient. But that was my intention. I wanted a simple algorithm that produces a perceivable time delay for certain inputs.

Further Reading

PEP: 318 Decorators for Functions and Methods

Exploring Python's Awesomeness

I recently saw The Kung Fu Panda movie, and it has this really cool and funny saying that made me think about the way I perceive the Python programming language:

"There is no charge for awesomeness... or attractiveness."

Python is a really awsome language, and many people (including myself) find it extremely attractive. Additionally, it's an open language with great free implementations. What else can you ask for?

This is my first ProgrammingBits post. It's my intention to use this blog to write some insights about deep technical (and maybe even philosophical) issues regarding the Python programming language and its use in education.

I just basically want to give back a little bit of what I've received from the generous Python community. Hopefully, some people will find my writings somehow useful. Or maybe just amusing. That's fine with me.