Untitled-2-Recovered

Advanced Design Patterns in Python



The aim of this tutorial is to show off Advanced design structures in Python and the best way to use them. Depending on what you need from a data structure, whether it’s fast lookup, immutability, indexing, etc, you can choose the best data structure for the job and most of the time, you will be combining data structures together to get a logical and easy to understand data model. Python data structures are very intuitive from a syntax point of view and they offer a large choice of operations. This tutorial tries to put together the most common and useful information about each data structure and offer a guide on when it is best to use one structure or another.

Comprehensions

If you’ve used Python for very long, you’ve at least heard of list comprehensions. They’re a way to fit a for loop, an if statement, and an assignment all in one line. In other words, you can map and filter a list in one expression.

A list comprehension consists of the following parts:

  • An Input Sequence.
  • A Variable representing members of the input sequence.
  • An Optional Predicate expression.
  • An Output Expression producing elements of the output list from members of the Input Sequence that satisfy the predicate.

Say we need to get a list of all the integers, whose value is above zero in a sequence and then square them:

Pretty simple, right? But it took 4 lines, two degrees of nesting, and an append to do something completely trivial. You could reduce the size of the code with the filter, lambda and map function:

The code is starting to expand in the horizontal direction now! Alas, what could we possibly do to simplify the code? List Comprehension to rescue.

list

  • The iterator part iterates through each member x of the input sequence num.
  • The predicate checks if the member is greater than zero.
  • If the member is greater than zero then it is passed to the output expression, squared, to become a member of the output list.

The list comprehension is enclosed within a list so, it is immediately evident that a list is being produced. There is only one function call to type and no call to the cryptic lambda instead the list comprehension uses a conventional iterator, an expression and an if expression for the optional parameter.

There is a downside to list comprehensions: the entire list has to be stored in memory at once. This isn’t a problem for small lists like the ones in the above examples, or even of lists several orders of magnitude larger. But eventually this becomes pretty inefficient.

Generators to rescue this time. Generator expressions do not load the whole list into memory at once, but instead create a ‘generator object’ so only one list element has to be loaded at any time.

Generator expressions have the same syntax as list comprehensions, but with parentheses around the outside instead of brackets:

This is ever so slightly more efficient than using a list comprehension.

Lets change it to a more efficient code:

It’s probably good practice to use generator expressions unless there’s some reason not to, but you’re not going to see any real difference in efficiency unless the list is very large.

You can use zip() and dealing with two or more elements at a time:

A two-level list comprehension using os.walk():

Decorators

Decorator provide a very useful method to add functionality to existing functions and classes. Sounds a bit like Aspect-Oriented Programming (AOP) in Java, doesn’t it? Except that it’s both much simpler and (as a result) much more powerful. For example, suppose you’d like to do something at the entry and exit points of a function (such as perform some kind of security, tracing, locking, etc. – all the standard arguments for AOP).

A decorator is a function that wraps another function: the main function is called and its return value is passed to the decorator. The decorator then returns a function that replaces the wrapped function as far as the rest of the program is concerned.

The @ indicates the application of the decorator.

Now let’s go back and implement the first example in decorator section. Here, we’ll do the more typical thing and actually use the code in the decorated functions:

Whenever you write code like this:

it’s the same as if you had performed these separate steps:

The code inside a decorator typically involves creating a new function that accepts any
arguments using *args and **kwargs, as shown with the wrapper() function in this
recipe. Inside this function, you place a call to the original input function and return its
result. However, you also place whatever extra code you want to add (e.g., timing). The
newly created function wrapper is returned as a result and takes the place of the original
function.

Lets look at another example

When the compiler passes over this code, function() is compiled and the resulting function object is passed to the decorator code, which does something to produce a function-like object that is then substituted for the original function().

What does the decorator code look like? Well, most examples show this as a function, but I’ve found that it’s easier to start understanding decorators by using classes as decoration mechanisms instead of functions. In addition, it’s more powerful.

The only constraint upon the object returned by the decorator is that it can be used as a function – which basically means it must be callable. Thus, any classes we use as decorators must implement __call__.

What should the decorator do? Well, it can do anything but usually you expect the original function code to be used at some point. This is not required, however:

Practical Example:

ContextLib

The contextlib module contains utilities for working with context managers and the with statement. Normally, to write a context manager, you define a class with an __enter__() and __exit__() method, like this:

Complete Example:

A context manager is enabled by the with statement, and the API involves two methods. The  __enter__() method is run when execution flow enters the code block inside the with. It returns an object to be used within the context. When execution flow leaves the with block, the __exit__() method of the context manager is called to clean up any resources being used.

Again writing the sample example using the @contextmanager decorator in the contextlib module.

In the function, all of the code prior to the yield executes as the __enter__() method of a context manager. All of the code after the yield executes as the __exit__() method. If there was an exception, it is raised at the yield statement.

Descriptors

Descriptors determine how attribute of object are accessed. A descriptor is a way to customize what happens when you reference an attribute on a model.

The real trick to building a descriptor is defining at least one of the following three methods. Note that instance below returns to the object where the attribute was accessed, and owner is the class where the descriptor was assigned as an attribute.

  • __get__(self, instance, owner) — This will be called when the attribute is retrieved (value = obj.attr), and whatever it returns is what will be given to the code that requested the attribute’s value.
  • __set__(self, instance, value) — This gets called when a value is set to the attribute (obj.attr = ‘value’), and shouldn’t return anything at all.
  • __delete__(self, instance) — This is called when the attribute is deleted from an object (del obj.attr).

LazyLoading Properties

Descriptors are a generalization of the concept of bound methods, which was central to the implementation of classic classes. In classic classes, when an instance attribute is not found in the instance dictionary, the search continues with the class dictionary, then the dictionaries of its base classes, and so on recursively. When the attribute is found in a class dictionary (as opposed to in the instance dictionary), the interpreter checks if the object found is a Python function object. If so, the returned value is not the object found, but a wrapper object that acts as a currying function. When the wrapper is called, it calls the original function object after inserting the instance in front of the argument list.

As mentioned above, descriptors are assigned to classes, and the special methods are called automatically when the attribute is accessed, and the method used depends on what type of access is being performed.

MetaClasses

Metaclasses offer a powerful way to change how classes in Python behave.

A metaclass is defined as “the class of a class”. Any class whose instances are themselves classes, is a metaclass.

We’ve created a class and an object of that class. Examining the __class__ of obj we saw that it’s demo. Next comes the interesting part. What is the class of demo? We can again examine it with __class__ and we see it’s type.

So type is the class of Python classes. In other words, while in the example above obj is a demo object, demo itself is a type object.

So, according to what we’ve seen above, this makes type a metaclass – in fact, the most commonly used metaclass in Python, since it’s the default metaclass of all classes.

Since a metaclass is the class of a class, it is used to construct classes (just as a class is used to construct objects). But wait a second, don’t we create classes with a standard class definition? Definitely, but what Python does under the hood is the following:

  • When it sees a class definition, Python executes it to collect the attributes (including methods) into a dictionary.
  • When the class definition is over, Python determines the metaclass of the class. Let’s call itMeta
  • Eventually, Python executes Meta(name, bases, dct), where:
    • Meta is the metaclass, so this invocation is instantiating it.
    • name is the name of the newly created class
    • bases is a tuple of the class’s base classes
    • dct maps attribute names to objects, listing all of the class’s attributes

How do we determine the metaclass of a class? Simply stated, if either a class or one of its bases has a __metaclass__ attribute, it’s taken as the metaclass. Otherwise, type is the metaclass.

Patterns

“It’s easier to ask for forgiveness than permission (EFAP)”

One pythonic principle is “It’s easier to ask for forgiveness than permission (EFAP)”. Opposed to the approach to look before you leap, this principle states that you should first try an action and if it fails react appropriately. Python’ strong exception handling supports this principle and helps to develop robust and fault tolerant programs.

Singelton

Singeltons are objects of which only one instance is supposed to exist. Python provides several ways to implement singeltons.

Null Objects

Null objects can be used instead of the type None to avoid tests for None.

Observer

The observer pattern allows several objects to have access to the same data.

Constructor

Parameters of constructors are often assigned to instance variables. This pattern can replace a many lines of manual assignment with only one line of code.

Conclusion

Thanks for reading. Drop your comments for further discussion.


  • Nichochar

    Thanks for this. Very useful handbook. Bookmarked!

  • Yegor Roganov

    Nice article, but I find the article confusing. Data structures are about storing and organizing *data*, and have nothing to do with python concepts such as decorators, descriptors etc.

    • benregn

      Should maybe have been called “Advanced Python Concepts/Techniques” instead.

  • mattcaldwell

    Cool stuff, but you really should change the name of this article–these are NOT data structures and, although you touched on some great stuff and provided nice examples, the title is very misleading.

    • ajkumar25

      Sorry if anything went misleading. I have changed the title.

      • StoneCypher

        They’re also not design patterns :|

        • ajkumar25

          I disagree. These are constructs that you should try to use more often whenever the design pattern fits what you are trying to achieve. They only seem complex when you first encounter them but after you actually learn these design patterns, you will see that it simplifies your code.
          The genius of Python is that it is not merely a very clean and clear programming language, but it provides optimized execution for various design patterns as long as you tell Python that is what you are doing. When you write a for statement (or while) with an index which is incremented and tested, your loop will run much slower than it would in C, because you aren’t cooperating with Python. But when you tell Python that you have a loop over a certain list of things, then Python can execute that loop as fast as C would because the loop itself is actually written in C inside the Python VM. That is what a list comprehension or a generator do for you.
          And learning these constructs will help you greatly when you want to fully grok functional programming, monads and functional reactive programming. All of the major languages are evolving towards full functional power, and in many cases they are copying their features from Python after seeing how successful they have been in the Python community. Scala is probably the most notable example that is NOT copying Python because they are blazing new territory. But if you look at the last couple of enhancements in Javascript and what is planned for the future, it looks a lot like Python features.

          • StoneCypher

            But none of that is what a design pattern is.

  • bobbyrussell

    This blog post would more appropriately be titled ‘Advanced Design Patterns In Python.’ Good read nonetheless.

    • ajkumar25

      Changed : )

  • dbg

    article name should be called something like “Python’s most powerful syntactic constructs” or something like that. this isn’t about data structures OR design patterns.

    • Jakob van Bethlehem

      I fully agree with all of the above and this suggestion in particular – most people would think of design patterns of Factory, FactoryMethod, Chain, Template, Bridge, etc, etc. This blog is about advanced Python programming paradigms (and should have added more on top of the Python docs itself, where these things are also quite well explained)

  • PeanutGallery6

    You should call it Python Interview questions. :) That is the only place where I see most of these.

    • ajkumar25

      I think i should leave this post as **untitled** :D

  • Candace Cady

    I don’t care what you call this discussion, your comments about ‘powerful’ and ‘efficiency’ caught my eye! thanks for posting

  • Gabriel Pozoz

    This’s a useful post, thanks!

  • Artem K.

    Thanks for the great article!

  • Paddy3118

    You could have mentioned that bit about using the Borg pattern for what other languages would use the singleton for. (Is it just me or does “singleton” remind you of a fine Scottish whiskey)?