Developer Tools in Python

Python has evolved an extensive ecosystem of modules intended to make the lives of Python developers easier by eliminating the need to build everything from scratch. That same philosophy has been applied to the tools developers use to do their work, even if they are not used in the final version of a program. This article focusses on all the Developer Tools ever to be required by a Python developer.

The most basic form of help for developers is the documentation for code they are using. The pydoc module generates formatted reference documentation from the docstrings included in the source code for any importable module. Python includes two testing frameworks for automatically exercising code and verifying that it works correctly. doctest extracts test scenarios from examples included in documentation, either inside the source or as stand-alone files. unittest is a fullfeatured automated testing framework with support for fixtures, predefined test suites, and test discovery.

The trace module monitors the way Python executes a program, producing a report showing how many times each line was run. That information can be used to find code paths that are not being tested by an automated test suite and to study the function call graph to find dependencies between modules. Writing and running tests will uncover problems in most programs. Python helps make debugging easier, since in most cases, unhandled errors are printed to the console as tracebacks. When a program is not running in a text console environment, traceback can be used to prepare similar output for a log file or message dialog. For situations where a standard traceback does not provide enough information, use cgitb to see details like local variable settings at each level of the stack and source context. cgitb can also format tracebacks in HTML, for reporting errors in web applications.

Once the location of a problem is identified, stepping through the code using the interactive debugger in the pdb module can make it easier to fix by showing what path through the code was followed to get to the error situation and experimenting with changes using live objects and code. After a program is tested and debugged so that it works correctly, the next step is to work on performance. Using profile and timeit, a developer can measure the speed of a program and find the slow parts so they can be isolated and improved. Python programs are run by giving the interpreter a byte-compiled version of the original program source. The byte-compiled versions can be created on the fly or once when the program is packaged. The compileall module exposes the interface installation programs and packaging tools used to create files containing the byte code for a module. It can be used in a development environment to make sure a file does not have any syntax errors and to build the byte-compiled files to package when the program is released.
At the source code level, the pyclbr module provides a class browser that a text editor or other program can use to scan Python source for interesting symbols, such as functions and classes, without importing the code and potentially triggering side-effects.

Documentation Strings and the doctest Module

If the first line of a function, class, or module is a string, that string is known as a documentation string.The inclusion of documentation strings is considered good style because these strings are used to supply information to Python software development tools. For example, the help() command inspects documentation strings, and Python IDEs look
at the strings as well. Because programmers tend to view documentation strings while
experimenting in the interactive shell, it is common for the strings to include short
interactive examples. For example:

A common problem with writing documentation is keeping the documentation synchronized
with the actual implementation of a function. For example, a programmer might modify a function but forget to update the documentation. To address this problem, use the doctest module. doctest collects documentation strings, scans them for interactive sessions, and executes them as a series of tests.To use doctest, you typically create a separate module for testing. For example, if the previous Test class example is in a file mult.py, you would create a file testmult.py for testing, as follows:

In this code, the call to doctest.testmod(module) runs tests on the specified module and returns the number of failures and total number of tests executed. No output is produced if all of the tests pass. Otherwise, you will get a failure report that shows the difference between the expected and received output. If you want to see verbose output of the tests, you can use testmod(module, verbose=True).

As an alternative to creating a separate testing file, library modules can test themselves by including code such as this at the end of the file:

To run the tests, use doctest as the main program via the -m option. Usually, no output is produced while the tests are running, so include the -v option to make the output more verbose.

Unit Testing and the unittest Module

For more exhaustive program testing, use the unittest module.With unit testing, a developer writes a collection of isolated test cases for each element that makes up a program (for example, individual functions, methods, classes, and modules).These tests are then run to verify correct behavior of the basic building blocks that make up larger programs. As programs grow in size, unit tests for various components can be combined to create large testing frameworks and testing tools.This can greatly simplify the task of verifying correct behavior as well as isolating and fixing problems when they do occur.

Basic use of unittest involves defining a class that inherits from unittest.TestCase. Within this class, individual tests are defined by methods starting with the name ’test‘—for example, ‘testsimplestring‘, ‘testtypeconvert‘, and so on. (It is important to emphasize that the names are entirely up to you as long as they start with ’test‘.) Within each test, various assertions are used to check for different conditions.

Practical Example:

Say you have a program that has a method whose output goes to standard Output
(sys.stdout). This almost always means that it emits text to the screen. You’d like to
write a test for your code to prove that, given the proper input, the proper output is

The built-in print function, by default, sends output to sys.stdout. In order to test that output is actually getting there, you can mock it out using a stand-in object, and then make assertions about what happened. Using the unittest.mock module’s patch() method makes it convenient to replace objects only within the context of a running test, returning things to their original state immediately after the test is complete. Here’s the test code for urlprint():

The urlprint() function takes three arguments, and the test starts by setting up dummy arguments for each one. The expected_url variable is set to a string containing the expected output. To run the test, the unittest.mock.patch() function is used as a context manager to replace the value of sys.stdout with a StringIO object as a substitute. The fake_out variable is the mock object that’s created in this process. This can be used inside the body of the with statement to perform various checks. When the with statement completes, patch conveniently puts everything back the way it was before the test ever ran. It’s worth noting that certain C extensions to Python may write directly to standard output, bypassing the setting of sys.stdout. This example won’t help with that scenario, but it should work fine with pure Python code (if you need to capture I/O from such C extensions, you can do it by opening a temporary file and performing various tricks involving file descriptors to have standard output temporarily redirected to that file).

The Python Debugger and the pdb Module

Python includes a simple command-based debugger which is found in the pdb module.
The pdb module supports post-mortem debugging, inspection of stack frames, breakpoints,
single-stepping of source lines, and code evaluation.

There are several functions for invoking the debugger from a program or from the interactive Python shell.

Of all of the functions for launching the debugger, the set_trace() function may be the easiest to use in practice. If you are working on a complicated application but you have detected a problem in one part of it, you can insert a set_trace() call into the code and simply run the application.When encountered, this will suspend the program and go directly to the debugger where you can inspect the execution environment. Execution resumes after you leave the debugger.

Say your program is broken and you’d like some simple strategies for debugging it.

If your program is crashing with an exception, running your program as python3 -i
can be a useful tool for simply looking around. The -i option starts
an interactive shell as soon as a program terminates. From there, you can explore the
environment. For example, suppose you have this code:

Running python3 -i produces the following:

If you don’t see anything obvious, a further step is to launch the Python debugger after
a crash. For example:

If your code is deeply buried in an environment where it is difficult to obtain an interactive
shell (e.g., in a server), you can often catch errors and produce tracebacks yourself.
For example:

If your program isn’t crashing, but it’s producing wrong answers or you’re mystified by
how it works, there is often nothing wrong with just injecting a few print() calls in
places of interest. However, if you’re going to do that, there are a few related techniques
of interest. First, the traceback.print_stack() function will create a stack track of
your program immediately at that point. For example:

Alternatively, you can also manually launch the debugger at any point in your program
using pdb.set_trace() like this:

This can be a useful technique for poking around in the internals of a large program
and answering questions about the control flow or arguments to functions. For instance,
once the debugger starts, you can inspect variables using print or type a command such
as w to get the stack traceback.

Don’t make debugging more complicated than it needs to be. Simple errors can often be resolved by merely knowing how to read program tracebacks (e.g., the actual error is usually the last line of the traceback). Inserting a few selected print() functions in your code can also work well if you’re in the process of developing it and you simply want some diagnostics (just remember to remove the statements later). A common use of the debugger is to inspect variables inside a function that has crashed. Knowing how to enter the debugger after such a crash has occurred is a useful skill to know. Inserting statements such as pdb.set_trace() can be useful if you’re trying to unravel an extremely complicated program where the underlying control flow isn’t obvious. Essentially, the program will run until it hits the set_trace() call, at which point it will immediately enter the debugger. From there, you can try to make more sense of it. If you’re using an IDE for Python development, the IDE will typically provide its own debugging interface on top of or in place of pdb. Consult the manual for your IDE for more information.

Here’s a list of resources to get started with the Python debugger:

  1. Read Steve Ferb’s article “Debugging in Python”
  2. Watch Eric Holscher’s screencast “Using pdb, the Python Debugger”
  3. Read Ayman Hourieh’s article “Python Debugging Techniques”
  4. Read the Python documentation for pdb — The Python Debugger
  5. Read Chapter 9—When You Don’t Even Know What to Log: Using Debuggers—of Karen Tracey’s Django 1.1 Testing and Debugging.

Program Profiling

The profile and cProfile modules are used to collect profiling information. Both modules work in the same way, but cProfile is implemented as a C extension, is significantly
faster, and is more modern. Either module is used to collect both coverage information (that is, what functions get executed) as well as performance statistics.The easiest way to profile a program is to execute it from the command line as follows:

Alternatively, the following function in the profile module can be used:

Executes the contents of command using the exec statement under the profiler.
filename is the name of a file in which raw profiling data is saved. If it’s omitted, a
report is printed to standard output.
The result of running the profiler is a report such as the following:

When there are two numbers in the first column (for example, “121/1″), the latter is
the number of primitive calls and the former is the actual number of calls. Simply inspecting the generated report of the profiler is often enough for most applications of this module—for example, if you simply want to see how your program is spending its time. However, if you want to save the data and analyze it further, the pstats module can be used.

Say you would like to find out where your program spends its time and make timing measurements.

If you simply want to time your whole program, it’s usually easy enough to use something
like the Unix time command. For example:

More often than not, profiling your code lies somewhere in between these two extremes.
For example, you may already know that your code spends most of its time in a few
selected functions. For selected profiling of functions, a short decorator can be useful.
For example:

To use this decorator, you simply place it in front of a function definition to get timings
from it. For example:

To time a block of statements, you can define a context manager. For example:

Here is an example of how the context manager works:

For studying the performance of small code fragments, the timeit module can be useful.
For example:

timeit works by executing the statement specified in the first argument a million times
and measuring the time. The second argument is a setup string that is executed to set
up the environment prior to running the test. If you need to change the number of
iterations, supply a number argument like this:

When making performance measurements, be aware that any results you get are approximations. The time.perf_counter() function used in the solution provides the
highest-resolution timer possible on a given platform. However, it still measures wallclock
time, and can be impacted by many different factors, such as machine load.
If you are interested in process time as opposed to wall-clock time, use time.pro
instead. For example:

Last, but not least, if you’re going to perform detailed timing analysis, make sure to read
the documentation for the time, timeit, and other associated modules, so that you have
an understanding of important platform-related differences and other pitfalls.

The most basic starting point in the profile module is run(). It takes a string statement
as argument and creates a report of the time spent executing different lines of code
while running the statement.

Making Your Programs Run Faster

Your program runs too slow and you’d like to speed it up without the assistance of more
extreme solutions, such as C extensions or a just-in-time (JIT) compiler.

While the first rule of optimization might be to “not do it,” the second rule is almost
certainly “don’t optimize the unimportant.”

More often than not, you’ll find that your program spends its time in a few hotspots,
such as inner data processing loops. Once you’ve identified those locations, you can use
the no-nonsense techniques presented in the following sections to make your program
run faster.

Use functions

A lot of programmers start using Python as a language for writing simple scripts. When
writing scripts, it is easy to fall into a practice of simply writing code with very little
structure. For example:

A little-known fact is that code defined in the global scope like this runs slower than
code defined in a function. The speed difference has to do with the implementation of
local versus global variables (operations involving locals are faster). So, if you want to
make the program run faster, simply put the scripting statements in a function:

The speed difference depends heavily on the processing being performed, but in our
experience, speedups of 15-30% are not uncommon.

Selectively eliminate attribute access

Every use of the dot (.) operator to access attributes comes with a cost. Under the covers,
this triggers special methods, such as __getattribute__() and __getattr__(), which
often lead to dictionary lookups.
You can often avoid attribute lookups by using the from module import name form of
import as well as making selected use of bound methods. To illustrate, consider the
following code fragment:

When tested on my machine, this program runs in about 40 seconds. Now change the
compute_roots() function as follows:

This version runs in about 29 seconds. The only difference between the two versions of
code is the elimination of attribute access. Instead of using math.sqrt(), the code uses
sqrt(). The result.append() method is additionally placed into a local variable result_append and reused in the inner loop.
However, it must be emphasized that these changes only make sense in frequently executed
code, such as loops. So, this optimization really only makes sense in carefully selected places.

Understand locality of variables

As previously noted, local variables are faster than global variables. For frequently accessed
names, speedups can be obtained by making those names as local as possible.
For example, consider this modified version of the compute_roots() function just

In this version, sqrt has been lifted from the math module and placed into a local
variable. If you run this code, it now runs in about 25 seconds (an improvement over
the previous version, which took 29 seconds). That additional speedup is due to a local
lookup of sqrt being a bit faster than a global lookup of sqrt.
Locality arguments also apply when working in classes. In general, looking up a value
such as self.name will be considerably slower than accessing a local variable. In inner
loops, it might pay to lift commonly accessed attributes into a local variable. For example:

Avoid gratuitous abstraction

Any time you wrap up code with extra layers of processing, such as decorators, properties,
or descriptors, you’re going to make it slower. As an example, consider this class:

Now, try a simple timing test:

As you can observe, accessing the property y is not just slightly slower than a simple
attribute x, it’s about 4.5 times slower. If this difference matters, you should ask yourself
if the definition of y as a property was really necessary. If not, simply get rid of it and
go back to using a simple attribute instead. Just because it might be common for programs
in another programming language to use getter/setter functions, that doesn’t
mean you should adopt that programming style for Python.

Use the built-in containers

Built-in data types such as strings, tuples, lists, sets, and dicts are all implemented in C,
and are rather fast. If you’re inclined to make your own data structures as a replacement
(e.g., linked lists, balanced trees, etc.), it may be rather difficult if not impossible to match
the speed of the built-ins. Thus, you’re often better off just using them.

Avoid making unnecessary data structures or copies

Sometimes programmers get carried away with making unnecessary data structures
when they just don’t have to. For example, someone might write code like this:

Perhaps the thinking here is to first collect a bunch of values into a list and then to start
applying operations such as list comprehensions to it. However, the first list is completely
unnecessary. Simply write the code like this:

Related to this, be on the lookout for code written by programmers who are overly
paranoid about Python’s sharing of values. Overuse of functions such as copy.deep
copy() may be a sign of code that’s been written by someone who doesn’t fully understand
or trust Python’s memory model. In such code, it may be safe to eliminate many
of the copies.

Before optimizing, it’s usually worthwhile to study the algorithms that you’re using first.
You’ll get a much bigger speedup by switching to an O(n log n) algorithm than by
trying to tweak the implementation of an an O(n**2) algorithm.
If you’ve decided that you still must optimize, it pays to consider the big picture. As a
general rule, you don’t want to apply optimizations to every part of your program,
because such changes are going to make the code hard to read and understand. Instead,
focus only on known performance bottlenecks, such as inner loops.
You need to be especially wary interpreting the results of micro-optimizations. For
example, consider these two techniques for creating a dictionary:

The latter choice has the benefit of less typing (you don’t need to quote the key names).
However, if you put the two code fragments in a head-to-head performance battle, you’ll
find that using dict() runs three times slower! With this knowledge, you might be
inclined to scan your code and replace every use of dict() with its more verbose alternative.
However, a smart programmer will only focus on parts of a program where it might actually matter, such as an inner loop. In other places, the speed difference just isn’t going to matter at all. If, on the other hand, your performance needs go far beyond the simple techniques in
this article, you might investigate the use of tools based on just-in-time (JIT) compilation techniques. For example, the PyPy project is an alternate implementation of the Python interpreter that analyzes the execution of your program and generates native machine code for frequently executed parts. It can sometimes make Python programs run an order of magnitude faster, often approaching (or even exceeding) the speed of code written in C. Unfortunately, as of this writing, PyPy does not yet fully support Python 3. So, that is something to look for in the future. You might also consider the Numba project. Numba is a dynamic compiler where you annotate selected Python functions that you want to optimize with a decorator. Those functions are then compiled into native machine code through the use of LLVM. It too can produce significant performance gains. However, like PyPy, support for Python 3 should be viewed as somewhat experimental.

Last, but not least, the words of John Ousterhout come to mind: “The best performance
improvement is the transition from the nonworking to the working state.” Don’t worry
about optimization until you need to. Making sure your program works correctly is
usually more important than making it run fast (at least initially).

  • dash

    really enjoyed the tutorial, keep doing this dude :)