

Movable IDLE for Python 2.5 on Windows


Movable IDLE for Python 2.5



Like this post? Digg it or it.


Categories: , , Tags: , , , ,

The Python Object Model Revisited (data descriptors)

emoticon:paper A few weeks ago I demonstrated the complexity of the Python object model by fetching docstrings from objects. A while after posting it I thought of a bug - or at least a way in which it could return the wrong result when looking up an attribute on an object. It will probably come as no surprise that this is due to the descriptor protocol.

Descriptors are special types of objects that have __get__ and or __set__ and __delete__ methods and have special behaviour when fetched, set or deleted as object attributes. They are how methods, class methods, static methods, properties and __slots__ are implemented in Python.

Descriptors that have both __get__ and __set__ are called data descriptors (properties are the canonical example), descriptors with only __get__ are non-data descriptors (methods being the canonical example). Data descriptors have interesting behaviour when they are on a class which has the same member in the instance dictionary.

Instance members are stored in the __dict__ attribute of the object. Normally if this instance dictionary has a member then fetching that member will pull it out of the dictionary. The exception is that if the class has a data-descriptor with the same name then that will be invoked instead of the object in the instance dictionary. This is easy to demonstrate:

>>> class A(object):
...     @property
...     def a(self):
...         return 'property'
>>> a = A()
>>> a.__dict__['a'] = 'attribute'
>>> a.a

So a data-descriptor on the class will override a member with the same name on the instance - but the 18 lines of code I wrote before for fetching docstrings from attributes will always look on the instance first.

The same is true for inherited data-descriptors:

>>> class B(A): pass
>>> b = B()
>>> b.__dict__['a'] = 'attribute'
>>> b.a

Non-data descriptors don't override instance attributes and data-descriptors on a base class don't override normal class attributes on a subclass.

To handle this we need to check both the instance and walk the inheritance hierarchy. If we find the member we are looking for in both then we check the member from the class for a __set__ method. If the member from the class (or one of its base classes) has a __set__ member then we return that - otherwise we return the member from the instance.

Our modified full code that takes this into account has grown to 22 lines and now looks like:

import types
import inspect

def get_doc(obj, member):
    found = []
    if hasattr(obj, '__dict__') and member in obj.__dict__:

    if isinstance(obj, (type, types.ClassType)):
        search_order = inspect.getmro(obj)
        search_order = inspect.getmro(obj.__class__)

    for entry in search_order:
        if member in entry.__dict__:
            if hasattr(entry.__dict__[member], '__set__'):
                return entry.__dict__[member].__doc__
            return found[0].__doc__

def get_docstrings(obj):
        members = dir(obj)
    except Exception:
        members = []
    return [(member, get_doc(obj, member)) for member in members]


In practise there is another exception that we haven't handled here. Although you can override methods with instance attributes (very useful for monkey patching methods for test purposes) you can't do this with the Python protocol methods. These are the 'magic methods' whose names begin and end with double underscores. When invoked by the Python interpreter they are looked up directly on the class and not on the instance (however if you look them up directly - e.g. x.__repr__ - normal attribute lookup rules apply).

There is a corner case (that I alluded to in my previous post), classes can define __slots__ and create a dummy __dict__ member. If this member isn't a dictionary then our code will barf horribly - but really this is such an evil corner case that I'm not going to worry about it. Smile

I have seen one use case for __slots__ in combination with a fake __dict__ member: proxying attribute access. This is a part of the werkzeug web framework - the LocalProxy class defines __dict__ as a property which returns the __dict__ member of the object it is proxying...

Like this post? Digg it or it.

Posted by Fuzzyman on 2009-06-22 23:08:08 | |

Categories: , Tags: ,

discover: Test discovery for unittest backported to Python 2.4+

emoticon:development I kind of promised you no more entries on unittest for a while, but oh well. Smile

I've backported the test discovery in Python-trunk, what will become Python 2.7 & Python 3.2. Test discovery allows you to run all the unittest based tests (or just a subset of them) in your project without you having to write your own test collection or running machinery. Once installed, test discovery can be invoked with python -m discover. I've tested the discover module with Python 2.4 and 3.0.

Most of the work of backporting was providing an implementation of os.path.relpath (added in Python 2.6) and refactoring the command line handling for standalone use.

The discover module also implements the load_tests protocol which allows you to customize test loading from modules and packages. Test discovery and load_tests are implemented in the DiscoveringTestLoader which can be used from your own test framework.

This is the test discovery mechanism and load_tests protocol for unittest backported from Python 2.7 to work with Python 2.4 or more recent (including Python 3).


python -m discover


Usage: [options]

  -v, --verbose    Verbose output
  -s directory     Directory to start discovery ('.' default)
  -p pattern       Pattern to match test files ('test*.py' default)
  -t directory     Top level directory of project (default to
                   start directory)

For test discovery all test modules must be importable from the top
level directory of the project.

For example to use a different pattern for matching test modules run:

python -m discover -p '*'

(Remember to put quotes around the test pattern or shells like bash will do shell expansion rather than passing the pattern through to discover.)

Test discovery is implemented in As well as using discover as a command line script you can import DiscoveringTestLoader, which is a subclass of unittest.TestLoader, and use it in your test framework.

This method finds and returns all test modules from the specified start directory, recursing into subdirectories to find them. Only test files that match pattern will be loaded. (Using shell style pattern matching.)

All test modules must be importable from the top level of the project. If the start directory is not the top level directory then the top level directory must be specified separately.

The load_tests protocol allows test modules and packages to customize how they are loaded. This is implemented in discover.DiscoveringTestLoader.loadTestsFromModule. If a test module defines a load_tests function then tests are loaded from the module by calling load_tests with three arguments: loader, standard_tests, None.

If a test package name (directory with matches the pattern then the package will be checked for a load_tests function. If this exists then it will be called with loader, tests, pattern.

If load_tests exists then discovery does not recurse into the package, load_tests is responsible for loading all tests in the package.

The pattern is deliberately not stored as a loader attribute so that packages can continue discovery themselves. top_level_dir is stored so load_tests does not need to pass this argument in to is maintained in a google code project (where bugs and feature requests should be posted):

The latest development version of can be found at:

Like this post? Digg it or it.

Posted by Fuzzyman on 2009-06-20 19:35:56 | |

Categories: , Tags: , ,

Catching up: Pythonutils 0.4.0, akismet 0.2.0 and article updates





This release (0.2.0) adds compatibility with Google AppEngine.

All changes in 0.2.0:

  • Added a - install with pip or easy_install

pythonutils 0.4.0



Like this post? Digg it or it.


Categories: , , , Tags: , , ,

Hosted by Webfaction