Learning Python

Among other things, these past few months I have been working on setting up Buildbot, including adding various enhancements and bug fixes that are needed to properly build and test the MariaDB and MySQL code base.

Since Buildbot is written in Python, this means I have also had to learn Python. I am an old-time Perl hacker, so this exercise feels a bit like living in enemy territory 😉

Anyway, Python is often touted as a more “pretty” language. And in many ways it is. Still, it is not without its own gotchas. Think “scope rules”. Obviously someone haven’t been reading up on the subject before implementing things in Python, causing the language to behave stupidly (and certainly different from what one expects) in the following three cases that I hit during my Buildbot work.

First assignment is implicit scope declaration

    def foo():
        s = 0
        def inc():
            s = s + 1
        print s
        inc()
        print s

This results in this error:

UnboundLocalError: local variable 's' referenced before assignment

Why? Because assigning to `s’ declares a new variable. Yep, that’s right, a nested scope can read the value of a variable in an outer scope, but it cannot assign it!

This is the work-around:

    def foo():
        s = { 'blarg': 0 }
        def inc():
            s['blarg'] = s['blarg'] + 1
        print s['blarg']
        inc()
        print s['blarg']

Now the inner scope in inc() does not assign to the outer variable `s’. It merely reads the value, and updates the dictionary it contains. So now things work. Hm …

Class vs. instance members

    class Bar:
        s = 0
        def foo(self, x):
            self.s += x
            print self.s

    a = Bar()
    a.foo(5)
    b = Bar()
    b.foo(8)

So this example actually works as one would expect from first glance (it prints “5” then “8”). But then when I looked closer, I did not understand how it could work. That s = 0 creates a class member, shared by all instances of the class. So how can each instance still get their own private copy, each correctly initialised to 0?

Ah, the answer is another variant of assignment creating a new scope. Look at self.s += x. This statement first reads s.self, which provides the value of the class member. It then assigns the new value to s.self, but since this is assignment, it now refers to an instance member, so it creates a new instance member! I don’t know what those Python guys where thinking when they made self.s refer to two different variables in a single statement …

So this means that while the above example works as expected, this very similar one does not:

    class Bar:
        s = []
        def foo(self, x):
            self.s.append(x)
            print self.s

    a = Bar()
    a.foo(5)
    b = Bar()
    b.foo(8)

The last statement prints [5,8] as self.s is now a class member shared among all instances.

The work-around here is to initialise member variables in the constructor __init__(), not in the class declaration.

    class Bar:
        def __init__(self):
            self.s = []
        def foo(self, x):
            self.s.append(x)
            print self.s

    a = Bar()
    a.foo(5)
    b = Bar()
    b.foo(8)

Late-binding closure construction

    b = []
    for i in range(10):
        b.append(lambda x: i)

    b[0](42)
    b[3](42)

This outputs the same value “9” twice. All of the functions in the list return 9! Oops.

The reason is apparently that the closure created by (lambda ...) does late binding of captured outer variables, meaning that it refers to the name, not to the value at the time of closure construction. This is unlike any other language I have ever seen that has lexical scoping, so quite confusing.

I know of two work-arounds in this case, neither of them pretty.

One is to use a dummy extra parameter with a default value:

    b = []
    for i in range(10):
        b.append(lambda x, dummy=i: dummy)

    b[0](42)
    b[3](42)

See, when the variable i appears in the default value of a parameter, it is bound early (so to the value of i is used, not the name), different from when the variable appears in the body of the lambda expression.

The other work-around is to build and call an extra closure to force binding to the correct value:

    b = []
    for i in range(10):
        b.append( (lambda j: (lambda x: j)) (i) )

    b[0](42)
    b[3](42)

This time, passing the value of i to the outer lambda forces early binding, so we get the expected results.

Something to be aware of for an old-time Perl hacker like me, used to using functional style when programming…