You are viewing kristiannielsen

Kristian Nielsen - Learning Python
July 12th, 2009
09:27 am

[Link]

Previous Entry Add to Memories Share Next Entry
Learning Python

Among other things, these past few months I have been working on setting up Buildbot, including adding various enhancements and bug fixes that are needed to properly build and test the MariaDB and MySQL code base.

Since Buildbot is written in Python, this means I have also had to learn Python. I am an old-time Perl hacker, so this exercise feels a bit like living in enemy territory ;-)

Anyway, Python is often touted as a more "pretty" language. And in many ways it is. Still, it is not without its own gotchas. Think "scope rules". Obviously someone haven't been reading up on the subject before implementing things in Python, causing the language to behave stupidly (and certainly different from what one expects) in the following three cases that I hit during my Buildbot work.

First assignment is implicit scope declaration

    def foo():
	s = 0
	def inc():
	    s = s + 1
	print s
	inc()
	print s
This results in this error:
UnboundLocalError: local variable 's' referenced before assignment
Why? Because assigning to `s' declares a new variable. Yep, that's right, a nested scope can read the value of a variable in an outer scope, but it cannot assign it!

This is the work-around:

    def foo():
	s = { 'blarg': 0 }
	def inc():
	    s['blarg'] = s['blarg'] + 1
	print s['blarg']
	inc()
	print s['blarg']
Now the inner scope in inc() does not assign to the outer variable `s'. It merely reads the value, and updates the dictionary it contains. So now things work. Hm ...

Class vs. instance members

    class Bar:
	s = 0
	def foo(self, x):
	    self.s += x
	    print self.s

    a = Bar()
    a.foo(5)
    b = Bar()
    b.foo(8)
So this example actually works as one would expect from first glance (it prints "5" then "8"). But then when I looked closer, I did not understand how it could work. That s = 0 creates a class member, shared by all instances of the class. So how can each instance still get their own private copy, each correctly initialised to 0?

Ah, the answer is another variant of assignment creating a new scope. Look at self.s += x. This statement first reads s.self, which provides the value of the class member. It then assigns the new value to s.self, but since this is assignment, it now refers to an instance member, so it creates a new instance member! I don't know what those Python guys where thinking when they made self.s refer to two different variables in a single statement ...

So this means that while the above example works as expected, this very similar one does not:

    class Bar:
	s = []
	def foo(self, x):
	    self.s.append(x)
	    print self.s

    a = Bar()
    a.foo(5)
    b = Bar()
    b.foo(8)
The last statement prints [5,8] as self.s is now a class member shared among all instances.

The work-around here is to initialise member variables in the constructor __init__(), not in the class declaration.

    class Bar:
	def __init__(self):
	    self.s = []
	def foo(self, x):
	    self.s.append(x)
	    print self.s

    a = Bar()
    a.foo(5)
    b = Bar()
    b.foo(8)

Late-binding closure construction

    b = []
    for i in range(10):
	b.append(lambda x: i)

    b[0](42)
    b[3](42)
This outputs the same value "9" twice. All of the functions in the list return 9! Oops.

The reason is apparently that the closure created by (lambda ...) does late binding of captured outer variables, meaning that it refers to the name, not to the value at the time of closure construction. This is unlike any other language I have ever seen that has lexical scoping, so quite confusing.

I know of two work-arounds in this case, neither of them pretty.

One is to use a dummy extra parameter with a default value:

    b = []
    for i in range(10):
	b.append(lambda x, dummy=i: dummy)

    b[0](42)
    b[3](42)
See, when the variable i appears in the default value of a parameter, it is bound early (so to the value of i is used, not the name), different from when the variable appears in the body of the lambda expression.

The other work-around is to build and call an extra closure to force binding to the correct value:

    b = []
    for i in range(10):
	b.append( (lambda j: (lambda x: j)) (i) )

    b[0](42)
    b[3](42)
This time, passing the value of i to the outer lambda forces early binding, so we get the expected results.

Something to be aware of for an old-time Perl hacker like me, used to using functional style when programming...

Tags: , , ,

(5 comments | Leave a comment)

Comments
 
From:(Anonymous)
Date:July 12th, 2009 09:29 am (UTC)

late binding in perl

(Link)
perl -we 'use strict; my $i; my @a; for $i (1..10) { push @a, sub { print "$i\n" }} map &$_, @a'

perl -we 'use strict; my $i=0; my @a; while ($i++ < 10) { push @a, sub { print "$i\n" }} map &$_, @a'

note the difference...
From:(Anonymous)
Date:July 12th, 2009 10:03 am (UTC)

Sergei Golubchik

(Link)
Yeah, been there done that.
Had to learn python when I was hacking projects that use it,
synce, bzr, for example.

Still prefer perl when I have a choice :)
From:(Anonymous)
Date:July 12th, 2009 04:49 pm (UTC)

list comprehensions

(Link)
For the closure/functional style of code writing, you may want to check out list comprehensions and generator expressions. (I know you were just demonstrating scoping oddness- but I mention them because they are much more common in Python programming than using lambda these days.

For instance, you could rewrite your for loop with lambda above as:

b= []
[ b.append(x) for x in range(10)]

Or, just as:

b= [ x for x in range(10) ]

Of course, that's no fun, you could just do b=range(10) :) ... You can do more exciting things inside the list comprehension, like:

b= [ x for x in range(10) if x % 2 == 0 ]

From:paddy3118.blogspot.com
Date:July 13th, 2009 05:45 am (UTC)

Info on Scope rules for python with examples

(Link)
Try this page (http://rosettacode.org/wiki/Scope_modifiers#Python) for more information, with examples, for things like nonlocal. - Paddy.
From:(Anonymous)
Date:October 23rd, 2009 11:51 pm (UTC)

Better way to solve your closure code

(Link)
Late-binding closure construction

from functools import partial # this is what you want

def K_combinator(x, y): # classic CS name for this function
return x

b = []
for i in range(10):
b.append(partial(K_combinator, i))
b[0](42)
b[3](42)

I presume you want to do more than the code you've shown.
partial gets me through a _lot_ of those "problems" and I
can write in a more functional style.
Powered by LiveJournal.com