
Python Language Features and Tricks (sahandsaba.com)
404 points by Bocker on Mar 8, 2014 | hide | past | web | favorite | 85 comments



I've been programming in Python for about 3 years and learned a couple of tricks from this post, which was why I submitted it.

There's always tricks to learn, and it's hard to remember everything when you're first starting out with a new language.

Extended unpacking was a new one on me. Kind of like a simple version of pattern matching. I think this one will prove useful!

(I've been using python for many years, but not full time. It's the language I use for one-off scripts, small tools that make use of its handy standard library, or prototypes for things I'm going to write in some other language. That's my excuse for the gaps in my knowledge.)

I've been using python since the 90s, and I learned a few things from this post.

Right after this string of replies, I was expecting Guido van Rossum would be next, saying he created Python and even he learnt something new...

> ... I wasn't aware of named slices for some reason ....

I wasn't either. But I wouldn't call them a hidden feature of the language. A slice object is an object. So you can make it the value of a named variable, just like any other object. Somehow that wasn't obvious, though.

Now I'm wondering how else I might be able to improve my code by giving names to non-obvious things.

Same here, learned something new today.

A common theme running through named tuples, itertools and collections is Raymond Hettinger. He has also put together lots of tutorials and talks.

Great suggestion. Just searched for him on youtube and a couple of excellent looking talks popped up. Halfway through and this one seems great so far: https://www.youtube.com/watch?v=OSGv2VnC0go

Really enjoy his style as well :)

Thank you sir, something else to go through on a quiet Sunday :).


I think unpacking is just unpacking, languages that have pattern matching allow you to do something identical:

# let i = (1,2,3) ;;

val i : int * int * int = (1, 2, 3)

# let a,b,c = i;;

val a : int = 1

val b : int = 2

val c : int = 3

Unpacking pretty much does one thing, and it makes that one thing easier and a lot more readable. Combine that with list comprehension (from what I understand C#'s LINQ is similar) and you end up with code that's highly maintainable/readable (as long as you name your variables appropriately, of course).

I believe Python supports pattern matching other than Regex as well.

Python list comprehensions are not lazy, though. If you generate [for x in xrange(1, 10000)] you'll get 10000 elements in your list. My understanding is that LINQ list comprehensions are that.

> I believe Python supports pattern matching other than Regex as well.

That's pattern matching on strings. The kind of pattern matching being discussed here is pattern matching on data types (see [1]).

1: http://en.wikipedia.org/wiki/Pattern_matching

Python does have generator expressions, for instance what you wrote above is written (lazily) as:(x for x in xrange(1, 10000))

It doesnt have functional style pattern matching though.

'Lazy lists' are called generators in Python. Just use (...) Instead of [...] in your example.

I'm aware of generator expressions, but the fact that list comprehensions are not lazy can trip people if they are used to other languages with this feature, since they are usually lazy.

Reading through http://en.wikipedia.org/wiki/List_comprehension, it seems that there are roughly equal numbers of languages where "list comprehension" produces strict list as languages where it produces lazy lists.

I think it's a lack of naming convention - the word "list" on its own means "strict list" in some languages and "lazy list" in others.

Fair enough, I retract my statement :) Maybe people are not surprised after all.

I don't agree, people are more likely to come from python that to python from a language that has lazy list comprehensions. But regardless, using a language requires that you learn it and I doubt many people think list comp is lazy. Pythonic programming obeys explicit over implicit, which which if you want a lazy version you explicitly use a lazy version.


What does this have to do with anything? If the construct was defined as lazy, then it would be just as explicit. There is nothing about generator expressions that says "here, here, this is lazy".







I suppose everything is relative...

In isolated offline code, I think Ruby approaches the unreadable - but only because DSLs are fashionable, and you have to understand the library involved to understand the magic that's going on behind the scenes. With sufficient online searching, you can understand things at a surface level, and with enough time with a debugger you can appreciate all the mechanics that are happening, but IMHO it still leaves you with an unpleasant taste in your mouth at the sheer quantity of magic.

I do prefer Ruby to Python though. I much prefer monadic enumerators combined with lambdas to list comprehensions, even though these tend to be eager in Ruby but have the option of being lazy in Python. Itertools helps, but is deeply hampered by Python's overly verbose lambda syntax.

Been thinking about Python vs Ruby lately. I dread reading other people's Ruby code (eg. if there is a problem in a library I'm using) but I don't have that same dread reading Python code. Now if Python had blocks and RubyGems then I would never give Ruby a second look.

What does RubyGems provide that isn't covered by pypy/eggs? I thought ruby and python packaging was approximated feature parity here (which is to say, both leave something to be desired, but work quite well).

But that's the thing y'know... It's not just about DSL but it's also Ruby's syntax that include lots of symbols (of course DSL made the learning curve increase as well).

By symbol I mean the use of tilda, pound sign, ampersand colon.

Seems to me that it's more of personal preference than anything else and this is why for me personally, Ruby is pretty close to Perl.

Somehow I prefer verbose than too terse. I prefer to read the code in words than in mix of words and symbols.

The comparison functions are there to address the comparison for equality of things that don't have an order. So in some sense, they are more explicit (a minute with the docs explains that implementations only need to directly define 2 of them). -1,0,1 style comparisons still exist in Python 2.7, they were dropped in 3.0. Probably one of those things that could have gone either way.

I think comparisons about readability aren't very useful (they are subjective, 'enough' counts), but I sure like the relative terseness of Python (to the point that I lament people who come over and write Java in Python (I don't mean to direct that at you, it's just a thing that happens, where people don't take advantage of things that are very idiomatic and thus clear even when terse)).

Fair point and I learned early on during my college years not to import habit from one language to the other one. Mainly because "idiomatic" tend to depend on the internal implementation of the language.

The language has expanded a lot recently (2.x) series. It is no longer the language it once was. It is more advanced and not as svelte.

There's similar thing re eq/gte/etc for Python. Particulars escape me at the moment.



Instead of



When did support for dictionary comprehensions make it into 2.x? I could've sworn it didn't used to work, but I just tried it in the shell and sure enough, it does in 2.7.3.

Wow I didn't know that! I was always doing something like this

    mi = dict((v,k) for k,v in m.iteritems())

more importantly it's wrong - the iteration order of .values() and .keys() is not guaranteed to be consistent

The order of items in the dictionary isn't stable, but the order of .keys() is guaranteed to be the same as the order of .values() (as long as you don't modify the dict in between calling one then the other).


I have two questions.



Unpacking is a limited form of what is called destructuring in other languages like Clojure. I would say that, in terms of feature-richness: unpacking < destructuring < pattern matching.

I'd extend that one step further

    unpacking < destructuring < pattern matching < first-class patterns
where first-class patterns are increasingly becoming available in some languages which offer pattern matching (in particular, Haskell will have them soon).

I did a little googling, but am finding it difficult to find good clear information - do you have any articles where I can read about first-class patterns?


That's the haskell extension.

To see a very nice use of them, check out this paper (pdf) http://strictlypositive.org/CJ.pdf

How do view patterns make patterns "first-class"? To me that means able to manipulate them as values, I don't see how view patterns allow that, they're just syntactic sugar for case expressions.


I spoke too eagerly—the new feature is just named and namespaced patterns. It's a bit of a bump in power, but it's not fully general yet.

For true(-ish) first-class patterns take a look at Prisms in the lens package or some of the other first-class pattern libraries.

I don't know Clojure, but I do know Python, and I'd like to say that unpacking is more flexible than you may realize. For example, you can do this:

  a,(b,c),d = [1,[2,3],4]

I don't mean to belittle Python here, I think its a great language and I find its unpacking useful. I'm only trying to demonstrate that the concept can be (and is in some other languages, like Clojure) taken further to make it more useful still.

I think that Python's unpacking allows most, if not all, of Clojures sequence destructuring for tuples and lists. Clojure takes it a bit further, however, by applying it to all sequences. For example, you could do this:

    a,b,c = "XYZ"
because strings are also sequences. You can also do something like this (excuse the awkward syntax as I try to express it in pseudo-Python):

    a,b : c as d = [1,2,3,4,5]
    # a = 1
    # b = 2
    # c = [3, 4, 5]
    # d = [1, 2, 3, 4, 5]
This might not be so useful in python, since the list already is d and c is simply slicing the end from the list, but Clojure allows you to destructure function arguments: (defn foo [[a, b & c :as d]] ...) when passed the above list (foo [1 2 3 4 5]) would bind the variables as shown in the above comments.

Where destructuring really shines, though, is that you can destructure maps (dictionaries in Python) and vectors can also be treated as maps (their keys are the indices), so you can do stuff like this:

    a, {[b {:keys [c, d]} :foo}, e = [1, {foo: [2, {c: 3, d: 4}, bar: 9}, 5]
    # a = 1
    # b = 2
    # c = 3
    # d = 4
    # e = 5



Tuple unpacking:

  t = (1, 2)
  a, b = t
  # a == 1 and b == 2
Using slices is normal with lists

Cool like this,hope implement in php. No need to explode

PHP has the list construct which is a little uglier, and has a couple of odd behaviors

    php > $t = array(1, 2);
    php > list($a, $b) = $t;
    php > echo "$a, $b";
    1, 2



I used to bookmark tutorials like this, but now ...

I just bookmark the HN discussion. Not only does that retain a link to the tutorial, but it also retains a link to a discussion that usually adds value to the tutorial.

I'm assuming that HN discussions remain available for many years. Anyone know for sure if that is/isn't true?

Instapaper has become my bookmarking tool. I don't even really sort it out anymore. I know it should be there when I need it and mostly the actual article is saved.

I hope so.

Nice reference.

1.29 happened to be exactly what I was looking for:


You can also do






Slices make sense when you realize that [n:n] is always empty, and [:n] + [n:] is always the whole list. From these two identities, all else follows.


Just remember that for positive indices, len(slice(n,m)) == m - n



I don't understand how this one to flatten lists works:

    [x for l in a for x in l]

So that flattening can also be written as:

    x = [[1,2], [3,4], [5,6]]
    [x for x in x for x in x]
More perversely, we can also flatten with my invention:

    list(None for x in x if (yield from x) and False) # Python >=3.3
More reasonably, of course,


The expression is a list comprehension with 2 nested for statements. It is similar to this, which names its result:

   result = []
   for l in a:
      for x in l:

Yes. And the order of the two for's in the list comprehension was deliberately kept the same as the order of the two for loops in your explicit code, on purpose, for ease of remembering how the former (i.e. list comp) works.

That order is the part that confused me. I expected to read it right-to-left, instead it's left to right for the for statements, then the expression on the left at the end.

I see now that the Python docs explain this very clearly...

Yes. Meant to say that but forgot - that the Python docs explain it. That's where I read it myself :)

A good reference, to be sure, but man, do I resent the term “trick” in programming. It implies a deception, or something clever that you wouldn’t think to look for, like opening a wine bottle with a shoe. These aren’t tricks, they’re (largely) standard library features that you would simply expect to exist. But maybe I’m underestimating the NIH effect.


The fair comparison is probably to some data structure (list/array/whatever makes sense) and its rich companion library.


Same as php.keep upgrading.

I know bugger all Python, but I know negative indexing.

Good for you. Did you know everything else?

> Good for you. Did you know everything else?

Obviously not, after all I did say "I know bugger all Python".

I did know about slices.

patterns / tricks = language deficiencies


For TRE: http://neopythonic.blogspot.com/2009/04/tail-recursion-elimi...

For the GIL, there are alternative implementations of Python (Jython, IronPython, etc.)

Hm... Python only supports 1000 recursions? That seems unsafe. It seems like anyone who writes functional-style code will run the risk of a stack overflow.

Believe it or not, it's actually deliberate. And not that uncommon in other languages too. The only way you can realistially have 'unlimited' recursion is to use tail call optimisation, which is deliberaterly not implemented in python:


Is Guido's take on it.

Essentially, as I understand it, his take is that massive recursion is confusing, and 'unpythonic'. And with the whole 'explicit is better than implicit' thing, I guess there is a point.

Some times recursion is the best/most obvious solution, and then it is a bit annoying to not have TCO, but you can usually work around it.

You can increase the recursion limit like this:

import sys


Kind of a "hackish" way to do things but you can do it if you need to.

^Although I think the idea is that you shouldn't need to. Any recursive implementation can either be easily translated to work iteratively, or can be implemented in such a way that 1000 levels of recursion should be more than enough.

For example, 1000 levels of recursion is more than enough to count all the nodes in a binary tree unless the tree is extremely poorly balanced or inordinately large.

No better than PHP? That's more than a little ridiculous. Every language has shortcomings, and there's no silver bullet/perfect multitool that can or should be used for everything.

It's too bad it's apparently not good for... whatever it is you do. It is pretty great for some things, and having spent more than a few years with both Python and PHP, I can say without reservation that there is absolutely no comparison in terms of language quality.

> I suggest you are suffering from the Blub Paradox[0][1][2].

Writing some C will cure that disease, but the treatment has possible side effects, like carpal tunnel syndrome.

P.G. is right about PHP. This language is better than Python in various ways. For ex. it does not scare people away if you want to sell your project, because everyone and their grandma knows PHP. Most successful web pages (eg. FB) started out with PHP, not Lisp, C++, Python, Java, but PHP.

Every language has patterns or tricks, and every language has deficiencies, but these aren't the same thing at all.

It seems you have read some generic criticism of Python to copy from, and don't have much first-hand experience

There are only handful of language runtimes that got rid of GIL entirely. Namely JVM, .NET, and Rubinius. Other languages either use non-native threads or doesn't support threading at all. (E.g. Nodejs has no threading.)

Applications are open for YC Winter 2019
