I haven't been able to find an understandable explanation of how to actually use Python's itertools.groupby() function. What I'm trying to do is this: take a list - in this case, the children of an objectified lxml element - divide it into groups based on some criteria, and then later iterate over each of these groups separately.

I've reviewed the documentation (, and the examples, (, but I've had trouble trying to apply them beyond a simple list of numbers.


+7  A: 

Can you show us your code?

The example on the Python docs is quite straight forward:

groups = []
uniquekeys = []
for k, g in groupby(data, keyfunc):

So in your case, data is a list of nodes, keyfunc is where the logic of your criteria function goes and then groupby() groups the data. You must be careful to sort the data by the criteria before you call groupby or it won't work. groupby method actually just iterates through a list and whenever the key changes it creates a new group.

Sebastjan Trepča
Can you remove the   chars in your answer, please? TIA!
Torsten Marek
+34  A: 

After some experimentation, I've overcome my mental block. In retrospect, it's all obvious, but in the spirit of Stack Overflow, here's what I learned.

As Sebastjan said, you first have to sort your data. This is important.

The part I didn't get is that in the example construction

groups = []
uniquekeys = []
for k, g in groupby(data, keyfunc):

"k" is the current grouping key, and "g" is an iterator that you can use to iterate over the group defined by that grouping key. In other words, the groupby iterator itself returns iterators. Here's an example of that, using clearer variable names:

from itertools import groupby


    for thing in group:
    print " "

This will give you the output:

A bear is a animal.
A duck is a animal.

A cactus is a plant.

A speed boat is a vehicle.
A school bus is a vehicle.

In this example, "things" is a list of tuples where the first item in each tuple is the group the second item belongs to. The groupby() function takes two arguments: (1) the data to group and (2) the function to group it with. Here, "lambda x: x[0]" tells groupby() to use the first item in each tuple as the grouping key.

In the above "for" statement, groupby returns three (key, group iterator) pairs - once for each unique key. You can use the returned iterator to iterate over each individual item in that group.


    listOfThings = " and ".join(["%s" % thing[1] for thing in group])

This will give you the output:

animals: bear and duck.
plants: cactus.
vehicles: speed boat and school bus.

Python's pretty cool, no?

James Sulak
Thanks for that example. I must be dumb, but unlike Sebastjan Trepča, I didn't think the doc example was so "straight forward".
I agree, I didn't think the doc was straight forward either.
Neil Kodner
+2  A: 

A neato trick with groupby is run length encoding in one line:

[(c,len(list(cgen))) for c,cs in groupby(some_string)]


+2  A: 

A correction to answer by @nt:

[(c,len(list(cs))) for c,cs in groupby(some_string)]

Thanks, the list(cs) part was what I was missing when trying to make use of the _grouper() objects returned by groupby(). This example can be an elegant solution for generating Morris sequence.

You could have answer that in a comment.


from itertools import groupby [(c,len(list(cs))) for c,cs in groupby('Pedro Manoel')]




name = list('Pedro Manoel') name.sort() [(c,len(list(cs))) for c,cs in groupby(name)]



Just remembering, if the list is not sorted, the groupby funcion will not work!

Actually it works. You might think this behavior as broken, but it's useful in some cases. See answers to this question for an example:
Denis Otkidach

I have same problem.

I want to group by on dict key

>>> x
[{'a': 10, 'b': 90}, {'a': 20}, {'a': 30}, {'a': 10}]
>>> [(name, list(group)) for name, group in groupby(x, lambda p:p['a'])]
[(10, [{'a': 10, 'b': 90}]), (20, [{'a': 20}]), (30, [{'a': 30}]), (10, [{'a': 10}])]

This must group on key 10 :(