PyMongo aggregation script no response
<p>I have a mongodb instance filled with tweets and I am using PyMongo as a driver. Each tweet has a field that specifies its language. I have the following aggregation query:</p>

<pre><code>def get_top_k_hashtags(client, db_name, lang_list, k_filter):
  k_filter_base = k_filter
  k_filter = "$" + k_filter
  dbo = client[db_name]
  pipeline = [
      {"$match": {"lang": {"$in": lang_list}}},
      {"$project": {k_filter_base: 1, "_id": 0}},
      {"$unwind": k_filter},
      {"$group": {"_id": k_filter + ".text", "count": {"$sum": 1}}},
      {"$sort": SON([("count", -1), ("_id", -1)])}
 ]
 return dbo.tweets.aggregate(pipeline, allowDiskUse=True)
</code></pre>

<p>Which I call with the following test function</p>

<pre><code>def test_get_top_k_hashtags(client, db_name, lang_list, k_filter):
  frequency = []
  cursor = get_top_k_hashtags(
      client, db_name, lang_list, k_filter)
  for document in cursor:
      frequency.append({'hashtag': document['_id'],
                      'value': document['count']})
  pprint(frequency)
  write_json_file('hashtag_distribution', DATA_PATH, frequency)

test_get_top_k_hashtags(client, 'twitter', ['ru'], 'entities.hashtags')
</code></pre>

<p>What I am doing is getting an aggregate count for all the hashtags under a specific language, in this case 'ru'. I have around 17K tweets that match 'ru' and 13million tweets for 'en'. The aggregation works in both mongo shell and my script when I run it with 'ru' tweets, but when I try it for 'en' tweets the script hangs and never completes. </p>

<p>Running aggregation for 'en' in mongo shell completes successfully. When I run it from the script I can check the operation in mongo shell and see that it starts and completes, but I never get any output. </p>

<p>The issue is that in the case of the Python script, results are not returned when I try to aggregate a larger subset of the collection, even though it is successfully processed in the db. I have tried catching exceptions from Pymongo to no avail. I suspect that something is timing out but I can't figure out what it is or how to catch it. </p>