{"action":"opened","issue":{"url":"https://api.github.com/repos/tschellenbach/Stream-Framework/issues/147","labels_url":"https://api.github.com/repos/tschellenbach/Stream-Framework/issues/147/labels{/name}","comments_url":"https://api.github.com/repos/tschellenbach/Stream-Framework/issues/147/comments","events_url":"https://api.github.com/repos/tschellenbach/Stream-Framework/issues/147/events","html_url":"https://github.com/tschellenbach/Stream-Framework/issues/147","id":96205290,"number":147,"title":"Fix for Pickle/Redis encoding issue (related to #57)","user":{"login":"izhan","id":4621962,"avatar_url":"https://avatars.githubusercontent.com/u/4621962?v=3","gravatar_id":"","url":"https://api.github.com/users/izhan","html_url":"https://github.com/izhan","followers_url":"https://api.github.com/users/izhan/followers","following_url":"https://api.github.com/users/izhan/following{/other_user}","gists_url":"https://api.github.com/users/izhan/gists{/gist_id}","starred_url":"https://api.github.com/users/izhan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/izhan/subscriptions","organizations_url":"https://api.github.com/users/izhan/orgs","repos_url":"https://api.github.com/users/izhan/repos","events_url":"https://api.github.com/users/izhan/events{/privacy}","received_events_url":"https://api.github.com/users/izhan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-07-21T01:30:58Z","updated_at":"2015-07-21T01:30:58Z","closed_at":null,"body":"Thanks again for creating an awesome framework.  Really loving it so far.\r\n\r\nI wanted to bring attention to a bug I (and other users...see #57) encountered with Redis and `extra_context`, and our fix for it.  I am currently using a `RedisNotificationFeed` and adding activities with `extra_context` set to arbitrary input (in this case, user-inputted comments).  The script is currently crashing with certain production data, specifically non-ASCII characters in positions 128 to 255.  For instance, if `extra_context` contains the degree symbol `°`, it throws the error.  However, `extra_context` with `狗` doesn't.\r\n\r\nHere's the specific error output:\r\n```\r\n  File \"directory/project/notifications/notification_manager.py\", line 49, in do_something\r\n    feed.add(activity)\r\n  File \"directory/venv/lib/python2.7/site-packages/stream_framework/feeds/base.py\", line 197, in add\r\n    return self.add_many([activity], *args, **kwargs)\r\n  File \"directory/venv/lib/python2.7/site-packages/stream_framework/feeds/aggregated_feed/notification_feed.py\", line 68, in add_many\r\n    self, activities, **kwargs)\r\n  File \"directory/venv/lib/python2.7/site-packages/stream_framework/feeds/aggregated_feed/base.py\", line 90, in add_many\r\n    current_activities = self[:self.merge_max_length]\r\n  File \"directory/venv/lib/python2.7/site-packages/stream_framework/feeds/base.py\", line 305, in __getitem__\r\n    start, bound)\r\n  File \"directory/venv/lib/python2.7/site-packages/stream_framework/feeds/base.py\", line 347, in get_activity_slice\r\n    ordering_args=self._ordering_args)\r\n  File \"directory/venv/lib/python2.7/site-packages/stream_framework/storage/base.py\", line 311, in get_slice\r\n    key, start, stop, filter_kwargs=filter_kwargs, ordering_args=ordering_args)\r\n  File \"directory/venv/lib/python2.7/site-packages/stream_framework/storage/redis/timeline_storage.py\", line 83, in get_slice_from_storage\r\n    key_score_pairs = cache.get_results(start, stop, **result_kwargs)\r\n  File \"directory/venv/lib/python2.7/site-packages/stream_framework/storage/redis/structures/sorted_set.py\", line 182, in get_results\r\n    key, start=start, num=limit, withscores=True, min=min_score, max=max_score)\r\n  File \"directory/venv/lib/python2.7/site-packages/redis/client.py\", line 1761, in zrevrangebyscore\r\n    return self.execute_command(*pieces, **options)\r\n  File \"directory/venv/lib/python2.7/site-packages/redis/client.py\", line 565, in execute_command\r\n    return self.parse_response(connection, command_name, **options)\r\n  File \"directory/venv/lib/python2.7/site-packages/redis/client.py\", line 577, in parse_response\r\n    response = connection.read_response()\r\n  File \"directory/venv/lib/python2.7/site-packages/redis/connection.py\", line 569, in read_response\r\n    response = self._parser.read_response()\r\n  File \"directory/venv/lib/python2.7/site-packages/redis/connection.py\", line 264, in read_response\r\n    response = [self.read_response() for i in xrange(length)]\r\n  File \"directory/venv/lib/python2.7/site-packages/redis/connection.py\", line 266, in read_response\r\n    response = response.decode(self.encoding)\r\n  File \"directory/venv/lib/python2.7/encodings/utf_8.py\", line 16, in decode\r\n    return codecs.utf_8_decode(input, errors, True)\r\nUnicodeDecodeError: 'utf8' codec can't decode byte 0xb0 in position 360: invalid start byte\r\n```\r\n\r\nNote that we are running Python 2.  The core issue is that `pickle.dumps` returns bytes that Redis tries to decode to `utf-8`.  Our current solution is disabling this decoding by changing `decode_responses` to `False` in `stream_framework/storage/redis/connection.py` i.e.\r\n```\r\npool = redis.ConnectionPool(\r\n    host=config['host'],\r\n    port=config['port'],\r\n    password=config.get('password'),\r\n    db=config['db'],\r\n    decode_responses=False\r\n)\r\n```\r\n\r\nWhat are your thoughts on a long-term solution that works for both Python 2 and 3?  I'd be happy to investigate more."}}