{"action":"created","comment":{"url":"https://api.github.com/repos/scrapy/scrapy/pulls/comments/23524786","id":23524786,"diff_hunk":"@@ -0,0 +1,298 @@\n+.. _topics-periodic-stats:\n+\n+==============\n+Periodic Stats\n+==============\n+\n+Sometimes you want to monitor statistics or be notified about certain events that occur in your crawling process.\n+Scrapy provides the tools for periodically generate and process data from a :ref:`topics-stats`.\n+\n+Some use cases can be:\n+\n+- Monitor and graph scraping data over time (memory usage, http codes, errors, scraped items, etc).\n+- Get notified when a crawl job has finished or alerted when something happens.\n+- Use your own stat values to monitor custom data or events.\n+\n+You can for instance generate and export your stats and create some time-based graphs using a 3rd party\n+tool like `Kibana`_.\n+\n+\n+To achieve this is you must configure a :ref:`topics-stats-periodicstatscollector` with a list of:\n+\n+- :ref:`topics-stats-valueobservers`: To select and configure the filtered periodic stats.\n+- :ref:`topics-stats-statspipelines` To process the generated stats.\n+\n+.. _topics-stats-periodicstatscollector:\n+\n+Periodic Stats Collector\n+========================\n+\n+:class:`PeriodicStatsCollector` is a special :class:`MemoryStatsCollector` class, it has exactly the same funcionality\n+but can be configured to periodically generate and process stats.\n+\n+All configuration is done via settings. To use it, simply change the :setting:`STATS_CLASS` setting like this::\n+\n+    STATS_CLASS = 'scrapy.contrib.periodicstats.PeriodicStatsCollector'\n+\n+\n+Stat values can be filtered defining :ref:`topics-stats-valueobservers`, but sometimes you want all the available\n+stats. You can get this with the :setting:`PERIODIC_STATS_EXPORT_ALL` setting::\n+\n+    PERIODIC_STATS_EXPORT_ALL = True\n+\n+And to define the interval for generating all the stats you must use\n+the :setting:`PERIODIC_STATS_EXPORT_ALL_INTERVAL` setting::\n+\n+    PERIODIC_STATS_EXPORT_ALL_INTERVAL = 30  # time in seconds\n+\n+\n+You can always deactivate the periodic stats generation with the :setting:`PERIODIC_STATS_ENABLED` setting\n+(:class:`MemoryStatsCollector` functionality will still remain the same)::\n+\n+    PERIODIC_STATS_ENABLED = False\n+\n+\n+\n+.. _topics-stats-valueobservers:\n+\n+Value Observers\n+===============\n+\n+:class:`PeriodicStatsObserver` class allows to select data from the available :ref:`topics-stats` and define when it\n+should be generated.\n+\n+They can be defined using the :setting:`PERIODIC_STATS_OBSERVERS` setting::\n+\n+    from scrapy.contrib import periodicstats as stats\n+\n+    PERIODIC_STATS_OBSERVERS = [\n+        stats.Observer(key='downloader/response_count'),\n+    ]\n+\n+The previous example will generate the number of responses every second::\n+\n+    {'downloader/response_count': 5}   # for t=1\n+    {'downloader/response_count': 34}  # for t=2\n+    {'downloader/response_count': 45}  # for t=3\n+    ...\n+\n+More examples of use below.\n+\n+\n+PeriodicStatsObserver\n+---------------------\n+\n+.. module:: scrapy.contrib.periodicstats\n+.. class:: PeriodicStatsObserver(key[, use_re_key=False, use_partial_values=False, export_key=None, export_interval=1, only_export_on_change=False, only_export_on_close=False])\n+\n+    Class that monitors a stats value and decides to show it or not across intervals.\n+\n+    :param key: the stats key to use\n+    :type key: string\n+\n+    :param use_re_key: whether key parameter is a regular expression or not\n+    :type use_re_key: bool\n+\n+    :param use_partial_values: defines how numeric values are calculated. If active only variations during the interval will be shown, if disabled (default) the total cumulated value will be used.\n+    :type use_partial_values: bool\n+\n+    :param export_key: the key to use for this value, if not defined ``key`` parameter will be used.\n+    :type export_key: string\n+\n+    :param export_interval: the interval in seconds that will be used to generate this value.\n+    :type export_interval: int\n+\n+    :param only_export_on_change: If active, value will be only generated if has changed since the last interval.\n+    :type only_export_on_change: bool\n+\n+    :param only_export_on_close: If active, value will be only generated when the spider is closed.\n+    :type only_export_on_close: bool\n+\n+You can define many observers::\n+\n+    PERIODIC_STATS_OBSERVERS = [\n+        stats.Observer(key='downloader/request_count'),\n+        stats.Observer(key='downloader/response_count'),\n+        stats.Observer(key='httpcache/hit'),\n+    ]\n+\n+    \"\"\"\n+    Generated stats:\n+    {'downloader/request_count': 10, 'downloader/request_count': 10, 'httpcache/hit': 0}  # for t=1\n+    {'downloader/request_count': 23, 'downloader/request_count': 20, 'httpcache/hit': 3}  # for t=2\n+    {'downloader/request_count': 23, 'downloader/request_count': 20, 'httpcache/hit': 3}  # for t=3\n+    {'downloader/request_count': 34, 'downloader/request_count': 28, 'httpcache/hit': 7}  # for t=4\n+    ...\n+    \"\"\"\n+\n+To use a different name for the stats value use the ``export_key`` parameter::\n+\n+    PERIODIC_STATS_OBSERVERS = [\n+        stats.Observer(key='downloader/request_count', export_key='requests'),\n+        stats.Observer(key='downloader/response_count', export_key='responses'),\n+        stats.Observer(key='httpcache/hit', export_key='from_cache'),\n+    ]\n+\n+    \"\"\"\n+    Generated stats:\n+    {'requests': 10, 'responses': 10, 'from_cache': 0}   # for t=1\n+    {'requests': 23, 'responses': 20, 'from_cache': 3}   # for t=2\n+    {'requests': 23, 'responses': 20, 'from_cache': 3}   # for t=3\n+    {'requests': 34, 'responses': 28, 'from_cache': 7}   # for t=4\n+    ...\n+    \"\"\"\n+\n+You can use several observers at the same key, but then ``export_key`` can't be repeated::\n+\n+    PERIODIC_STATS_OBSERVERS = [\n+        stats.Observer(key='downloader/request_count', export_key='requests_counter'),\n+        stats.Observer(key='downloader/request_count', export_key='another_request_counter'),\n+    ]\n+\n+    \"\"\"\n+    Generated stats:\n+    {'requests_counter': 10, 'another_request_counter': 10}  # for t=1\n+    {'requests_counter': 23, 'another_request_counter': 23}  # for t=2\n+    {'requests_counter': 23, 'another_request_counter': 23}  # for t=3\n+    {'requests_counter': 34, 'another_request_counter': 34}  # for t=4\n+    ...\n+    \"\"\"\n+\n+By default values are accumulated, but we can also use just differences between intervals with the\n+``use_partial_values`` parameter::\n+\n+    PERIODIC_STATS_OBSERVERS = [\n+        stats.Observer(key='downloader/request_count', export_key='requests', use_partial_values=True),\n+    ]\n+\n+    \"\"\"\n+    Generated stats:\n+    {'requests': 10}  # for t=1\n+    {'requests': 13}  # for t=2\n+    {'requests': 0}   # for t=3\n+    {'requests': 11}  # for t=4\n+    ...\n+    \"\"\"\n+\n+To minimize generated data we can choose to export data only when it changes with the\n+``only_export_on_change`` parameter::\n+\n+    PERIODIC_STATS_OBSERVERS = [\n+        stats.Observer(key='downloader/request_count', export_key='requests', only_export_on_change=True),\n+    ]\n+\n+    \"\"\"\n+    Generated stats:\n+    {'requests': 10}  # for t=1\n+    {'requests': 13}  # for t=2\n+    {}                # for t=3\n+    {'requests': 11}  # for t=4\n+    ...\n+    \"\"\"\n+\n+We can define export intervals per key with the ``export_interval`` parameter::\n+\n+    PERIODIC_STATS_OBSERVERS = [\n+        stats.Observer(key='downloader/request_count', export_key='requests', export_interval=1),\n+        stats.Observer(key='downloader/response_count', export_key='responses', export_interval=2),\n+    ]\n+\n+    \"\"\"\n+    Generated stats:\n+    {'requests': 10, 'responses': 10}  # for t=1\n+    {'requests': 23}                   # for t=2\n+    {'requests': 23, 'responses': 20}  # for t=3\n+    {'requests': 34}                   # for t=4\n+    ...\n+    \"\"\"\n+\n+To export only stats when a spider is finished use the ``only_export_on_close`` parameter::\n+\n+    PERIODIC_STATS_OBSERVERS = [\n+        stats.Observer(key='downloader/request_count', export_key='requests', only_export_on_close=True),\n+    ]\n+\n+    \"\"\"\n+    Generated stats:\n+    {}                  # for t=1\n+    {}                  # for t=2\n+    {}                  # for t=3\n+    {}                  # for t=4\n+    ...\n+    {'requests': 2759}  # on spider_close\n+    \"\"\"\n+\n+Regular expressions can be used to group/filter data with the ``use_re_key`` parameter:::\n+\n+    PERIODIC_STATS_OBSERVERS = [\n+        stats.Observer(key='downloader/response_status_count/2..', use_re_key=True, export_key='2xx'),\n+        stats.Observer(key='downloader/response_status_count/3..', use_re_key=True, export_key='3xx'),\n+        stats.Observer(key='downloader/response_status_count/4..', use_re_key=True, export_key='4xx'),\n+        stats.Observer(key='downloader/response_status_count/5..', use_re_key=True, export_key='5xx'),\n+    ]\n+\n+    \"\"\"\n+    Generated stats:\n+    {'2xx': 10}                               # for t=1\n+    {'2xx': 34, '3xx': 1, '4xx':1}            # for t=2\n+    {'2xx': 67, '3xx': 4, '4xx':2, '5xx': 1}  # for t=3\n+    {'2xx': 90, '3xx': 8, '4xx':2, '5xx': 1}  # for t=4\n+    ...\n+    \"\"\"\n+\n+.. _topics-stats-statspipelines:\n+\n+Stats Pipelines\n+===============\n+\n+:class:`PeriodicStatsPipeline` objects receive and process the stats generated by the :class:`PeriodicStatsObserver`\n+objects for every :ref:`topics-stats-periodicstatscollector` iteration.\n+\n+You can define the pipeline objects to use with the :setting:`PERIODIC_STATS_PIPELINES` setting::\n+\n+    PERIODIC_STATS_PIPELINES = [\n+        'scrapy.contrib.periodicstats.PeriodicStatsLogger',\n+    ]\n+\n+\n+PeriodicStatsPipeline\n+---------------------\n+\n+.. module:: scrapy.contrib.periodicstats\n+.. class:: PeriodicStatsPipeline\n+\n+    .. method:: open_spider(spider)\n+\n+        Called when the spider is opened.\n+\n+        :param spider: the spider for which the stats are processed.\n+        :type spider: :class:`~scrapy.spider.Spider` object\n+\n+    .. method:: close_spider(spider)\n+\n+        Called when the spider is closed.\n+\n+        :param spider: the spider for which the stats are processed.\n+        :type spider: :class:`~scrapy.spider.Spider` object\n+\n+    .. method:: process_stats(spider, interval, stats)\n+\n+        Called for every :class:`PeriodicStatsCollector` iteration with the stats to be processed.\n+\n+        :meth:`process_stats` should return the passed stats or ``None``.\n+\n+        If it returns the passed stats, they will be passed to the next middleware.","path":"docs/topics/periodic-stats.rst","position":284,"original_position":284,"commit_id":"7ffd7e47324272000ffe866447fdd16454eff3d9","original_commit_id":"7ffd7e47324272000ffe866447fdd16454eff3d9","user":{"login":"kmike","id":107893,"avatar_url":"https://avatars.githubusercontent.com/u/107893?v=3","gravatar_id":"","url":"https://api.github.com/users/kmike","html_url":"https://github.com/kmike","followers_url":"https://api.github.com/users/kmike/followers","following_url":"https://api.github.com/users/kmike/following{/other_user}","gists_url":"https://api.github.com/users/kmike/gists{/gist_id}","starred_url":"https://api.github.com/users/kmike/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmike/subscriptions","organizations_url":"https://api.github.com/users/kmike/orgs","repos_url":"https://api.github.com/users/kmike/repos","events_url":"https://api.github.com/users/kmike/events{/privacy}","received_events_url":"https://api.github.com/users/kmike/received_events","type":"User","site_admin":false},"body":"middleware -> stats pipeline?","created_at":"2015-01-26T11:44:15Z","updated_at":"2015-01-26T11:44:15Z","html_url":"https://github.com/scrapy/scrapy/pull/1014#discussion_r23524786","pull_request_url":"https://api.github.com/repos/scrapy/scrapy/pulls/1014","_links":{"self":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/comments/23524786"},"html":{"href":"https://github.com/scrapy/scrapy/pull/1014#discussion_r23524786"},"pull_request":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/1014"}}},"pull_request":{"url":"https://api.github.com/repos/scrapy/scrapy/pulls/1014","id":27497523,"html_url":"https://github.com/scrapy/scrapy/pull/1014","diff_url":"https://github.com/scrapy/scrapy/pull/1014.diff","patch_url":"https://github.com/scrapy/scrapy/pull/1014.patch","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/1014","number":1014,"state":"open","locked":false,"title":"Added periodic stats. Allows periodical stats generation and processing.","user":{"login":"gatufo","id":8648810,"avatar_url":"https://avatars.githubusercontent.com/u/8648810?v=3","gravatar_id":"","url":"https://api.github.com/users/gatufo","html_url":"https://github.com/gatufo","followers_url":"https://api.github.com/users/gatufo/followers","following_url":"https://api.github.com/users/gatufo/following{/other_user}","gists_url":"https://api.github.com/users/gatufo/gists{/gist_id}","starred_url":"https://api.github.com/users/gatufo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatufo/subscriptions","organizations_url":"https://api.github.com/users/gatufo/orgs","repos_url":"https://api.github.com/users/gatufo/repos","events_url":"https://api.github.com/users/gatufo/events{/privacy}","received_events_url":"https://api.github.com/users/gatufo/received_events","type":"User","site_admin":false},"body":"Added support to periodically selecting, exporting/processing scrapy stats.\r\n\r\nSome use cases can be:\r\n* Monitor and graph scraping data over time (memory usage, http codes, errors, scraped items, etc).\r\n* Get notified when a crawl job has finished or alerted when something happens.\r\n* Use your own stat values to monitor custom data or events.\r\n\r\nTime-based graphs can be generated using a 3rd party tool like [Kibana](http://www.elasticsearch.org/overview/kibana/).\r\n\r\nJob is done by a ``StatCollector`` class called ``PeriodicStatsCollector``. It can be configurated with:\r\n* ``Value Observers``: To select and configure the filtered periodic stats.\r\n* ``Stats Pipelines``: To process the generated stats.\r\n\r\nAll configuration is done via settings, an example that will log responses count every 30 seconds:\r\n\r\n```python\r\nfrom scrapy.contrib import periodicstats as stats\r\n\r\nSTATS_CLASS = 'scrapy.contrib.periodicstats.PeriodicStatsCollector'\r\nPERIODIC_STATS_OBSERVERS = [\r\n    stats.Observer(key='downloader/response_count', export_interval=30),\r\n]\r\nPERIODIC_STATS_PIPELINES = [\r\n    'scrapy.contrib.periodicstats.PeriodicStatsLogger',\r\n]\r\n```\r\nThat will produce the following output:\r\n```\r\n2015-01-16 11:29:10+0100 [scrapy] INFO: Dumping Scrapy periodic stats:\r\n\t{'interval': 1,\r\n\t 'spider': 'example',\r\n\t 'stats': {'downloader/response_count': 51}}\r\n2015-01-16 11:29:40+0100 [scrapy] INFO: Dumping Scrapy periodic stats:\r\n\t{'interval': 2,\r\n\t 'spider': 'example',\r\n\t 'stats': {'downloader/response_count': 489}}\r\n...\r\n```\r\n\r\n## Value Observers\r\n\r\nThey allow to select data from the available Stats Collection and define when it should be generated. \r\n\r\nSome examples...\r\n\r\nYou can define many observers:\r\n```python\r\nPERIODIC_STATS_OBSERVERS = [\r\n    stats.Observer(key='downloader/request_count'),\r\n    stats.Observer(key='downloader/response_count'),\r\n    stats.Observer(key='httpcache/hit'),\r\n]\r\n\r\n\"\"\"\r\nGenerated stats:\r\n{'downloader/request_count': 10, 'downloader/request_count': 10, 'httpcache/hit': 0}  # for t=1\r\n{'downloader/request_count': 23, 'downloader/request_count': 20, 'httpcache/hit': 3}  # for t=2\r\n{'downloader/request_count': 23, 'downloader/request_count': 20, 'httpcache/hit': 3}  # for t=3\r\n{'downloader/request_count': 34, 'downloader/request_count': 28, 'httpcache/hit': 7}  # for t=4\r\n...\r\n\"\"\"\r\n```\r\n\r\nTo use a different name for the stats value use the ``export_key`` parameter:\r\n\r\n```python\r\nPERIODIC_STATS_OBSERVERS = [\r\n    stats.Observer(key='downloader/request_count', export_key='requests'),\r\n    stats.Observer(key='downloader/response_count', export_key='responses'),\r\n    stats.Observer(key='httpcache/hit', export_key='from_cache'),\r\n]\r\n\r\n\"\"\"\r\nGenerated stats:\r\n{'requests': 10, 'responses': 10, 'from_cache': 0}   # for t=1\r\n{'requests': 23, 'responses': 20, 'from_cache': 3}   # for t=2\r\n{'requests': 23, 'responses': 20, 'from_cache': 3}   # for t=3\r\n{'requests': 34, 'responses': 28, 'from_cache': 7}   # for t=4\r\n...\r\n\"\"\"\r\n```\r\n\r\nYou can use several observers at the same key, but then ``export_key`` can’t be repeated:\r\n```python\r\nPERIODIC_STATS_OBSERVERS = [\r\n    stats.Observer(key='downloader/request_count', export_key='requests_counter'),\r\n    stats.Observer(key='downloader/request_count', export_key='another_request_counter'),\r\n]\r\n\r\n\"\"\"\r\nGenerated stats:\r\n{'requests_counter': 10, 'another_request_counter': 10}  # for t=1\r\n{'requests_counter': 23, 'another_request_counter': 23}  # for t=2\r\n{'requests_counter': 23, 'another_request_counter': 23}  # for t=3\r\n{'requests_counter': 34, 'another_request_counter': 34}  # for t=4\r\n...\r\n\"\"\"\r\n```\r\n\r\nBy default values are accumulated, but we can also use just differences between intervals with the ``use_partial_values`` parameter:\r\n\r\n```python\r\nPERIODIC_STATS_OBSERVERS = [\r\n    stats.Observer(key='downloader/request_count', export_key='requests', use_partial_values=True),\r\n]\r\n\r\n\"\"\"\r\nGenerated stats:\r\n{'requests': 10}  # for t=1\r\n{'requests': 13}  # for t=2\r\n{'requests': 0}   # for t=3\r\n{'requests': 11}  # for t=4\r\n...\r\n\"\"\"\r\n```\r\n\r\nTo minimize generated data we can choose to export data only when it changes with the ``only_export_on_change`` parameter:\r\n\r\n```python\r\nPERIODIC_STATS_OBSERVERS = [\r\n    stats.Observer(key='downloader/request_count', export_key='requests', only_export_on_change=True),\r\n]\r\n\r\n\"\"\"\r\nGenerated stats:\r\n{'requests': 10}  # for t=1\r\n{'requests': 13}  # for t=2\r\n{}                # for t=3\r\n{'requests': 11}  # for t=4\r\n...\r\n\"\"\"\r\n```\r\n\r\nWe can define export intervals per key with the ``export_interval`` parameter:\r\n\r\n```python\r\nPERIODIC_STATS_OBSERVERS = [\r\n    stats.Observer(key='downloader/request_count', export_key='requests', export_interval=1),\r\n    stats.Observer(key='downloader/response_count', export_key='responses', export_interval=2),\r\n]\r\n\r\n\"\"\"\r\nGenerated stats:\r\n{'requests': 10, 'responses': 10}  # for t=1\r\n{'requests': 23}                   # for t=2\r\n{'requests': 23, 'responses': 20}  # for t=3\r\n{'requests': 34}                   # for t=4\r\n...\r\n\"\"\"\r\n```\r\n\r\nTo export only stats when a spider is finished use the ``only_export_on_close`` parameter:\r\n\r\n```python\r\nPERIODIC_STATS_OBSERVERS = [\r\n    stats.Observer(key='downloader/request_count', export_key='requests', only_export_on_close=True),\r\n]\r\n\r\n\"\"\"\r\nGenerated stats:\r\n{}                  # for t=1\r\n{}                  # for t=2\r\n{}                  # for t=3\r\n{}                  # for t=4\r\n...\r\n{'requests': 2759}  # on spider_close\r\n\"\"\"\r\n```\r\n\r\nRegular expressions can be used to group/filter data with the ``use_re_key`` parameter::\r\n\r\n```python\r\nPERIODIC_STATS_OBSERVERS = [\r\n    stats.Observer(key='downloader/response_status_count/2..', use_re_key=True, export_key='2xx'),\r\n    stats.Observer(key='downloader/response_status_count/3..', use_re_key=True, export_key='3xx'),\r\n    stats.Observer(key='downloader/response_status_count/4..', use_re_key=True, export_key='4xx'),\r\n    stats.Observer(key='downloader/response_status_count/5..', use_re_key=True, export_key='5xx'),\r\n]\r\n\r\n\"\"\"\r\nGenerated stats:\r\n{'2xx': 10}                               # for t=1\r\n{'2xx': 34, '3xx': 1, '4xx':1}            # for t=2\r\n{'2xx': 67, '3xx': 4, '4xx':2, '5xx': 1}  # for t=3\r\n{'2xx': 90, '3xx': 8, '4xx':2, '5xx': 1}  # for t=4\r\n...\r\n\"\"\"\r\n```\r\n\r\n## Stats Pipelines\r\nBasically work like item pipelines, they have a ``process_stats`` method where they will receive the generated stats from the observers for every interval.\r\n\r\n```python\r\nclass PeriodicStatsPipeline(object):\r\n    \"\"\"\r\n    Base class for periodic stats pipeline\r\n    \"\"\"\r\n    def __init__(self, crawler):\r\n        pass\r\n\r\n    @classmethod\r\n    def from_crawler(cls, crawler):\r\n        return cls(crawler)\r\n\r\n    def open_spider(self, spider):\r\n        pass\r\n\r\n    def close_spider(self, spider, reason):\r\n        pass\r\n\r\n    def process_stats(self, spider, period, stats):\r\n        raise NotImplementedError\r\n\r\n\r\nclass PeriodicStatsLogger(PeriodicStatsPipeline):\r\n    \"\"\"\r\n    Stats processor that print stats debug info\r\n    \"\"\"\r\n    def process_stats(self, spider, interval, stats):\r\n        log.msg('Dumping Scrapy periodic stats:\\n' + pprint.pformat({\r\n            'spider': spider.name,\r\n            'interval': interval,\r\n            'stats': stats,\r\n        }))\r\n        return stats\r\n```\r\n\r\nNot implemented yet, but some uses may be exporting to ELK through [redis](http://redis.io/)>[logstash](http://logstash.net/) or exporting stats to [hubstorage](https://github.com/scrapinghub/hubstorage) using its API.\r\n\r\n\r\n\r\n\r\n","created_at":"2015-01-16T10:52:10Z","updated_at":"2015-01-26T11:44:15Z","closed_at":null,"merged_at":null,"merge_commit_sha":"b42dd02b2cf54c4d467d5a680ab7fbdfb51b5bc9","assignee":null,"milestone":{"url":"https://api.github.com/repos/scrapy/scrapy/milestones/6","labels_url":"https://api.github.com/repos/scrapy/scrapy/milestones/6/labels","id":914091,"number":6,"title":"Scrapy 1.0","description":"","creator":{"login":"kmike","id":107893,"avatar_url":"https://avatars.githubusercontent.com/u/107893?v=3","gravatar_id":"","url":"https://api.github.com/users/kmike","html_url":"https://github.com/kmike","followers_url":"https://api.github.com/users/kmike/followers","following_url":"https://api.github.com/users/kmike/following{/other_user}","gists_url":"https://api.github.com/users/kmike/gists{/gist_id}","starred_url":"https://api.github.com/users/kmike/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kmike/subscriptions","organizations_url":"https://api.github.com/users/kmike/orgs","repos_url":"https://api.github.com/users/kmike/repos","events_url":"https://api.github.com/users/kmike/events{/privacy}","received_events_url":"https://api.github.com/users/kmike/received_events","type":"User","site_admin":false},"open_issues":7,"closed_issues":0,"state":"open","created_at":"2014-12-24T09:57:08Z","updated_at":"2015-01-23T15:39:08Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/scrapy/scrapy/pulls/1014/commits","review_comments_url":"https://api.github.com/repos/scrapy/scrapy/pulls/1014/comments","review_comment_url":"https://api.github.com/repos/scrapy/scrapy/pulls/comments/{number}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/1014/comments","statuses_url":"https://api.github.com/repos/scrapy/scrapy/statuses/7ffd7e47324272000ffe866447fdd16454eff3d9","head":{"label":"gatufo:periodic-stats","ref":"periodic-stats","sha":"7ffd7e47324272000ffe866447fdd16454eff3d9","user":{"login":"gatufo","id":8648810,"avatar_url":"https://avatars.githubusercontent.com/u/8648810?v=3","gravatar_id":"","url":"https://api.github.com/users/gatufo","html_url":"https://github.com/gatufo","followers_url":"https://api.github.com/users/gatufo/followers","following_url":"https://api.github.com/users/gatufo/following{/other_user}","gists_url":"https://api.github.com/users/gatufo/gists{/gist_id}","starred_url":"https://api.github.com/users/gatufo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatufo/subscriptions","organizations_url":"https://api.github.com/users/gatufo/orgs","repos_url":"https://api.github.com/users/gatufo/repos","events_url":"https://api.github.com/users/gatufo/events{/privacy}","received_events_url":"https://api.github.com/users/gatufo/received_events","type":"User","site_admin":false},"repo":{"id":28043879,"name":"scrapy","full_name":"gatufo/scrapy","owner":{"login":"gatufo","id":8648810,"avatar_url":"https://avatars.githubusercontent.com/u/8648810?v=3","gravatar_id":"","url":"https://api.github.com/users/gatufo","html_url":"https://github.com/gatufo","followers_url":"https://api.github.com/users/gatufo/followers","following_url":"https://api.github.com/users/gatufo/following{/other_user}","gists_url":"https://api.github.com/users/gatufo/gists{/gist_id}","starred_url":"https://api.github.com/users/gatufo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gatufo/subscriptions","organizations_url":"https://api.github.com/users/gatufo/orgs","repos_url":"https://api.github.com/users/gatufo/repos","events_url":"https://api.github.com/users/gatufo/events{/privacy}","received_events_url":"https://api.github.com/users/gatufo/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/gatufo/scrapy","description":"Scrapy, a fast high-level screen scraping and web crawling framework for Python.","fork":true,"url":"https://api.github.com/repos/gatufo/scrapy","forks_url":"https://api.github.com/repos/gatufo/scrapy/forks","keys_url":"https://api.github.com/repos/gatufo/scrapy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/gatufo/scrapy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/gatufo/scrapy/teams","hooks_url":"https://api.github.com/repos/gatufo/scrapy/hooks","issue_events_url":"https://api.github.com/repos/gatufo/scrapy/issues/events{/number}","events_url":"https://api.github.com/repos/gatufo/scrapy/events","assignees_url":"https://api.github.com/repos/gatufo/scrapy/assignees{/user}","branches_url":"https://api.github.com/repos/gatufo/scrapy/branches{/branch}","tags_url":"https://api.github.com/repos/gatufo/scrapy/tags","blobs_url":"https://api.github.com/repos/gatufo/scrapy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/gatufo/scrapy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/gatufo/scrapy/git/refs{/sha}","trees_url":"https://api.github.com/repos/gatufo/scrapy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/gatufo/scrapy/statuses/{sha}","languages_url":"https://api.github.com/repos/gatufo/scrapy/languages","stargazers_url":"https://api.github.com/repos/gatufo/scrapy/stargazers","contributors_url":"https://api.github.com/repos/gatufo/scrapy/contributors","subscribers_url":"https://api.github.com/repos/gatufo/scrapy/subscribers","subscription_url":"https://api.github.com/repos/gatufo/scrapy/subscription","commits_url":"https://api.github.com/repos/gatufo/scrapy/commits{/sha}","git_commits_url":"https://api.github.com/repos/gatufo/scrapy/git/commits{/sha}","comments_url":"https://api.github.com/repos/gatufo/scrapy/comments{/number}","issue_comment_url":"https://api.github.com/repos/gatufo/scrapy/issues/comments/{number}","contents_url":"https://api.github.com/repos/gatufo/scrapy/contents/{+path}","compare_url":"https://api.github.com/repos/gatufo/scrapy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/gatufo/scrapy/merges","archive_url":"https://api.github.com/repos/gatufo/scrapy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/gatufo/scrapy/downloads","issues_url":"https://api.github.com/repos/gatufo/scrapy/issues{/number}","pulls_url":"https://api.github.com/repos/gatufo/scrapy/pulls{/number}","milestones_url":"https://api.github.com/repos/gatufo/scrapy/milestones{/number}","notifications_url":"https://api.github.com/repos/gatufo/scrapy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/gatufo/scrapy/labels{/name}","releases_url":"https://api.github.com/repos/gatufo/scrapy/releases{/id}","created_at":"2014-12-15T15:55:34Z","updated_at":"2014-12-15T15:55:36Z","pushed_at":"2015-01-20T18:54:25Z","git_url":"git://github.com/gatufo/scrapy.git","ssh_url":"git@github.com:gatufo/scrapy.git","clone_url":"https://github.com/gatufo/scrapy.git","svn_url":"https://github.com/gatufo/scrapy","homepage":"http://scrapy.org","size":13538,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":false,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"scrapy:master","ref":"master","sha":"2183984832d332f63357c272f67be2cb48b1d418","user":{"login":"scrapy","id":733635,"avatar_url":"https://avatars.githubusercontent.com/u/733635?v=3","gravatar_id":"","url":"https://api.github.com/users/scrapy","html_url":"https://github.com/scrapy","followers_url":"https://api.github.com/users/scrapy/followers","following_url":"https://api.github.com/users/scrapy/following{/other_user}","gists_url":"https://api.github.com/users/scrapy/gists{/gist_id}","starred_url":"https://api.github.com/users/scrapy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scrapy/subscriptions","organizations_url":"https://api.github.com/users/scrapy/orgs","repos_url":"https://api.github.com/users/scrapy/repos","events_url":"https://api.github.com/users/scrapy/events{/privacy}","received_events_url":"https://api.github.com/users/scrapy/received_events","type":"Organization","site_admin":false},"repo":{"id":529502,"name":"scrapy","full_name":"scrapy/scrapy","owner":{"login":"scrapy","id":733635,"avatar_url":"https://avatars.githubusercontent.com/u/733635?v=3","gravatar_id":"","url":"https://api.github.com/users/scrapy","html_url":"https://github.com/scrapy","followers_url":"https://api.github.com/users/scrapy/followers","following_url":"https://api.github.com/users/scrapy/following{/other_user}","gists_url":"https://api.github.com/users/scrapy/gists{/gist_id}","starred_url":"https://api.github.com/users/scrapy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scrapy/subscriptions","organizations_url":"https://api.github.com/users/scrapy/orgs","repos_url":"https://api.github.com/users/scrapy/repos","events_url":"https://api.github.com/users/scrapy/events{/privacy}","received_events_url":"https://api.github.com/users/scrapy/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/scrapy/scrapy","description":"Scrapy, a fast high-level screen scraping and web crawling framework for Python.","fork":false,"url":"https://api.github.com/repos/scrapy/scrapy","forks_url":"https://api.github.com/repos/scrapy/scrapy/forks","keys_url":"https://api.github.com/repos/scrapy/scrapy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/scrapy/scrapy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/scrapy/scrapy/teams","hooks_url":"https://api.github.com/repos/scrapy/scrapy/hooks","issue_events_url":"https://api.github.com/repos/scrapy/scrapy/issues/events{/number}","events_url":"https://api.github.com/repos/scrapy/scrapy/events","assignees_url":"https://api.github.com/repos/scrapy/scrapy/assignees{/user}","branches_url":"https://api.github.com/repos/scrapy/scrapy/branches{/branch}","tags_url":"https://api.github.com/repos/scrapy/scrapy/tags","blobs_url":"https://api.github.com/repos/scrapy/scrapy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/scrapy/scrapy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/scrapy/scrapy/git/refs{/sha}","trees_url":"https://api.github.com/repos/scrapy/scrapy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/scrapy/scrapy/statuses/{sha}","languages_url":"https://api.github.com/repos/scrapy/scrapy/languages","stargazers_url":"https://api.github.com/repos/scrapy/scrapy/stargazers","contributors_url":"https://api.github.com/repos/scrapy/scrapy/contributors","subscribers_url":"https://api.github.com/repos/scrapy/scrapy/subscribers","subscription_url":"https://api.github.com/repos/scrapy/scrapy/subscription","commits_url":"https://api.github.com/repos/scrapy/scrapy/commits{/sha}","git_commits_url":"https://api.github.com/repos/scrapy/scrapy/git/commits{/sha}","comments_url":"https://api.github.com/repos/scrapy/scrapy/comments{/number}","issue_comment_url":"https://api.github.com/repos/scrapy/scrapy/issues/comments/{number}","contents_url":"https://api.github.com/repos/scrapy/scrapy/contents/{+path}","compare_url":"https://api.github.com/repos/scrapy/scrapy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/scrapy/scrapy/merges","archive_url":"https://api.github.com/repos/scrapy/scrapy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/scrapy/scrapy/downloads","issues_url":"https://api.github.com/repos/scrapy/scrapy/issues{/number}","pulls_url":"https://api.github.com/repos/scrapy/scrapy/pulls{/number}","milestones_url":"https://api.github.com/repos/scrapy/scrapy/milestones{/number}","notifications_url":"https://api.github.com/repos/scrapy/scrapy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/scrapy/scrapy/labels{/name}","releases_url":"https://api.github.com/repos/scrapy/scrapy/releases{/id}","created_at":"2010-02-22T02:01:14Z","updated_at":"2015-01-26T08:47:45Z","pushed_at":"2015-01-22T17:58:13Z","git_url":"git://github.com/scrapy/scrapy.git","ssh_url":"git@github.com:scrapy/scrapy.git","clone_url":"https://github.com/scrapy/scrapy.git","svn_url":"https://github.com/scrapy/scrapy","homepage":"http://scrapy.org","size":22343,"stargazers_count":7213,"watchers_count":7213,"language":"Python","has_issues":true,"has_downloads":false,"has_wiki":true,"has_pages":false,"forks_count":2018,"mirror_url":null,"open_issues_count":209,"forks":2018,"open_issues":209,"watchers":7213,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/1014"},"html":{"href":"https://github.com/scrapy/scrapy/pull/1014"},"issue":{"href":"https://api.github.com/repos/scrapy/scrapy/issues/1014"},"comments":{"href":"https://api.github.com/repos/scrapy/scrapy/issues/1014/comments"},"review_comments":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/1014/comments"},"review_comment":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/scrapy/scrapy/pulls/1014/commits"},"statuses":{"href":"https://api.github.com/repos/scrapy/scrapy/statuses/7ffd7e47324272000ffe866447fdd16454eff3d9"}}}}