{"action":"created","issue":{"url":"https://api.github.com/repos/feedbin/refresher/issues/4","labels_url":"https://api.github.com/repos/feedbin/refresher/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/feedbin/refresher/issues/4/comments","events_url":"https://api.github.com/repos/feedbin/refresher/issues/4/events","html_url":"https://github.com/feedbin/refresher/issues/4","id":77865737,"number":4,"title":"Refresher stuck, worker fails","user":{"login":"enkitosh","id":1476867,"avatar_url":"https://avatars.githubusercontent.com/u/1476867?v=3","gravatar_id":"","url":"https://api.github.com/users/enkitosh","html_url":"https://github.com/enkitosh","followers_url":"https://api.github.com/users/enkitosh/followers","following_url":"https://api.github.com/users/enkitosh/following{/other_user}","gists_url":"https://api.github.com/users/enkitosh/gists{/gist_id}","starred_url":"https://api.github.com/users/enkitosh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enkitosh/subscriptions","organizations_url":"https://api.github.com/users/enkitosh/orgs","repos_url":"https://api.github.com/users/enkitosh/repos","events_url":"https://api.github.com/users/enkitosh/events{/privacy}","received_events_url":"https://api.github.com/users/enkitosh/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-05-19T00:21:28Z","updated_at":"2015-09-22T01:18:21Z","closed_at":"2015-09-22T01:18:21Z","body":"I'm having a weird issue where the refresher seems to fail to parse a particular feed and that leads to errors in sidekiq workers where the error message looks like thread creation failed.\r\n\r\nDebug:\r\n\r\n18:22:35 worker.1 | 2015-05-18T23:22:35.882Z 1648 TID-ovz5w9b5s WARN: can't dup NilClass\r\n18:22:35 worker.1 | 2015-05-18T23:22:35.882Z 1648 TID-ovz5w9b5s WARN: /var/www/feedbin/refresher/app/models/feed_fetcher.rb:269:in `dup'\r\n18:22:35 worker.1 | /var/www/feedbin/refresher/app/models/feed_fetcher.rb:269:in `build_public_id'\r\n18:22:35 worker.1 | /var/www/feedbin/refresher/app/models/feed_fetcher.rb:236:in `block in normalize'\r\n18:22:35 worker.1 | /var/www/feedbin/refresher/app/models/feed_fetcher.rb:204:in `map'\r\n18:22:35 worker.1 | /var/www/feedbin/refresher/app/models/feed_fetcher.rb:204:in `normalize'\r\n18:22:35 worker.1 | /var/www/feedbin/refresher/app/models/feed_fetcher.rb:169:in `parse'\r\n18:22:35 worker.1 | /var/www/feedbin/refresher/app/workers/feed_refresher_fetcher.rb:12:in `perform'\r\n18:22:35 worker.1 | /var/www/feedbin/refresher/app/workers/feed_refresher_fetcher_critical.rb:7:in `perform'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:75:in `execute_job'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:52:in `block (2 levels) in process'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:127:in `call'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:127:in `block in invoke'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/middleware/server/retry_jobs.rb:74:in `call'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/middleware/server/logging.rb:11:in `block in call'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/logging.rb:24:in `with_context'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/middleware/server/logging.rb:7:in `call'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:129:in `block in invoke'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:132:in `call'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/middleware/chain.rb:132:in `invoke'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:51:in `block in process'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:98:in `stats'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/sidekiq-3.3.2/lib/sidekiq/processor.rb:50:in `process'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `public_send'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/celluloid-0.16.0/lib/celluloid/calls.rb:26:in `dispatch'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/celluloid-0.16.0/lib/celluloid/calls.rb:122:in `dispatch'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/celluloid-0.16.0/lib/celluloid/cell.rb:60:in `block in invoke'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/celluloid-0.16.0/lib/celluloid/cell.rb:71:in `block in task'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/celluloid-0.16.0/lib/celluloid/actor.rb:357:in `block in task'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/celluloid-0.16.0/lib/celluloid/tasks.rb:57:in `block in initialize'\r\n18:22:35 worker.1 | /home/user/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/celluloid-0.16.0/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'\r\n\r\nThis usually happens with feeds from reuter.com, example: http://feeds.reuters.com/reuters/businessNews/\r\n\r\nI'm trying to reproduce this using the feed_fetcher.rb like it is called from the sidekiq worker.\r\nSo using pry I do this. Note: I pick out the xml_string of the feed failing but It is quite long so I will not post it here.\r\n\r\n    body = xml_string\r\n    feed_url = \"http://feeds.reuters.com/reuters/topNews\"\r\n    feed_fetcher = FeedFetcher.new(feed_url)\r\n    feed_fetcher.parse(body, feed_url)\r\n\r\nThis gives me:\r\nNoMethodError: undefined method `strip' for nil:NilClass\r\nfrom /home/user/Code/misc/refresher/app/models/feed_fetcher.rb:177:in `normalize'\r\n\r\nOk so normalize is going through the feedjira object when this fails.\r\n\r\n    feedjira = Feedjira::Feed.parse(body)\r\n\r\nIn line 177 it is failing on \r\n\r\n      feedjira.feed_url      = feedjira.feed_url.strip\r\n\r\nshell:\r\n\r\n    pry> feedjira.feed_url\r\n      => nil\r\n\r\nWhich could be causing the issue. However I'm not sure that explains why my workers just keep being stuck. With Sidekiq my usual experience is that if a worker fails it is either disregarded or retried.\r\n\r\nSetup\r\n--------\r\n\r\nWhen running refresher standalone I'm using the latest git pull from this repo and\r\nredis running on docker, port: 6379\r\n\r\nThe trace posted at the top of this post happens when using refresher with a fork of feedbin. In this setup I run \r\n\r\nelasticsearch (0.90.7)\r\nredis (3.0)\r\nmemcached (1.4.24)\r\n\r\nI run all these services on docker which hasn't been a problem so far.\r\nI've tried moving refresher to a seperate server running on 4GB ubuntu 14.04 since I thought at first that this might be related to threads/allocation limits but it leads to the same errors.\r\n\r\nWith both feedbin and refresher I start them using the default Procfile\r\nbundle exec foreman start\r\nrails server\r\n\r\nI have been shifting the project around a bit lately so this could be a problem on my side, how services are being run. I wonder if this is Sidekiq/celluloid related.  I just wanted to ask and see if you or anyone else has experienced similar problems with particular feeds and refresher getting stuck like this?"},"comment":{"url":"https://api.github.com/repos/feedbin/refresher/issues/comments/142154335","html_url":"https://github.com/feedbin/refresher/issues/4#issuecomment-142154335","issue_url":"https://api.github.com/repos/feedbin/refresher/issues/4","id":142154335,"user":{"login":"enkitosh","id":1476867,"avatar_url":"https://avatars.githubusercontent.com/u/1476867?v=3","gravatar_id":"","url":"https://api.github.com/users/enkitosh","html_url":"https://github.com/enkitosh","followers_url":"https://api.github.com/users/enkitosh/followers","following_url":"https://api.github.com/users/enkitosh/following{/other_user}","gists_url":"https://api.github.com/users/enkitosh/gists{/gist_id}","starred_url":"https://api.github.com/users/enkitosh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/enkitosh/subscriptions","organizations_url":"https://api.github.com/users/enkitosh/orgs","repos_url":"https://api.github.com/users/enkitosh/repos","events_url":"https://api.github.com/users/enkitosh/events{/privacy}","received_events_url":"https://api.github.com/users/enkitosh/received_events","type":"User","site_admin":false},"created_at":"2015-09-22T01:18:21Z","updated_at":"2015-09-22T01:18:21Z","body":"Turned out this was just a problem with a lot of unused middleware in my case. Closing this"}}