{"action":"created","issue":{"url":"https://api.github.com/repos/jaimeiniesta/metainspector/issues/127","labels_url":"https://api.github.com/repos/jaimeiniesta/metainspector/issues/127/labels{/name}","comments_url":"https://api.github.com/repos/jaimeiniesta/metainspector/issues/127/comments","events_url":"https://api.github.com/repos/jaimeiniesta/metainspector/issues/127/events","html_url":"https://github.com/jaimeiniesta/metainspector/issues/127","id":67778931,"number":127,"title":"MetaInspector coming up empty on some sites","user":{"login":"Justin-Maxwell","id":8223291,"avatar_url":"https://avatars.githubusercontent.com/u/8223291?v=3","gravatar_id":"","url":"https://api.github.com/users/Justin-Maxwell","html_url":"https://github.com/Justin-Maxwell","followers_url":"https://api.github.com/users/Justin-Maxwell/followers","following_url":"https://api.github.com/users/Justin-Maxwell/following{/other_user}","gists_url":"https://api.github.com/users/Justin-Maxwell/gists{/gist_id}","starred_url":"https://api.github.com/users/Justin-Maxwell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Justin-Maxwell/subscriptions","organizations_url":"https://api.github.com/users/Justin-Maxwell/orgs","repos_url":"https://api.github.com/users/Justin-Maxwell/repos","events_url":"https://api.github.com/users/Justin-Maxwell/events{/privacy}","received_events_url":"https://api.github.com/users/Justin-Maxwell/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-04-11T12:11:52Z","updated_at":"2015-04-11T22:13:15Z","closed_at":null,"body":"We're using the MetaInspector gem at tib.tibdit.com, I was trying out a link from ello.co, and MetaInspector comes up empty, both for us, and via metainspectordemo.\r\n\r\nSee: https://metainspectordemo.herokuapp.com/scrape?url=http%3A%2F%2Fello.co%2Fjaymax \r\n\r\nBut https://developers.facebook.com/tools/debug/og/object?q=http%3A%2F%2Fello.co%2Fjaymax produces reasonable results\r\n\r\nExample 2.\r\n\r\nWhile almost useless from an information point of view, fb does get at least a name out of \r\nhttps://developers.facebook.com/tools/debug/og/object?q=http://linkedin.com/in/tibditJustin\r\n\r\nBut https://metainspectordemo.herokuapp.com/scrape?url=http://linkedin.com/in/tibditJustin again shows no information from the page."},"comment":{"url":"https://api.github.com/repos/jaimeiniesta/metainspector/issues/comments/91934815","html_url":"https://github.com/jaimeiniesta/metainspector/issues/127#issuecomment-91934815","issue_url":"https://api.github.com/repos/jaimeiniesta/metainspector/issues/127","id":91934815,"user":{"login":"jaimeiniesta","id":2629,"avatar_url":"https://avatars.githubusercontent.com/u/2629?v=3","gravatar_id":"","url":"https://api.github.com/users/jaimeiniesta","html_url":"https://github.com/jaimeiniesta","followers_url":"https://api.github.com/users/jaimeiniesta/followers","following_url":"https://api.github.com/users/jaimeiniesta/following{/other_user}","gists_url":"https://api.github.com/users/jaimeiniesta/gists{/gist_id}","starred_url":"https://api.github.com/users/jaimeiniesta/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaimeiniesta/subscriptions","organizations_url":"https://api.github.com/users/jaimeiniesta/orgs","repos_url":"https://api.github.com/users/jaimeiniesta/repos","events_url":"https://api.github.com/users/jaimeiniesta/events{/privacy}","received_events_url":"https://api.github.com/users/jaimeiniesta/received_events","type":"User","site_admin":false},"created_at":"2015-04-11T22:13:15Z","updated_at":"2015-04-11T22:13:15Z","body":"I have tried this on my development machine, and on the first requests I could see a lot of data (title, images, description, etc.) -- both running MetaInspector on irb, and from the demo app. No special headers were needed.\r\n\r\nBut then, after 4 or 5 requests to the same URL, I also got a similar response to yours, where you can see that the server is blocking the request and challenging me with [a captcha](http://ello.co/distil_r_captcha.html?Ref=/jaymax&distil_RID=E1CD3036-E096-11E4-9C58-B0E54AFE83E4&distil_TID=20150411220536):\r\n\r\n```ruby\r\n{\"url\"=>\"http://ello.co/jaymax\", \"scheme\"=>\"http\", \"host\"=>\"ello.co\", \"root_url\"=>\"http://ello.co/\", \"title\"=>\"\", \"best_title\"=>nil, \"description\"=>\"\", \"links\"=>{\"internal\"=>[], \"external\"=>[], \"non_http\"=>[]}, \"images\"=>[], \"charset\"=>nil, \"feed\"=>nil, \"content_type\"=>\"text/html\", \"meta_tags\"=>{\"name\"=>{\"robots\"=>[\"NOINDEX, NOFOLLOW\"]}, \"http-equiv\"=>{\"cache-control\"=>[\"max-age=0\", \"no-cache\"], \"expires\"=>[\"0\", \"Tue, 01 Jan 1980 1:00:00 GMT\"], \"pragma\"=>[\"no-cache\"], \"refresh\"=>[\"10; url=/distil_r_captcha.html?Ref=/jaymax&distil_RID=E1CD3036-E096-11E4-9C58-B0E54AFE83E4&distil_TID=20150411220536\"]}, \"property\"=>{}, \"charset\"=>[nil]}, \"favicon\"=>nil, \"response\"=>{\"status\"=>200, \"headers\"=>{\"server\"=>\"nginx\", \"date\"=>\"Sat, 11 Apr 2015 22:05:36 GMT\", \"content-type\"=>\"text/html\", \"transfer-encoding\"=>\"chunked\", \"connection\"=>\"close\", \"vary\"=>\"Accept-Encoding\", \"expires\"=>\"Thu, 01 Jan 1970 00:00:01 GMT\", \"cache-control\"=>\"no-cache, private, no-cache, no-store, must-revalidate\", \"edge-control\"=>\"no-store, bypass-cache\", \"surrogate-control\"=>\"no-store, bypass-cache\"}}}\r\n```\r\n\r\nSo, my theory is that this server is blocking the requests believing it's a bot (and in part we are), and then blocking by IP for a certain period of time.\r\n\r\nTo prevent this, you might prefer to always include an 'User-Agent' string in your headers like we do on the [demo app](https://github.com/jaimeiniesta/metainspectordemo/blob/master/main.rb#L30-L43)."}}