{"action":"created","issue":{"url":"https://api.github.com/repos/sjdirect/abot/issues/91","labels_url":"https://api.github.com/repos/sjdirect/abot/issues/91/labels{/name}","comments_url":"https://api.github.com/repos/sjdirect/abot/issues/91/comments","events_url":"https://api.github.com/repos/sjdirect/abot/issues/91/events","html_url":"https://github.com/sjdirect/abot/issues/91","id":110195117,"number":91,"title":"Suggestion on JavaScript support","user":{"login":"aloneguid","id":3155189,"avatar_url":"https://avatars.githubusercontent.com/u/3155189?v=3","gravatar_id":"","url":"https://api.github.com/users/aloneguid","html_url":"https://github.com/aloneguid","followers_url":"https://api.github.com/users/aloneguid/followers","following_url":"https://api.github.com/users/aloneguid/following{/other_user}","gists_url":"https://api.github.com/users/aloneguid/gists{/gist_id}","starred_url":"https://api.github.com/users/aloneguid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aloneguid/subscriptions","organizations_url":"https://api.github.com/users/aloneguid/orgs","repos_url":"https://api.github.com/users/aloneguid/repos","events_url":"https://api.github.com/users/aloneguid/events{/privacy}","received_events_url":"https://api.github.com/users/aloneguid/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-10-07T10:27:05Z","updated_at":"2015-10-14T14:32:32Z","closed_at":null,"body":"First of all I'm crawling pages with Abot for a few months and it's great, thanks for this project and please keep the work going, it's awesome!\r\n\r\nI've encountered a problem with a few massive and important (for my business) websites that render links with JavaScript which Abot ignores with built-in features for very obvious reasons. To solve that I've created an extension which uses Awesomium project (Chrome runtime) and that works very well as it does execute JS locally and speed wise it's not too far from HttpWebRequest, obviously there is an overhead, but it's bearable.\r\n\r\nI wouldn't like to keep it in my private repo and want to contribute to Abot. I was thinking of creating a new project alongside your solution which people can reference to replace the built-in crawler - Awesomium has lots of external dependencies, no nuget package and is a bit heavy. Can you please suggest the best way to add this into Abot?"},"comment":{"url":"https://api.github.com/repos/sjdirect/abot/issues/comments/148068148","html_url":"https://github.com/sjdirect/abot/issues/91#issuecomment-148068148","issue_url":"https://api.github.com/repos/sjdirect/abot/issues/91","id":148068148,"user":{"login":"aloneguid","id":3155189,"avatar_url":"https://avatars.githubusercontent.com/u/3155189?v=3","gravatar_id":"","url":"https://api.github.com/users/aloneguid","html_url":"https://github.com/aloneguid","followers_url":"https://api.github.com/users/aloneguid/followers","following_url":"https://api.github.com/users/aloneguid/following{/other_user}","gists_url":"https://api.github.com/users/aloneguid/gists{/gist_id}","starred_url":"https://api.github.com/users/aloneguid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aloneguid/subscriptions","organizations_url":"https://api.github.com/users/aloneguid/orgs","repos_url":"https://api.github.com/users/aloneguid/repos","events_url":"https://api.github.com/users/aloneguid/events{/privacy}","received_events_url":"https://api.github.com/users/aloneguid/received_events","type":"User","site_admin":false},"created_at":"2015-10-14T14:32:32Z","updated_at":"2015-10-14T14:32:32Z","body":"Hey guys, I almost gave up due to the NuGet dependency hell. NuGet doesn't play well with a mixture of native/managed dlls, or when it does, upstream projects don't get the binaries to the target build folder. Whatever I do I can't make CI transparent enough and embed Awesomium, which makes it almost useless.\r\n\r\nOn the other hand, we decided that we can't use Abot in the crawling solution these days, unfortunately. We used Abot for some time and it was great, but due to overcomplicated nature of our crawlers and today's web most of the cool Abot features don't play well when it comes to speed and vast differences of rendering engines.\r\n\r\nI'll keep trying to work on this, however I may give up and build a framework specifically for Awesomium which simply accepts site base URL on input and produces a zip archive on output."}}