{"action":"created","issue":{"url":"https://api.github.com/repos/cantino/huginn/issues/1171","labels_url":"https://api.github.com/repos/cantino/huginn/issues/1171/labels{/name}","comments_url":"https://api.github.com/repos/cantino/huginn/issues/1171/comments","events_url":"https://api.github.com/repos/cantino/huginn/issues/1171/events","html_url":"https://github.com/cantino/huginn/issues/1171","id":119831476,"number":1171,"title":"How to scrape data from a dynamic website that requires user interaction","user":{"login":"cindyborg","id":16108918,"avatar_url":"https://avatars.githubusercontent.com/u/16108918?v=3","gravatar_id":"","url":"https://api.github.com/users/cindyborg","html_url":"https://github.com/cindyborg","followers_url":"https://api.github.com/users/cindyborg/followers","following_url":"https://api.github.com/users/cindyborg/following{/other_user}","gists_url":"https://api.github.com/users/cindyborg/gists{/gist_id}","starred_url":"https://api.github.com/users/cindyborg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cindyborg/subscriptions","organizations_url":"https://api.github.com/users/cindyborg/orgs","repos_url":"https://api.github.com/users/cindyborg/repos","events_url":"https://api.github.com/users/cindyborg/events{/privacy}","received_events_url":"https://api.github.com/users/cindyborg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2015-12-01T23:21:28Z","updated_at":"2015-12-14T20:32:16Z","closed_at":null,"body":"I just found Huginn and is learning how to use it.\r\nI need to scrape data from a website with dynamic data. It requires user input first to get the rendered page with the data I want (ie. select an option from a drop down list, then click a button). \r\nIs there a way to achieve this through huginn agents only? ( I have looked at Website agent and webhook agent). Or do I have to write extra ruby code to simulate the user interactions and pass in the rendered page to an agent? If that's the case, how would I pass in the rendered page?\r\n\r\nThanks\r\nCindy"},"comment":{"url":"https://api.github.com/repos/cantino/huginn/issues/comments/164550782","html_url":"https://github.com/cantino/huginn/issues/1171#issuecomment-164550782","issue_url":"https://api.github.com/repos/cantino/huginn/issues/1171","id":164550782,"user":{"login":"signallhill","id":8616418,"avatar_url":"https://avatars.githubusercontent.com/u/8616418?v=3","gravatar_id":"","url":"https://api.github.com/users/signallhill","html_url":"https://github.com/signallhill","followers_url":"https://api.github.com/users/signallhill/followers","following_url":"https://api.github.com/users/signallhill/following{/other_user}","gists_url":"https://api.github.com/users/signallhill/gists{/gist_id}","starred_url":"https://api.github.com/users/signallhill/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/signallhill/subscriptions","organizations_url":"https://api.github.com/users/signallhill/orgs","repos_url":"https://api.github.com/users/signallhill/repos","events_url":"https://api.github.com/users/signallhill/events{/privacy}","received_events_url":"https://api.github.com/users/signallhill/received_events","type":"User","site_admin":false},"created_at":"2015-12-14T20:32:16Z","updated_at":"2015-12-14T20:32:16Z","body":"@cantino Please help with the post Agent. Below is response header from network tab in Chrome\r\n\r\n\r\n\r\nGET /c/pagination?inputJSON=%7B%22startPage%22%3A30%2C%22articleId%22%3A%221adac%22%7D HTTP/1.1\r\nHost: www.xyz.com\r\nConnection: keep-alive\r\nAccept: application/json, text/javascript, */*; q=0.01\r\nX-Requested-With: XMLHttpRequest\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36\r\nReferer: https://www.xyz.com/c/d\r\nAccept-Encoding: gzip, deflate, sdch\r\nAccept-Language: en-US,en;q=0.8\r\nCookie: __uvt=; e18d5d532a99a.1445623125.5.1450122485.1449679512.a2400eb9-7418-4311-ac40-4993ed2b0e7b; spdiceses.b640=*\r\n"}}