{"action":"created","issue":{"url":"https://api.github.com/repos/keboola/generic-extractor/issues/4","labels_url":"https://api.github.com/repos/keboola/generic-extractor/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/keboola/generic-extractor/issues/4/comments","events_url":"https://api.github.com/repos/keboola/generic-extractor/issues/4/events","html_url":"https://github.com/keboola/generic-extractor/issues/4","id":119953333,"number":4,"title":"Pagination","user":{"login":"jiritobolka","id":951201,"avatar_url":"https://avatars.githubusercontent.com/u/951201?v=3","gravatar_id":"","url":"https://api.github.com/users/jiritobolka","html_url":"https://github.com/jiritobolka","followers_url":"https://api.github.com/users/jiritobolka/followers","following_url":"https://api.github.com/users/jiritobolka/following{/other_user}","gists_url":"https://api.github.com/users/jiritobolka/gists{/gist_id}","starred_url":"https://api.github.com/users/jiritobolka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jiritobolka/subscriptions","organizations_url":"https://api.github.com/users/jiritobolka/orgs","repos_url":"https://api.github.com/users/jiritobolka/repos","events_url":"https://api.github.com/users/jiritobolka/events{/privacy}","received_events_url":"https://api.github.com/users/jiritobolka/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-12-02T14:28:14Z","updated_at":"2015-12-03T01:16:24Z","closed_at":null,"body":"We're trying to get data from Hubspot but for Contacts (for example) we want to download all contacts, but always getting only 500 contacts which is page limit I would say...\r\n\r\n`\r\n{\r\n  \"parameters\": {\r\n    \"api\": {\r\n      \"baseUrl\": \"https://api.hubapi.com\",\r\n      \"authentication\": {\r\n        \"type\": \"url.query\"\r\n      },\r\n      \"query\": {\r\n        \"hapikey\": {\r\n          \"attr\": \"hkey\"\r\n        }\r\n      },\r\n      \"pagination.method\": \"offset\"\r\n    },\r\n    \"config\": {\r\n      \"outputBucket\": \"ex-hubspot\",\r\n      \"id\": \"Hubspot\",\r\n      \"hkey\": \"HKEY\",\r\n      \"http\": {\r\n        \"headers\": {\r\n          \"Accept\": \"application/json\"\r\n        }\r\n      },\r\n      \"jobs\": [\r\n        {\r\n          \"endpoint\": \"contacts/v1/lists/all/contacts/all?count=1000&offset=0\",\r\n          \"dataType\": \"contacts\",\r\n          \"dataField\": \"contacts\"\r\n        }\r\n      ]\r\n    }\r\n  }\r\n}\r\n`\r\n\r\nHow can I get all records or how can I get the extractor to iterate through pages and return all values, is it supported? Or do we just miss something? Thanks!"},"comment":{"url":"https://api.github.com/repos/keboola/generic-extractor/issues/comments/161485188","html_url":"https://github.com/keboola/generic-extractor/issues/4#issuecomment-161485188","issue_url":"https://api.github.com/repos/keboola/generic-extractor/issues/4","id":161485188,"user":{"login":"zimami","id":16126676,"avatar_url":"https://avatars.githubusercontent.com/u/16126676?v=3","gravatar_id":"","url":"https://api.github.com/users/zimami","html_url":"https://github.com/zimami","followers_url":"https://api.github.com/users/zimami/followers","following_url":"https://api.github.com/users/zimami/following{/other_user}","gists_url":"https://api.github.com/users/zimami/gists{/gist_id}","starred_url":"https://api.github.com/users/zimami/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zimami/subscriptions","organizations_url":"https://api.github.com/users/zimami/orgs","repos_url":"https://api.github.com/users/zimami/repos","events_url":"https://api.github.com/users/zimami/events{/privacy}","received_events_url":"https://api.github.com/users/zimami/received_events","type":"User","site_admin":false},"created_at":"2015-12-03T01:16:24Z","updated_at":"2015-12-03T01:16:24Z","body":"Hi kachnitel,\r\nthanks for your quick response, but it helps just partly. The main core of the problem is in a limit from Hubspot. \r\n\r\nWe have more than 10,000 records, but through one call of  \"endpoint\": \"contacts/v1/lists/all/contacts/all\" we can download just 100 records according to http://developers.hubspot.com/docs/methods/contacts/get_contacts. With your example, I was able to download little bit less than 1000 records. If I rewrite limit parameter from 1000 to 10,000 the result is same and it reach the limit of Hubspot API.\r\n\r\nAccording to Hubspot documentation is possible to get next set of records by next call with parameter &vidOffset=X. We can do it manually and write 100 jobs with the particular parameter, but it would be better have some \"fetch\" option and go through pages programmatically not by writing hundreds of lines of code. Thanks for any help."}}