{"action":"created","issue":{"url":"https://api.github.com/repos/NREL/api-umbrella/issues/140","labels_url":"https://api.github.com/repos/NREL/api-umbrella/issues/140/labels{/name}","comments_url":"https://api.github.com/repos/NREL/api-umbrella/issues/140/comments","events_url":"https://api.github.com/repos/NREL/api-umbrella/issues/140/events","html_url":"https://github.com/NREL/api-umbrella/issues/140","id":77959839,"number":140,"title":"How to access Elasticsearch analytics data?","user":{"login":"brylie","id":17307,"avatar_url":"https://avatars.githubusercontent.com/u/17307?v=3","gravatar_id":"","url":"https://api.github.com/users/brylie","html_url":"https://github.com/brylie","followers_url":"https://api.github.com/users/brylie/followers","following_url":"https://api.github.com/users/brylie/following{/other_user}","gists_url":"https://api.github.com/users/brylie/gists{/gist_id}","starred_url":"https://api.github.com/users/brylie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brylie/subscriptions","organizations_url":"https://api.github.com/users/brylie/orgs","repos_url":"https://api.github.com/users/brylie/repos","events_url":"https://api.github.com/users/brylie/events{/privacy}","received_events_url":"https://api.github.com/users/brylie/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-05-19T06:38:23Z","updated_at":"2015-06-09T04:57:46Z","closed_at":null,"body":"We would like to display analytics in a user dashboard, similar to API Umbrella Web. How can we access analytics data from the API Umbrella platform?"},"comment":{"url":"https://api.github.com/repos/NREL/api-umbrella/issues/comments/110226345","html_url":"https://github.com/NREL/api-umbrella/issues/140#issuecomment-110226345","issue_url":"https://api.github.com/repos/NREL/api-umbrella/issues/140","id":110226345,"user":{"login":"GUI","id":12112,"avatar_url":"https://avatars.githubusercontent.com/u/12112?v=3","gravatar_id":"","url":"https://api.github.com/users/GUI","html_url":"https://github.com/GUI","followers_url":"https://api.github.com/users/GUI/followers","following_url":"https://api.github.com/users/GUI/following{/other_user}","gists_url":"https://api.github.com/users/GUI/gists{/gist_id}","starred_url":"https://api.github.com/users/GUI/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GUI/subscriptions","organizations_url":"https://api.github.com/users/GUI/orgs","repos_url":"https://api.github.com/users/GUI/repos","events_url":"https://api.github.com/users/GUI/events{/privacy}","received_events_url":"https://api.github.com/users/GUI/received_events","type":"User","site_admin":false},"created_at":"2015-06-09T04:57:46Z","updated_at":"2015-06-09T04:57:46Z","body":"@brylie: Very sorry for the delay following up (I've been out of town a fair bit recently and have been busy catching up).\r\n\r\nRegarding analytics, yes, essentially you'll be using the ElasticSearch APIs to fetch or aggregate analytics. Here's a few related tidbits and pointers:\r\n\r\n- In the admin, the [api-umbrella/v1/analytics/drilldown.json](https://github.com/NREL/api-umbrella-web/blob/1f43fe1cc7f3ece819a028a64677e4dfd795a035/app/controllers/api/v1/analytics_controller.rb#L7-L98) endpoint provides an example of how we provide a wrapper API to perform the analytics.\r\n- You'll likely want to perform analytics queries via a similar wrapper API (rather than hitting the ElasticSearch HTTP API directly from the client-side), so you don't publicly expose your ElasticSearch database and so the wrapper can take care of things like authorization.\r\n- In that example, we scope the ElasticSearch analytics queries so that admins can only view the analytics their authorized to view (via the admin scopes and groups). This is done via the [LogSearchPolicy](https://github.com/NREL/api-umbrella-web/blob/1f43fe1cc7f3ece819a028a64677e4dfd795a035/app/policies/log_search_policy.rb). I imagine you'd want to scope the queries in a similar fashion for any user-facing endpoints to ensure that api key users can only view their own analytics.\r\n- In the Rails app we use a custom model, [LogSearch](https://github.com/NREL/api-umbrella-web/blob/1f43fe1cc7f3ece819a028a64677e4dfd795a035/app/models/log_search.rb), to handle calls to the ElasticSearch API. It then returns results in a [LogResult](https://github.com/NREL/api-umbrella-web/blob/1f43fe1cc7f3ece819a028a64677e4dfd795a035/app/models/log_result.rb) class. You don't have to use either class for querying ElasticSearch, but they just provide some wrapper functionality for how we commonly query our analytics.\r\n- The ElasticSearch data is split into per-month indexes named `api-umbrella-logs-YYYY-MM`. So you need to query all of the indexes that fall within the date ranges of your queries. [LogSearch#indexes](https://github.com/NREL/api-umbrella-web/blob/1f43fe1cc7f3ece819a028a64677e4dfd795a035/app/models/log_search.rb#L477-L485) provides an example.\r\n- Inside ElasticSearch, each log entry is logged as the `log` type.\r\n- Here's a test documenting the [available fields](https://github.com/NREL/api-umbrella-router/blob/master/test/integration/logging.js#L206-L298) on each log entry inside ElasticSearch.\r\n- Here's the [elasticsearch template](https://github.com/NREL/api-umbrella-router/blob/93327a6bfd2a52863af15e72da0d54eeb647c010/config/elasticsearch_templates.json#L37-L154) defining custom mappings.\r\n- If you're logged in as a super-user to the admin, you can inspect the ElasticSearch database via the \"head\" plugin we pre-install which can be a handy way to see what's actually being stored in ElasticSearch and how the different indices are laid out. After logging in as a super-user you can get there by going to `https://YOUR_DOMAIN/admin/elasticsearch/_plugin/head/`\r\n\r\nSorry again the internal documentation isn't in better shape and for the hodge-podge of information, but I hope that helps. Definitely feel free to reach out if anything isn't clear or you have questions."}}