{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/jmx_exporter/issues/23","labels_url":"https://api.github.com/repos/prometheus/jmx_exporter/issues/23/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/jmx_exporter/issues/23/comments","events_url":"https://api.github.com/repos/prometheus/jmx_exporter/issues/23/events","html_url":"https://github.com/prometheus/jmx_exporter/issues/23","id":94887276,"number":23,"title":"Attribute querying approach doesn't scale","user":{"login":"rhuss","id":99080,"avatar_url":"https://avatars.githubusercontent.com/u/99080?v=3","gravatar_id":"","url":"https://api.github.com/users/rhuss","html_url":"https://github.com/rhuss","followers_url":"https://api.github.com/users/rhuss/followers","following_url":"https://api.github.com/users/rhuss/following{/other_user}","gists_url":"https://api.github.com/users/rhuss/gists{/gist_id}","starred_url":"https://api.github.com/users/rhuss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhuss/subscriptions","organizations_url":"https://api.github.com/users/rhuss/orgs","repos_url":"https://api.github.com/users/rhuss/repos","events_url":"https://api.github.com/users/rhuss/events{/privacy}","received_events_url":"https://api.github.com/users/rhuss/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":18,"created_at":"2015-07-14T07:44:39Z","updated_at":"2015-07-21T09:24:11Z","closed_at":null,"body":"The approach to [first query](https://github.com/prometheus/jmx_exporter/blob/master/collector/src/main/java/io/prometheus/jmx/JmxScraper.java#L62-67), [read all attributes](https://github.com/prometheus/jmx_exporter/blob/master/collector/src/main/java/io/prometheus/jmx/JmxScraper.java#L88) and finally apply the rule set [on each attribute individually](https://github.com/prometheus/jmx_exporter/blob/master/collector/src/main/java/io/prometheus/jmx/JmxCollector.java#L211-211) doesn't  scale at all in the real world.\r\n\r\nIt is not uncommon that a Java server has hundreds MBean registered, each with five to ten attributes. Reading an attribute is normally cheap if only member variables are stored, but many MBean calculate the attribute value also on the fly so that an average readtime of 1-2ms is a good average for many MBeans. Considering this, a run **only for reading all attributes** can take e.g. 400 MBeans * 8  attributes * 2ms = 6.4 seconds\r\n\r\nOn each of this 3200 attributes the whole rule set is applied individually. Assuming that people tend to generate quite generic and large rulesets so that they can have a uniform agent installation everywhere, this multiples to the number above.\r\n\r\nConsidering a standard scrape interval of 15s, I think its fair to say that performing monitoring that way is intrusive.\r\n\r\nI've prepared a [demo project](https://github.com/rhuss/jmx-performance) to demonstrate this. It is only about reading attributes, not applying rule sets. The sample project consist of an MBean which mimics the algorithm of jmx_exporter. Time measurement is done with [metrics](http://metrics.dropwizard.io) and exported also via MBeans. See the [README](https://github.com/rhuss/jmx-performance/README) for details. \r\n\r\nThe result for an vanilla JBoss Wildfly 8.2.0 with only this test war deployed is:\r\n\r\n* MBeans: 880\r\n* Attributes total: 4137\r\n* Average attribute read time: 1.97 ms\r\n* Reading all MBeans and attributes: 11 s\r\n\r\nAnd again, all that without any rule applied, only reading of the attribute. And believe me, Wildfly is not the most populated server I've seen. Also it is not uncommon in integration scenarios to have many application specific MBeans, e.g one for each Camel Route which will add to this basic number. \r\n\r\nMy suggestion is to apply the rule before and use it for the query instead of this sledge hammer approach. "},"comment":{"url":"https://api.github.com/repos/prometheus/jmx_exporter/issues/comments/123234486","html_url":"https://github.com/prometheus/jmx_exporter/issues/23#issuecomment-123234486","issue_url":"https://api.github.com/repos/prometheus/jmx_exporter/issues/23","id":123234486,"user":{"login":"brian-brazil","id":7115638,"avatar_url":"https://avatars.githubusercontent.com/u/7115638?v=3","gravatar_id":"","url":"https://api.github.com/users/brian-brazil","html_url":"https://github.com/brian-brazil","followers_url":"https://api.github.com/users/brian-brazil/followers","following_url":"https://api.github.com/users/brian-brazil/following{/other_user}","gists_url":"https://api.github.com/users/brian-brazil/gists{/gist_id}","starred_url":"https://api.github.com/users/brian-brazil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brian-brazil/subscriptions","organizations_url":"https://api.github.com/users/brian-brazil/orgs","repos_url":"https://api.github.com/users/brian-brazil/repos","events_url":"https://api.github.com/users/brian-brazil/events{/privacy}","received_events_url":"https://api.github.com/users/brian-brazil/received_events","type":"User","site_admin":false},"created_at":"2015-07-21T09:24:11Z","updated_at":"2015-07-21T09:24:11Z","body":"> Of course, this would mean that the rules need to be analysed before they are applied. Its probably not super easy but should be feasible. If you don't mind I would give it a try and prepare a PR potentially.\r\n\r\nHere's a thought I just had. Doing such an analysis will be very difficult, however what would work is to cache what attributes we didn't use at the last collection and then not read them the next time around.\r\nThe first scrape will still be slow, but as long as it's not too long that should work out."}}