{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/jmx_exporter/issues/23","labels_url":"https://api.github.com/repos/prometheus/jmx_exporter/issues/23/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/jmx_exporter/issues/23/comments","events_url":"https://api.github.com/repos/prometheus/jmx_exporter/issues/23/events","html_url":"https://github.com/prometheus/jmx_exporter/issues/23","id":94887276,"number":23,"title":"Attribute querying approach doesn't scale","user":{"login":"rhuss","id":99080,"avatar_url":"https://avatars.githubusercontent.com/u/99080?v=3","gravatar_id":"","url":"https://api.github.com/users/rhuss","html_url":"https://github.com/rhuss","followers_url":"https://api.github.com/users/rhuss/followers","following_url":"https://api.github.com/users/rhuss/following{/other_user}","gists_url":"https://api.github.com/users/rhuss/gists{/gist_id}","starred_url":"https://api.github.com/users/rhuss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhuss/subscriptions","organizations_url":"https://api.github.com/users/rhuss/orgs","repos_url":"https://api.github.com/users/rhuss/repos","events_url":"https://api.github.com/users/rhuss/events{/privacy}","received_events_url":"https://api.github.com/users/rhuss/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-07-14T07:44:39Z","updated_at":"2015-07-14T10:56:19Z","closed_at":null,"body":"The approach to [first query](https://github.com/prometheus/jmx_exporter/blob/master/collector/src/main/java/io/prometheus/jmx/JmxScraper.java#L62-67), [read all attributes](https://github.com/prometheus/jmx_exporter/blob/master/collector/src/main/java/io/prometheus/jmx/JmxScraper.java#L88) and finally apply the rule set [on each attribute individually](https://github.com/prometheus/jmx_exporter/blob/master/collector/src/main/java/io/prometheus/jmx/JmxCollector.java#L211-211) doesn't  scale at all in the real world.\r\n\r\nIt is not uncommon that a Java server has hundreds MBean registered, each with five to ten attributes. Reading an attribute is normally cheap if only member variables are stored, but many MBean calculate the attribute value also on the fly so that an average readtime of 1-2ms is a good average for many MBeans. Considering this, a run **only for reading all attributes** can take e.g. 400 MBeans * 8  attributes * 2ms = 6.4 seconds\r\n\r\nOn each of this 3200 attributes the whole rule set is applied individually. Assuming that people tend to generate quite generic and large rulesets so that they can have a uniform agent installation everywhere, this multiples to the number above.\r\n\r\nConsidering a standard scrape interval of 15s, I think its fair to say that performing monitoring that way is intrusive.\r\n\r\nI've prepared a [demo project](https://github.com/rhuss/jmx-performance) to demonstrate this. It is only about reading attributes, not applying rule sets. The sample project consist of an MBean which mimics the algorithm of jmx_exporter. Time measurement is done with [metrics](http://metrics.dropwizard.io) and exported also via MBeans. See the [README](https://github.com/rhuss/jmx-performance/README) for details. \r\n\r\nThe result for an vanilla JBoss Wildfly 8.2.0 with only this test war deployed is:\r\n\r\n* MBeans: 880\r\n* Attributes total: 4137\r\n* Average attribute read time: 1.97 ms\r\n* Reading all MBeans and attributes: 11 s\r\n\r\nAnd again, all that without any rule applied, only reading of the attribute. And believe me, Wildfly is not the most populated server I've seen. Also it is not uncommon in integration scenarios to have many application specific MBeans, e.g one for each Camel Route which will add to this basic number. \r\n\r\nMy suggestion is to apply the rule before and use it for the query instead of this sledge hammer approach. "},"comment":{"url":"https://api.github.com/repos/prometheus/jmx_exporter/issues/comments/121200792","html_url":"https://github.com/prometheus/jmx_exporter/issues/23#issuecomment-121200792","issue_url":"https://api.github.com/repos/prometheus/jmx_exporter/issues/23","id":121200792,"user":{"login":"brian-brazil","id":7115638,"avatar_url":"https://avatars.githubusercontent.com/u/7115638?v=3","gravatar_id":"","url":"https://api.github.com/users/brian-brazil","html_url":"https://github.com/brian-brazil","followers_url":"https://api.github.com/users/brian-brazil/followers","following_url":"https://api.github.com/users/brian-brazil/following{/other_user}","gists_url":"https://api.github.com/users/brian-brazil/gists{/gist_id}","starred_url":"https://api.github.com/users/brian-brazil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/brian-brazil/subscriptions","organizations_url":"https://api.github.com/users/brian-brazil/orgs","repos_url":"https://api.github.com/users/brian-brazil/repos","events_url":"https://api.github.com/users/brian-brazil/events{/privacy}","received_events_url":"https://api.github.com/users/brian-brazil/received_events","type":"User","site_admin":false},"created_at":"2015-07-14T10:56:19Z","updated_at":"2015-07-14T10:56:19Z","body":"> This would ease maintenance, but of course, if the rules are not used in the MBean selection process, that doesn't make much sense.\r\n\r\nIt still makes sense I think.\r\n\r\n> Whats about making this a multi stage process: Use the MBean part of the rule to filter out MBeans in the query, then filter on the attributes and only then read it and filter on the value in a loop ?\r\n\r\nAs MBeans are used in such a wide variety of way, the way we generically support this is by putting everything into a string and letting you throw regexes at it. Accordingly we can't split out the MBean name as that'd require reverse engineering the regexes.\r\n\r\nA blacklist of some form is an option, to deal with particularly slow or problematic mbeans/attributes.\r\n\r\n> I don't think that JMX per se is slow, it depends on the amount of requests (4500 attribute reads/run in this example)\r\n\r\n4.5k metrics isn't particularly large. Doing a quick test with the Python client (which is less performant than the Java client), 4.5k metrics takes 60ms to render.\r\n\r\n> I'm pretty sure that your simpleclient easily becomes slow, too, if he had to read expensive (i.e. on-the-fly calculated) attributes, too. \r\n\r\nIt does permit that for special cases (e.g. that how the jmx exporter works), but in the general case we encourage usage such that updating metrics takes 10-15ns. On-the-fly calculations are discouraged, as they tend to be slow, harder to implement and there's usually a better way.\r\n\r\n> To summarize: I don't think that JMX is slow, its the way how values are fetched here.\r\n\r\nIt sounds like it may be how the values are generated. Something like this should take nanoseconds or microseconds per read, not milliseconds."}}