{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/node_exporter/issues/70","labels_url":"https://api.github.com/repos/prometheus/node_exporter/issues/70/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/node_exporter/issues/70/comments","events_url":"https://api.github.com/repos/prometheus/node_exporter/issues/70/events","html_url":"https://github.com/prometheus/node_exporter/issues/70","id":76080872,"number":70,"title":"high io load after running for a while","user":{"login":"a86c6f7964","id":165381,"avatar_url":"https://avatars.githubusercontent.com/u/165381?v=3","gravatar_id":"","url":"https://api.github.com/users/a86c6f7964","html_url":"https://github.com/a86c6f7964","followers_url":"https://api.github.com/users/a86c6f7964/followers","following_url":"https://api.github.com/users/a86c6f7964/following{/other_user}","gists_url":"https://api.github.com/users/a86c6f7964/gists{/gist_id}","starred_url":"https://api.github.com/users/a86c6f7964/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a86c6f7964/subscriptions","organizations_url":"https://api.github.com/users/a86c6f7964/orgs","repos_url":"https://api.github.com/users/a86c6f7964/repos","events_url":"https://api.github.com/users/a86c6f7964/events{/privacy}","received_events_url":"https://api.github.com/users/a86c6f7964/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-05-13T18:34:20Z","updated_at":"2015-05-28T16:14:32Z","closed_at":null,"body":"I have been running the node_exporter version 0.8.0 for a while now. Maybe 2 weeks, and I am now seeing a pretty high IO load. If I run iotop then I am seeing it reading megabytes of data in a matter of seconds. It is basically using 99% of the io on the system.\r\n\r\nIf I run lsof -p $PID then I see that there are a lot of open file descriptors to /pro/stat and /proc/$PID/limits and /proc/meminfo and /proc/$PID/stat\r\n\r\nif this helps\r\n```sudo lsof -p 25359 | awk '{print $9}' | sort | uniq -c | sort -n```\r\n```\r\n      1 *:7301\r\n      1 /dev/null\r\n      1 /proc/diskstats\r\n      1 /tmp/node_exporter.ip-XXXX.root.log.INFO.20150501-131008.25359\r\n      1 /usr/lib64/ld-2.19.so\r\n      1 /usr/lib64/libc-2.19.so\r\n      1 /usr/lib64/libnss_files-2.19.so\r\n      1 /usr/lib64/libpthread-2.19.so\r\n      1 /var/lib/prometheus/node_exporter\r\n      1 NAME\r\n      1 anon_inode\r\n      1 ip-XXXX.compute.internal:7301->ip-XXXX.compute.internal:49115\r\n      2 /\r\n      2 /proc/25359/mounts\r\n      2 /proc/25359/net/dev\r\n      2 socket\r\n     12 /proc/25359/limits\r\n     17 /proc/stat\r\n     19 /proc/25359/stat\r\n     20 /proc/meminfo\r\n     47 can't identify protocol\r\n```"},"comment":{"url":"https://api.github.com/repos/prometheus/node_exporter/issues/comments/106454193","html_url":"https://github.com/prometheus/node_exporter/issues/70#issuecomment-106454193","issue_url":"https://api.github.com/repos/prometheus/node_exporter/issues/70","id":106454193,"user":{"login":"a86c6f7964","id":165381,"avatar_url":"https://avatars.githubusercontent.com/u/165381?v=3","gravatar_id":"","url":"https://api.github.com/users/a86c6f7964","html_url":"https://github.com/a86c6f7964","followers_url":"https://api.github.com/users/a86c6f7964/followers","following_url":"https://api.github.com/users/a86c6f7964/following{/other_user}","gists_url":"https://api.github.com/users/a86c6f7964/gists{/gist_id}","starred_url":"https://api.github.com/users/a86c6f7964/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a86c6f7964/subscriptions","organizations_url":"https://api.github.com/users/a86c6f7964/orgs","repos_url":"https://api.github.com/users/a86c6f7964/repos","events_url":"https://api.github.com/users/a86c6f7964/events{/privacy}","received_events_url":"https://api.github.com/users/a86c6f7964/received_events","type":"User","site_admin":false},"created_at":"2015-05-28T16:14:32Z","updated_at":"2015-05-28T16:14:32Z","body":"Hmm `process_resident_memory_bytes` says 12MB ish, and so does `ps` but cAdvisor is telling me that it is using all 100MB that is allocated for that cgroup.\r\n\r\n`lsof -p 3662`\r\n```\r\nCOMMAND    PID USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME\r\nnode_expo 3662 root  cwd    DIR              202,9     4096        2 /\r\nnode_expo 3662 root  rtd    DIR              202,9     4096        2 /\r\nnode_expo 3662 root  txt    REG              202,9 10594835       30 /var/lib/prometheus/node_exporter\r\nnode_expo 3662 root  mem    REG              202,3    51152    33867 /usr/lib64/libnss_files-2.19.so\r\nnode_expo 3662 root  mem    REG              202,3  1787096    33884 /usr/lib64/libc-2.19.so\r\nnode_expo 3662 root  mem    REG              202,3   109288    33865 /usr/lib64/libpthread-2.19.so\r\nnode_expo 3662 root  mem    REG              202,3   144720    33863 /usr/lib64/ld-2.19.so\r\nnode_expo 3662 root    0r   CHR                1,3      0t0     1036 /dev/null\r\nnode_expo 3662 root    1u  unix 0xffff88006ecfde80      0t0 31659765 socket\r\nnode_expo 3662 root    2u  unix 0xffff88006ecfde80      0t0 31659765 socket\r\nnode_expo 3662 root    3u   REG               0,30 92607463 31659776 /tmp/node_exporter.ip-172-31-17-127.root.log.INFO.20150520-162710.3662\r\nnode_expo 3662 root    4u  IPv6           31659778      0t0      TCP *:7301 (LISTEN)\r\nnode_expo 3662 root    5u  0000               0,10        0     6638 anon_inode\r\n```"}}