{"action":"opened","issue":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4835","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4835/labels{/name}","comments_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4835/comments","events_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/4835/events","html_url":"https://github.com/rethinkdb/rethinkdb/issues/4835","id":106246421,"number":4835,"title":"2.1.3 server very slow, high load and high iowait","user":{"login":"hwestbrook","id":335309,"avatar_url":"https://avatars.githubusercontent.com/u/335309?v=3","gravatar_id":"","url":"https://api.github.com/users/hwestbrook","html_url":"https://github.com/hwestbrook","followers_url":"https://api.github.com/users/hwestbrook/followers","following_url":"https://api.github.com/users/hwestbrook/following{/other_user}","gists_url":"https://api.github.com/users/hwestbrook/gists{/gist_id}","starred_url":"https://api.github.com/users/hwestbrook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hwestbrook/subscriptions","organizations_url":"https://api.github.com/users/hwestbrook/orgs","repos_url":"https://api.github.com/users/hwestbrook/repos","events_url":"https://api.github.com/users/hwestbrook/events{/privacy}","received_events_url":"https://api.github.com/users/hwestbrook/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-09-13T22:49:08Z","updated_at":"2015-09-13T22:49:08Z","closed_at":null,"body":"After updating to 2.1.3, my server has become extremely slow. Investingating possible causes has focused me on the high load and high iowait stats. I'm running rethinkdb on a t3.micro. \r\n\r\nStats\r\n\r\niostat:\r\n```\r\nubuntu@ip-172-31-18-121:~$ iostat -x 5 5\r\nLinux 3.13.0-48-generic (ip-172-31-18-121) \t09/13/2015 \t_x86_64_\t(1 CPU)\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n           2.45    0.00    0.79   77.20   19.44    0.13\r\n\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nxvda              2.28     1.18   61.50    5.65  1173.37   266.14    42.87    29.50  439.36  441.02  421.23  11.99  80.49\r\nxvdf              0.03     1.12    0.16    0.04     0.75     4.65    52.90     0.00    1.53    0.54    5.31   0.77   0.02\r\n\r\navg-cpu:  %user   %nice %system %iowait  %steal   %idle\r\n           1.81    0.00    0.40   96.38    1.41    0.00\r\n\r\nDevice:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util\r\nxvda              0.00     5.63   87.93    6.24   915.09    57.95    20.67    18.11  181.07  184.51  132.52  10.53  99.15\r\nxvdf              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00\r\n```\r\n\r\ntop: \r\n```\r\ntop - 22:40:19 up 30 min,  2 users,  load average: 64.20, 63.86, 54.29\r\n%Cpu(s):  6.1 us,  1.0 sy,  0.0 ni,  0.0 id, 92.6 wa,  0.0 hi,  0.0 si,  0.3 st\r\n%Cpu(s):  3.1 us,  0.3 sy,  0.0 ni,  0.0 id, 96.3 wa,  0.0 hi,  0.0 si,  0.3 st\r\nMiB Mem:   992.473 total,  927.391 used,   65.082 free,    0.484 buffers\r\nMiB Swap: 5119.996 total,   22.984 used, 5097.012 free.   30.223 cached Mem\r\n\r\n    1 root      20   0   32.7m   1.0m   0.0m S  0.0  0.1   0:01.14 init\r\n 1133 rethink+  20   0 1263.2m 827.4m   4.6m S  5.3 83.4   1:11.82 rethinkdb\r\n    1 root      20   0   32.7m   1.0m   0.0m S  0.0  0.1   0:01.14 init\r\n    2 root      20   0    0.0m   0.0m   0.0m S  0.0  0.0   0:00.00 kthreadd\r\n    3 root      20   0    0.0m   0.0m   0.0m S  0.0  0.0   0:00.00 ksoftirqd/0\r\n```\r\n\r\nsar 1: \r\n```\r\nubuntu@ip-172-31-18-121:~$ sar 1\r\nLinux 3.13.0-48-generic (ip-172-31-18-121) \t09/13/2015 \t_x86_64_\t(1 CPU)\r\n\r\n10:41:10 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle\r\n10:41:11 PM     all      1.02      0.00      1.02     96.94      1.02      0.00\r\n10:41:12 PM     all      2.06      0.00      1.03     96.91      0.00      0.00\r\n10:41:13 PM     all      0.00      0.00      0.00    100.00      0.00      0.00\r\n10:41:14 PM     all      0.00      0.00      0.00     98.99      1.01      0.00\r\n10:41:15 PM     all      2.02      0.00      1.01     96.97      0.00      0.00\r\n10:41:16 PM     all     14.85      0.00      1.98     83.17      0.00      0.00\r\n10:41:17 PM     all      2.94      0.00      1.96     95.10      0.00      0.00\r\n10:41:18 PM     all      6.06      0.00      1.01     92.93      0.00      0.00\r\n10:41:19 PM     all      3.92      0.00      1.96     94.12      0.00      0.00\r\n10:41:20 PM     all      1.04      0.00      0.00     97.92      1.04      0.00\r\n10:41:21 PM     all      1.01      0.00      1.01     97.98      0.00      0.00\r\n10:41:22 PM     all      4.00      0.00      1.00     95.00      0.00      0.00\r\n```\r\n\r\nsar -w 1:\r\n```\r\nubuntu@ip-172-31-18-121:~$ sar -w 1\r\nLinux 3.13.0-48-generic (ip-172-31-18-121) \t09/13/2015 \t_x86_64_\t(1 CPU)\r\n\r\n10:41:12 PM    proc/s   cswch/s\r\n10:41:13 PM      0.00   1792.78\r\n10:41:14 PM      0.00    725.51\r\n10:41:15 PM      0.00   1597.98\r\n10:41:16 PM      0.00   1894.06\r\n10:41:17 PM      0.00   2175.49\r\n10:41:18 PM      0.00   1243.43\r\n10:41:19 PM      0.00   1517.82\r\n10:41:20 PM      0.00   1563.92\r\n10:41:21 PM      0.00   1329.00\r\n10:41:22 PM      0.00   1420.20\r\n10:41:23 PM      0.00   1267.68\r\n10:41:24 PM      0.00   1163.00\r\n10:41:25 PM      0.00    987.13\r\n10:41:26 PM      0.00   1247.47\r\n10:41:27 PM      0.00   1288.66\r\n```\r\n\r\niotop single process: \r\n```\r\nTotal DISK READ :     407.56 K/s | Total DISK WRITE :      59.35 K/s\r\nActual DISK READ:     407.56 K/s | Actual DISK WRITE:     265.11 K/s\r\n  PID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND\r\n 1133 be/4 rethinkd  407.56 K/s   59.35 K/s  0.00 % 23.98 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init\r\n    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]\r\n    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]\r\n    5 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H]\r\n    6 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/u30:0]\r\n```\r\n\r\niotop threads: \r\n```\r\nTotal DISK READ :    1024.98 K/s | Total DISK WRITE :     161.63 K/s\r\nActual DISK READ:    1024.98 K/s | Actual DISK WRITE:     701.72 K/s\r\n  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND\r\n 1157 be/4 rethinkd   39.42 K/s  102.50 K/s  0.00 % 76.82 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1158 be/4 rethinkd   23.65 K/s    0.00 B/s  0.00 % 52.24 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1153 be/4 rethinkd   59.13 K/s    3.94 K/s  0.00 % 51.23 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1142 be/4 rethinkd   19.71 K/s    0.00 B/s  0.00 % 46.86 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1188 be/4 rethinkd   15.77 K/s    0.00 B/s  0.00 % 46.83 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1201 be/4 rethinkd   15.77 K/s    3.94 K/s  0.00 % 46.80 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1172 be/4 rethinkd  402.11 K/s   11.83 K/s  0.00 % 45.95 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1166 be/4 rethinkd   11.83 K/s    0.00 B/s  0.00 % 43.95 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1204 be/4 rethinkd   86.73 K/s    0.00 B/s  0.00 % 43.37 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1165 be/4 rethinkd   11.83 K/s    0.00 B/s  0.00 % 42.99 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1151 be/4 rethinkd    7.88 K/s    0.00 B/s  0.00 % 42.02 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1141 be/4 rethinkd   11.83 K/s    0.00 B/s  0.00 % 41.02 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1160 be/4 rethinkd   11.83 K/s    3.94 K/s  0.00 % 40.04 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1170 be/4 rethinkd   11.83 K/s    0.00 B/s  0.00 % 39.06 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1182 be/4 rethinkd    7.88 K/s    3.94 K/s  0.00 % 38.07 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1176 be/4 rethinkd   31.54 K/s    0.00 B/s  0.00 % 37.04 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1196 be/4 rethinkd   15.77 K/s    0.00 B/s  0.00 % 36.92 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1147 be/4 rethinkd   15.77 K/s    0.00 B/s  0.00 % 36.07 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1168 be/4 rethinkd   27.60 K/s    0.00 B/s  0.00 % 32.54 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1164 be/4 rethinkd   51.25 K/s    3.94 K/s  0.00 % 29.55 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1189 be/4 rethinkd    7.88 K/s    3.94 K/s  0.00 % 23.65 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1185 be/4 rethinkd    7.88 K/s    0.00 B/s  0.00 % 22.64 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1154 be/4 rethinkd    3.94 K/s    0.00 B/s  0.00 % 21.66 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1163 be/4 rethinkd    3.94 K/s    0.00 B/s  0.00 % 20.67 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1171 be/4 rethinkd    3.94 K/s    0.00 B/s  0.00 % 19.69 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1187 be/4 rethinkd    0.00 B/s    3.94 K/s  0.00 % 18.72 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1190 be/4 rethinkd    3.94 K/s    0.00 B/s  0.00 % 18.70 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1186 be/4 rethinkd    3.94 K/s    3.94 K/s  0.00 % 17.80 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1180 be/4 rethinkd   11.83 K/s    0.00 B/s  0.00 % 17.75 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1162 be/4 rethinkd   11.83 K/s    0.00 B/s  0.00 % 17.01 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1150 be/4 rethinkd    7.88 K/s    0.00 B/s  0.00 % 16.84 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1144 be/4 rethinkd   11.83 K/s    0.00 B/s  0.00 % 15.80 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1198 be/4 rethinkd    7.88 K/s    3.94 K/s  0.00 % 15.76 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1184 be/4 rethinkd    0.00 B/s    0.00 B/s  0.00 % 15.35 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n  205 be/3 root        0.00 B/s    0.00 B/s  0.00 % 15.27 % [jbd2/xvda1-8]\r\n 1156 be/4 rethinkd    7.88 K/s    0.00 B/s  0.00 % 14.79 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1175 be/4 rethinkd    3.94 K/s    3.94 K/s  0.00 % 14.78 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1183 be/4 rethinkd    0.00 B/s    0.00 B/s  0.00 % 14.41 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1152 be/4 rethinkd    0.00 B/s    0.00 B/s  0.00 % 14.30 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1159 be/4 rethinkd    0.00 B/s    0.00 B/s  0.00 % 14.30 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1174 be/4 rethinkd    3.94 K/s    3.94 K/s  0.00 % 13.79 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1194 be/4 rethinkd    7.88 K/s    3.94 K/s  0.00 % 13.73 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1155 be/4 rethinkd    7.88 K/s    0.00 B/s  0.00 % 12.81 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1149 be/4 rethinkd   15.77 K/s    0.00 B/s  0.00 % 12.75 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n 1146 be/4 rethinkd    7.88 K/s    0.00 B/s  0.00 % 11.83 % rethinkdb --daemon --config-file /etc/rethinkdb/instances.d/instance1.c~thinkdb/instance1/pid_file --directory /var/lib/rethinkdb/instance1/data\r\n```\r\n\r\nNot sure what the issue is, after doing research about this. Some other notes:\r\n* DB is about 12GB, taking up 75% of disk. \r\n* only running on this one AWS server, not sharded\r\n* RethinkDB is the only real application running on this server\r\n* RethinkDB has 768MB memory\r\n* An index is currently being rebuilt on a large (7M document) table, but this was happening before that was started\r\n* The server is not under a lot of load, avg 200 reads per second, 50 writes per second (only while index being rebuilt)"}}