{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/1538","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/1538/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/1538/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/1538/events","html_url":"https://github.com/zfsonlinux/zfs/issues/1538","id":15794040,"number":1538,"title":"txg_sync and z_rd_int/* become top I/O consumers, the host renders unusable for a period of time","user":{"login":"alexclear","id":857605,"avatar_url":"https://avatars.githubusercontent.com/u/857605?v=3","gravatar_id":"","url":"https://api.github.com/users/alexclear","html_url":"https://github.com/alexclear","followers_url":"https://api.github.com/users/alexclear/followers","following_url":"https://api.github.com/users/alexclear/following{/other_user}","gists_url":"https://api.github.com/users/alexclear/gists{/gist_id}","starred_url":"https://api.github.com/users/alexclear/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexclear/subscriptions","organizations_url":"https://api.github.com/users/alexclear/orgs","repos_url":"https://api.github.com/users/alexclear/repos","events_url":"https://api.github.com/users/alexclear/events{/privacy}","received_events_url":"https://api.github.com/users/alexclear/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Difficulty%20-%20Medium","name":"Difficulty - Medium","color":"207de5"},{"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Performance","name":"Performance","color":"DDDDDD"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2013-06-20T11:24:53Z","updated_at":"2015-11-19T23:15:46Z","closed_at":null,"body":"I have a dual Xeon E5645 system w/8 SATA disks, /boot is on an USB drive and / is on ZFS:\r\n\r\n    root@debhost1 ~ zfs list\r\n    NAME                                 USED  AVAIL  REFER  MOUNTPOINT\r\n    vol0                                2.50T  1.96T   350M  /vol0\r\n    vol0/kvm-Common-Host                 775G  1.96T   186G  /var/lib/libvirt/images/Common-Host\r\n    vol0/kvm-Dev6-Host                  9.28G  1.96T  9.28G  /var/lib/libvirt/images/Dev6-Host\r\n    vol0/kvm-Dev7-Host                  9.25G  1.96T  9.25G  /var/lib/libvirt/images/Dev7-Host\r\n    vol0/kvm-HBase-Slave-1              99.1G  1.96T  23.1G  /var/lib/libvirt/images/HBase-Slave-1\r\n    vol0/kvm-HBase-Slave-2               107G  1.96T  23.8G  /var/lib/libvirt/images/HBase-Slave-2\r\n    vol0/kvm-W2K8                        577G  1.96T   184G  /var/lib/libvirt/images/W2K8\r\n    vol0/libvirt-images                  130G  1.96T    31K  /vol0/libvirt-images\r\n    vol0/libvirt-images/Alex-Client     23.2G  1.96T  9.74G  /var/lib/libvirt/images/Alex-Client\r\n    vol0/libvirt-images/Dev3-Host       27.1G  1.96T  27.1G  /var/lib/libvirt/images/Dev3-Host\r\n    vol0/libvirt-images/Dev4-Host       23.6G  1.96T  23.6G  /var/lib/libvirt/images/Dev4-Host\r\n    vol0/libvirt-images/Dev5-Host       20.2G  1.96T  20.2G  /var/lib/libvirt/images/Dev5-Host\r\n    vol0/libvirt-images/Dev7-Host       91.2M  1.96T  91.2M  /vol0/libvirt-images/Dev7-Host\r\n    vol0/libvirt-images/Dev8-Host       2.73G  1.96T  2.73G  /var/lib/libvirt/images/Dev8-Host\r\n    vol0/libvirt-images/HBase-Master    9.65G  1.96T  5.23G  /var/lib/libvirt/images/HBase-Master\r\n    vol0/libvirt-images/W7-01d          9.97G  1.96T  9.97G  /var/lib/libvirt/images/W7-01d\r\n    vol0/libvirt-images/W7-Scanner-New  12.9G  1.96T  12.9G  /var/lib/libvirt/images/W7-Scanner-New\r\n    vol0/rootfs                          260G  1.96T   189G  /\r\n    vol0/var                             579G  1.96T   579G  /var2\r\n    root@debhost1 ~ \r\n\r\nThe zpool is:\r\n\r\n    root@debhost1 ~ zpool status\r\n      pool: vol0\r\n     state: ONLINE\r\n      scan: scrub repaired 0 in 44h43m with 0 errors on Mon May 13 00:06:19 2013\r\n    config:\r\n    \r\n    \tNAME                                           STATE     READ WRITE CKSUM\r\n    \tvol0                                           ONLINE       0     0     0\r\n    \t  mirror-0                                     ONLINE       0     0     0\r\n    \t    scsi-SATA_WDC_WD1003FBYX-_WD-WMAW30418757  ONLINE       0     0     0\r\n    \t    scsi-SATA_WDC_WD1003FBYX-_WD-WMAW31056873  ONLINE       0     0     0\r\n    \t  mirror-1                                     ONLINE       0     0     0\r\n    \t    scsi-SATA_WDC_WD1003FBYX-_WD-WMAW31066713  ONLINE       0     0     0\r\n    \t    scsi-SATA_WDC_WD1003FBYX-_WD-WCAW33977049  ONLINE       0     0     0\r\n    \t  mirror-2                                     ONLINE       0     0     0\r\n    \t    scsi-SATA_WDC_WD1003FBYX-_WD-WMAW31077083  ONLINE       0     0     0\r\n    \t    scsi-SATA_WDC_WD1003FBYX-_WD-WCAW35326792  ONLINE       0     0     0\r\n    \t  mirror-3                                     ONLINE       0     0     0\r\n    \t    scsi-SATA_WDC_WD2000FYYZ-_WD-WCC1P0020205  ONLINE       0     0     0\r\n    \t    scsi-SATA_WDC_WD2000FYYZ-_WD-WCC1P0021665  ONLINE       0     0     0\r\n    \tcache\r\n    \t  ata-OCZ-AGILITY4_7M7U6Y6X7E6Q3K2D3E3U-part1  ONLINE       0     0     0\r\n    \r\n    errors: No known data errors\r\n    root@debhost1 ~ \r\n\r\nThis is a Debian 7.0-based system w/ZoL 0.6.1-1:\r\n\r\n    root@debhost1 ~ uname -a\r\n    Linux debhost1 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2 x86_64 GNU/Linux\r\n\r\nIt is used mainly for virtualization, most datasets have sync=disabled and recordsize=4K\r\nARC is capped by 20G, ARC utilization graph follows:\r\n\r\n![zfs_stats_utilization-day](https://f.cloud.github.com/assets/857605/680722/94abdae2-d999-11e2-9c29-1adfc691e2eb.png)\r\n\r\nL2ARC size and efficiency graphs follow:\r\n\r\n![zfs_stats_l2utilization-day](https://f.cloud.github.com/assets/857605/680725/be14b048-d999-11e2-8e07-e5a3dcf7bede.png)\r\n![zfs_stats_l2efficiency-day](https://f.cloud.github.com/assets/857605/680727/c2559dc0-d999-11e2-9c27-a4544ddbe5e5.png)\r\n\r\nDisk load is almost constant during working hours but we suffer from sporadical performance drops which do not seem to correlate to actual load generated by virtual machines. Or at least we were unable to find the correlation yet.\r\nOutput of iotop -d 5 -P looks like this during these performance drop periods:\r\n\r\n![screenshot-txg_sync](https://f.cloud.github.com/assets/857605/680752/9b83ff6a-d99a-11e2-8b8a-d72b02d6c12c.png)\r\n\r\nWe also collected iostat -x -k 2 20 output here: https://gist.github.com/alexclear/5821955 and zpool iostat -v 2 20 (at the same time) output here: https://gist.github.com/alexclear/5821960\r\n\r\nIt seems like the system experiences a lot of internal I/O (mostly random reads) not consumed directly by userspace processes during these periods. It is also very odd that L2ARC efficiency remains quite low."},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/158230426","html_url":"https://github.com/zfsonlinux/zfs/issues/1538#issuecomment-158230426","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/1538","id":158230426,"user":{"login":"behlendorf","id":148917,"avatar_url":"https://avatars.githubusercontent.com/u/148917?v=3","gravatar_id":"","url":"https://api.github.com/users/behlendorf","html_url":"https://github.com/behlendorf","followers_url":"https://api.github.com/users/behlendorf/followers","following_url":"https://api.github.com/users/behlendorf/following{/other_user}","gists_url":"https://api.github.com/users/behlendorf/gists{/gist_id}","starred_url":"https://api.github.com/users/behlendorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/behlendorf/subscriptions","organizations_url":"https://api.github.com/users/behlendorf/orgs","repos_url":"https://api.github.com/users/behlendorf/repos","events_url":"https://api.github.com/users/behlendorf/events{/privacy}","received_events_url":"https://api.github.com/users/behlendorf/received_events","type":"User","site_admin":false},"created_at":"2015-11-19T23:15:45Z","updated_at":"2015-11-19T23:15:45Z","body":"@pdf this may be a result of the change in scheduling priority for the ZFS threads, 1229323d5f82bcb0525b312c80330968cd41466e and an increase in their number aa9af22cdf8d16c197974c3a478d2053b3bed498."}}