{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3303","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3303/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3303/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3303/events","html_url":"https://github.com/zfsonlinux/zfs/issues/3303","id":68784650,"number":3303,"title":"arc_adapt left spinning after rsync with lots of small files","user":{"login":"angstymeat","id":6665497,"avatar_url":"https://avatars.githubusercontent.com/u/6665497?v=3","gravatar_id":"","url":"https://api.github.com/users/angstymeat","html_url":"https://github.com/angstymeat","followers_url":"https://api.github.com/users/angstymeat/followers","following_url":"https://api.github.com/users/angstymeat/following{/other_user}","gists_url":"https://api.github.com/users/angstymeat/gists{/gist_id}","starred_url":"https://api.github.com/users/angstymeat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/angstymeat/subscriptions","organizations_url":"https://api.github.com/users/angstymeat/orgs","repos_url":"https://api.github.com/users/angstymeat/repos","events_url":"https://api.github.com/users/angstymeat/events{/privacy}","received_events_url":"https://api.github.com/users/angstymeat/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":52,"created_at":"2015-04-15T20:52:26Z","updated_at":"2015-06-13T21:10:17Z","closed_at":null,"body":"This is a split off of #3235. The symptom is that after a large set up rsyncs containing lots of small files, I'm left with arc_adapt spinning at around 3.3% CPU time.\r\n\r\nI've compiled the info that @dweeezil wanted here: https://cloud.passcal.nmt.edu/index.php/s/l52T2UhZ0K7taY9.\r\n\r\nThe system is a Dell R515 with 16GB of RAM. The pool is a single raidz2 pool made of of 7 2TB SATA drives. Compression is enabled and set to lz4. atime is off.\r\n\r\nThe OS is Fedora 20 with the 3.17.8-200.fc20.x86_64 kernel.\r\n\r\nThe machine this happens on is an offsite server that hosts our offsite backups. There are 20+ rsync processes running that send large and small files from our internal systems to this system. The majority of the files are either large data files or the files you would typically find as part of a linux installation (/usr, /bin, /var, etc.)\r\n\r\nAlso, about 50 home directories are backed up containing a mix of large and small files.\r\n\r\nThis takes about 45 minutes. One hour after these jobs are started, the email servers begin their backup (so usually a 15 minute delay between the start of one set of backups and another). Also, our large data collector sends its backups at this time. These are large files, but a lot of them. It is sometime during this 2nd stage backup that this issue occurs.\r\n\r\nDuring the backup, I have another process that periodically runs an ```echo 2 > /proc/sys/vm/drop_caches```. I do this because once the ARC gets filled up performance drops drastically. Without doing this, it will take up to 10 hours to perform a backup. With it, it will take less than 2 hours.\r\n\r\nThis happens even if I do not run the periodic drop_caches, but seems to occur less often.\r\n\r\nThe drop_caches is a relatively new addition to my scripting on this system as this didn't appear to happen under 0.6.3. I don't have a good idea when it started, but I'm pretty sure it was sometime around the kmem rework on 0.6.4.\r\n\r\nI am unable to rollback and test under 0.6.3 because while I was testing 0.6.4, the new feature flags were enabled. This unit is not a critical system, so I have quite a bit of leeway with it, as long as I don't have to destroy and recreate the pool. I usually use it to test new versions of ZFS since it gets a lot of disk usage.\r\n"},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/111752784","html_url":"https://github.com/zfsonlinux/zfs/issues/3303#issuecomment-111752784","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3303","id":111752784,"user":{"login":"angstymeat","id":6665497,"avatar_url":"https://avatars.githubusercontent.com/u/6665497?v=3","gravatar_id":"","url":"https://api.github.com/users/angstymeat","html_url":"https://github.com/angstymeat","followers_url":"https://api.github.com/users/angstymeat/followers","following_url":"https://api.github.com/users/angstymeat/following{/other_user}","gists_url":"https://api.github.com/users/angstymeat/gists{/gist_id}","starred_url":"https://api.github.com/users/angstymeat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/angstymeat/subscriptions","organizations_url":"https://api.github.com/users/angstymeat/orgs","repos_url":"https://api.github.com/users/angstymeat/repos","events_url":"https://api.github.com/users/angstymeat/events{/privacy}","received_events_url":"https://api.github.com/users/angstymeat/received_events","type":"User","site_admin":false},"created_at":"2015-06-13T21:10:17Z","updated_at":"2015-06-13T21:10:17Z","body":"No snapshots are mounted and grep'ing /proc/mounts returns nothing. I have a lot of files, but not a lot of file systems. Here's zfs list:\r\n```\r\nstorage                         7.14T  1.01T   368K  /storage\r\nstorage/dbserve                 29.6G  1.01T  4.62G  /storage/dbserve\r\nstorage/fs                      1.05T  1.01T   320K  /storage/fs\r\nstorage/fs/TA                   1.04T  1.01T  1.04T  /storage/fs/TA\r\nstorage/fs/office               4.90G  1.01T  4.80G  /storage/fs/office\r\nstorage/fs/shipping             3.55G  1.01T  3.53G  /storage/fs/shipping\r\nstorage/home                     182G  1.01T   153G  /storage/home\r\nstorage/local-backup            5.73G  1.01T  1.90G  /storage/local-backup\r\nstorage/maildir                  291G  1.01T   274G  /storage/maildir\r\nstorage/maildir_old              278G  1.01T   267G  /storage/maildir_old\r\nstorage/qc-db                   49.0G  1.01T   288K  /storage/qc-db\r\nstorage/qc-db/database          49.0G  1.01T   336K  /storage/qc-db/database\r\nstorage/qc-db/database/bin-log   288K  1.01T   272K  /storage/qc-db/database/bin-log\r\nstorage/qc-db/database/data     48.5G  1.01T  48.5G  /storage/qc-db/database/data\r\nstorage/qc-db/database/innodb    451M  1.01T   371M  /storage/qc-db/database/innodb\r\nstorage/qc-db/database/logs     1015K  1.01T   823K  /storage/qc-db/database/logs\r\nstorage/svn                     2.12G  1.01T  2.07G  /storage/svn\r\nstorage/systems                 4.65T  1.01T  6.18G  /storage/systems\r\nstorage/systems/bedrock         5.18G  1.01T  4.81G  /storage/systems/bedrock\r\nstorage/systems/cloud            476M  1.01T   350M  /storage/systems/cloud\r\nstorage/systems/durga           3.30T  1.01T   288K  /storage/systems/durga\r\nstorage/systems/durga/data      3.30T  1.01T  3.07T  /storage/systems/durga/data\r\nstorage/systems/glisn           1.01T  1.01T   927G  /storage/systems/glisn\r\nstorage/systems/local-apps      16.4G  1.01T  11.2G  /storage/systems/local-apps\r\nstorage/systems/mail-dns        39.6G  1.01T  30.1G  /storage/systems/mail-dns\r\nstorage/systems/matlab          6.29G  1.01T  5.71G  /storage/systems/matlab\r\nstorage/systems/qandisa          103G  1.01T   102G  /storage/systems/qandisa\r\nstorage/systems/quartz           760M  1.01T   717M  /storage/systems/quartz\r\nstorage/systems/support         10.7G  1.01T  9.52G  /storage/systems/support\r\nstorage/systems/tiamat           121G  1.01T   121G  /storage/systems/tiamat\r\nstorage/systems/www             10.9G  1.01T  10.3G  /storage/systems/www\r\nstorage/systems/xeos            33.1G  1.01T  30.9G  /storage/systems/xeos\r\nstorage/systems/xion             192K  1.01T   192K  /storage/systems/xion\r\nstorage/vmware                   529G  1.01T   529G  /storage/vmware\r\nstorage/xeos-db                  101G  1.01T  39.9G  /storage/xeos-db\r\n```\r\n\r\nEach filesystem has about 18 snapshots, but they're only used when we need to recover something."}}