{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3303","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3303/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3303/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3303/events","html_url":"https://github.com/zfsonlinux/zfs/issues/3303","id":68784650,"number":3303,"title":"arc_adapt left spinning after rsync with lots of small files","user":{"login":"angstymeat","id":6665497,"avatar_url":"https://avatars.githubusercontent.com/u/6665497?v=3","gravatar_id":"","url":"https://api.github.com/users/angstymeat","html_url":"https://github.com/angstymeat","followers_url":"https://api.github.com/users/angstymeat/followers","following_url":"https://api.github.com/users/angstymeat/following{/other_user}","gists_url":"https://api.github.com/users/angstymeat/gists{/gist_id}","starred_url":"https://api.github.com/users/angstymeat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/angstymeat/subscriptions","organizations_url":"https://api.github.com/users/angstymeat/orgs","repos_url":"https://api.github.com/users/angstymeat/repos","events_url":"https://api.github.com/users/angstymeat/events{/privacy}","received_events_url":"https://api.github.com/users/angstymeat/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-04-15T20:52:26Z","updated_at":"2015-04-16T14:54:08Z","closed_at":null,"body":"This is a split off of #3235. The symptom is that after a large set up rsyncs containing lots of small files, I'm left with arc_adapt spinning at around 3.3% CPU time.\r\n\r\nI've compiled the info that @dweeezil wanted here: https://cloud.passcal.nmt.edu/index.php/s/l52T2UhZ0K7taY9.\r\n\r\nThe system is a Dell R515 with 16GB of RAM. The pool is a single raidz2 pool made of of 7 2TB SATA drives. Compression is enabled and set to lz4. atime is off.\r\n\r\nThe OS is Fedora 20 with the 3.17.8-200.fc20.x86_64 kernel.\r\n\r\nThe machine this happens on is an offsite server that hosts our offsite backups. There are 20+ rsync processes running that send large and small files from our internal systems to this system. The majority of the files are either large data files or the files you would typically find as part of a linux installation (/usr, /bin, /var, etc.)\r\n\r\nAlso, about 50 home directories are backed up containing a mix of large and small files.\r\n\r\nThis takes about 45 minutes. One hour after these jobs are started, the email servers begin their backup (so usually a 15 minute delay between the start of one set of backups and another). Also, our large data collector sends its backups at this time. These are large files, but a lot of them. It is sometime during this 2nd stage backup that this issue occurs.\r\n\r\nDuring the backup, I have another process that periodically runs an ```echo 2 > /proc/sys/vm/drop_caches```. I do this because once the ARC gets filled up performance drops drastically. Without doing this, it will take up to 10 hours to perform a backup. With it, it will take less than 2 hours.\r\n\r\nThis happens even if I do not run the periodic drop_caches, but seems to occur less often.\r\n\r\nThe drop_caches is a relatively new addition to my scripting on this system as this didn't appear to happen under 0.6.3. I don't have a good idea when it started, but I'm pretty sure it was sometime around the kmem rework on 0.6.4.\r\n\r\nI am unable to rollback and test under 0.6.3 because while I was testing 0.6.4, the new feature flags were enabled. This unit is not a critical system, so I have quite a bit of leeway with it, as long as I don't have to destroy and recreate the pool. I usually use it to test new versions of ZFS since it gets a lot of disk usage.\r\n"},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/93754831","html_url":"https://github.com/zfsonlinux/zfs/issues/3303#issuecomment-93754831","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3303","id":93754831,"user":{"login":"dweeezil","id":1432005,"avatar_url":"https://avatars.githubusercontent.com/u/1432005?v=3","gravatar_id":"","url":"https://api.github.com/users/dweeezil","html_url":"https://github.com/dweeezil","followers_url":"https://api.github.com/users/dweeezil/followers","following_url":"https://api.github.com/users/dweeezil/following{/other_user}","gists_url":"https://api.github.com/users/dweeezil/gists{/gist_id}","starred_url":"https://api.github.com/users/dweeezil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dweeezil/subscriptions","organizations_url":"https://api.github.com/users/dweeezil/orgs","repos_url":"https://api.github.com/users/dweeezil/repos","events_url":"https://api.github.com/users/dweeezil/events{/privacy}","received_events_url":"https://api.github.com/users/dweeezil/received_events","type":"User","site_admin":false},"created_at":"2015-04-16T14:54:08Z","updated_at":"2015-04-16T14:54:08Z","body":"And to make matters even more fun, torvalds/linux@4101b62 changed it yet again (post 4.0).  This issue also applies to the free_cached_objects callback.  I'm working on an enhanced autoconf test."}}