{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3976","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3976/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3976/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3976/events","html_url":"https://github.com/zfsonlinux/zfs/issues/3976","id":114487629,"number":3976,"title":"High CPU usage by \"z_fr_iss\" after deleting large files","user":{"login":"blind-oracle","id":592248,"avatar_url":"https://avatars.githubusercontent.com/u/592248?v=3","gravatar_id":"","url":"https://api.github.com/users/blind-oracle","html_url":"https://github.com/blind-oracle","followers_url":"https://api.github.com/users/blind-oracle/followers","following_url":"https://api.github.com/users/blind-oracle/following{/other_user}","gists_url":"https://api.github.com/users/blind-oracle/gists{/gist_id}","starred_url":"https://api.github.com/users/blind-oracle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/blind-oracle/subscriptions","organizations_url":"https://api.github.com/users/blind-oracle/orgs","repos_url":"https://api.github.com/users/blind-oracle/repos","events_url":"https://api.github.com/users/blind-oracle/events{/privacy}","received_events_url":"https://api.github.com/users/blind-oracle/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Performance","name":"Performance","color":"DDDDDD"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-11-01T19:34:02Z","updated_at":"2015-11-30T03:12:10Z","closed_at":null,"body":"Problem: after deleting several large files (~50Gb summary) ZFS goes crazy on CPU usage.\r\n\r\nFor about a minute it hogs *all* cores (it's 2 x 6 core E5-2620 with HT, 24 CPUs total) to 100% with about 48 running \"z_fr_iss_X\" threads (or more, hard to say).\r\n\r\nThen, when they are done, txg_sync goes for some time using 30% of one core, but that's Ok i presume.\r\n\r\nWithout PREEMPT kernel this literally blocks all userspace activity for this period of time.\r\nI can switch to PREEMPT as it's now supported by ZFS, but i'd rather not to.\r\n\r\nKernel 4.1.12, ZoL 0.6.5.3, 256Gb RAM.\r\n\r\nZFS created with:\r\n```\r\nzpool create zfs1 \\\r\n-o feature@async_destroy=enabled \\\r\n-o feature@empty_bpobj=enabled \\\r\n-o feature@lz4_compress=enabled \\\r\n-o feature@spacemap_histogram=enabled \\\r\n-o feature@enabled_txg=enabled \\\r\n-o feature@embedded_data=enabled \\\r\n-o feature@hole_birth=enabled \\\r\n-o feature@extensible_dataset=enabled \\\r\n-o ashift=9 \\\r\n-O recordsize=128k \\\r\n-O checksum=sha256 \\\r\n-O dedup=sha256 \\\r\n-O compression=lz4 \\\r\n-O atime=off \\\r\n-O relatime=off \\\r\n-O logbias=latency \\\r\n-O primarycache=all \\\r\n-O secondarycache=none \\\r\nraidz2 \\\r\n/dev/mapper/D01_9Y0RA \\\r\n/dev/mapper/D02_B9G2A \\\r\n/dev/mapper/D03_B01GA \\\r\n/dev/mapper/D04_B9GDA \\\r\n/dev/mapper/D05_AU1HA \\\r\n/dev/mapper/D06_BD4HA \\\r\n/dev/mapper/D07_7P85D \\\r\n/dev/mapper/D08_93WJA \\\r\n/dev/mapper/D09_4394K \\\r\n/dev/mapper/D10_BDYKA \\\r\nlog mirror \\\r\n/dev/mapper/SSD1_SLOG \\\r\n/dev/mapper/SSD2_SLOG \\\r\n/dev/mapper/SSD3_SLOG\r\n```\r\nPool size is ~21Tb and it's empty but for these test files.\r\nIf there's any more info required - ask."},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/160505795","html_url":"https://github.com/zfsonlinux/zfs/issues/3976#issuecomment-160505795","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/3976","id":160505795,"user":{"login":"dweeezil","id":1432005,"avatar_url":"https://avatars.githubusercontent.com/u/1432005?v=3","gravatar_id":"","url":"https://api.github.com/users/dweeezil","html_url":"https://github.com/dweeezil","followers_url":"https://api.github.com/users/dweeezil/followers","following_url":"https://api.github.com/users/dweeezil/following{/other_user}","gists_url":"https://api.github.com/users/dweeezil/gists{/gist_id}","starred_url":"https://api.github.com/users/dweeezil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dweeezil/subscriptions","organizations_url":"https://api.github.com/users/dweeezil/orgs","repos_url":"https://api.github.com/users/dweeezil/repos","events_url":"https://api.github.com/users/dweeezil/events{/privacy}","received_events_url":"https://api.github.com/users/dweeezil/received_events","type":"User","site_admin":false},"created_at":"2015-11-30T03:12:10Z","updated_at":"2015-11-30T03:12:10Z","body":"@tuxoko Interesting idea.  At the very least, it looks like it might help the ```dbuf_prefetch()``` case.  For the moment, however, I'm a lot more concerned about contention caused by the fanout of the various zio taskqs; and the write issue taskq in particular.  We've got a lot more locking going on than illumos and contention on the single taskq can be a big point of contention for high-concurrency write workloads."}}