{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2746","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2746/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2746/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2746/events","html_url":"https://github.com/zfsonlinux/zfs/issues/2746","id":44294022,"number":2746,"title":"zfs send/recv VERY slow for large files (>5GB)","user":{"login":"lintonv","id":8958894,"avatar_url":"https://avatars.githubusercontent.com/u/8958894?v=3","gravatar_id":"","url":"https://api.github.com/users/lintonv","html_url":"https://github.com/lintonv","followers_url":"https://api.github.com/users/lintonv/followers","following_url":"https://api.github.com/users/lintonv/following{/other_user}","gists_url":"https://api.github.com/users/lintonv/gists{/gist_id}","starred_url":"https://api.github.com/users/lintonv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lintonv/subscriptions","organizations_url":"https://api.github.com/users/lintonv/orgs","repos_url":"https://api.github.com/users/lintonv/repos","events_url":"https://api.github.com/users/lintonv/events{/privacy}","received_events_url":"https://api.github.com/users/lintonv/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Difficulty%20-%20Medium","name":"Difficulty - Medium","color":"207de5"},{"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Performance","name":"Performance","color":"DDDDDD"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":26,"created_at":"2014-09-29T13:40:14Z","updated_at":"2015-04-20T13:42:15Z","closed_at":null,"body":"I have been using ZFS SEND/RECV over SSH. The speeds are excellent for small files. But for large files (>2GB), it is terrible. \r\n\r\n- File Size less than 5 GB, the speed : 112MB/s (which is excellent and using the full speed of the interface)\r\n- File Size greater than 5 GB, the speed is 10MB/s (very, very slow)\r\n\r\nInitially, I thought the bottleneck was SSH and the MTU. But despite using jumbo frames (9000 MTU) and using other transfer tools like netcat, there is no change in the transfer speeds. \r\n\r\nThe bottleneck has to be ZFS. After reading the code in zfs_log.c, I was convinced that zfs_immediate_write_sz was the problem. There is a hard coded limit of 4K in there. But even after tweaking that value, I still saw no performance gain. \r\n"},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/94454148","html_url":"https://github.com/zfsonlinux/zfs/issues/2746#issuecomment-94454148","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2746","id":94454148,"user":{"login":"lintonv","id":8958894,"avatar_url":"https://avatars.githubusercontent.com/u/8958894?v=3","gravatar_id":"","url":"https://api.github.com/users/lintonv","html_url":"https://github.com/lintonv","followers_url":"https://api.github.com/users/lintonv/followers","following_url":"https://api.github.com/users/lintonv/following{/other_user}","gists_url":"https://api.github.com/users/lintonv/gists{/gist_id}","starred_url":"https://api.github.com/users/lintonv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lintonv/subscriptions","organizations_url":"https://api.github.com/users/lintonv/orgs","repos_url":"https://api.github.com/users/lintonv/repos","events_url":"https://api.github.com/users/lintonv/events{/privacy}","received_events_url":"https://api.github.com/users/lintonv/received_events","type":"User","site_admin":false},"created_at":"2015-04-20T13:42:15Z","updated_at":"2015-04-20T13:42:15Z","body":"The root cause is as identified in the above comment. \r\n\r\nBut I found a way around the slow ZFS send performance. We increased the number of devices presented in the zpool by striping the disks. We use 8 SSDs and created 4 partitions for each. Therefore, zpool sees a total of 32 devices. This reduces the impact of the highly-serial ZFS send operation, as multiple threads are created in parallel (for each device) and the stream is build quicker. \r\n\r\nWith this, we are saturating our 1GIG link consistently. \r\n\r\n@behlendorf Do you want me to close this issue now? "}}