why is virtio-scsi much slower than virtio-blk in my experiment (over and ceph rbd image)?
<p>Hi I recently did a experiment of virtio-scsi over rbd through qemu target (for its DISCARD/TRIM support), and compared the throughput and iops with that of a virtio-blk over rbd setup on the same machine, using fio in the guest. Turnout the throughput in sequential read write is 7 times smaller (42.3MB/s vs 309MB/s) and the iops in random read write is 10 times smaller (546 vs 5705).</p>

<p>What I did is setting up a virtual machine using OpenStack Juno, which give me the virtio-blk over rbd setup. Then I modified the relevant part in libvirt configure xml, from this:</p>

<pre><code>&lt;disk type='network' device='disk'&gt;
  &lt;driver name='qemu' type='raw' cache='writeback'/&gt;
  &lt;auth username='cinder'&gt;
    &lt;secret type='ceph' uuid='482b83f9-be95-448e-87cc-9fa602196590'/&gt;
  &lt;/auth&gt;
  &lt;source protocol='rbd' name='vms/c504ea8b-18e6-491e-9470-41c60aa50b81_disk'&gt;
    &lt;host name='192.168.20.105' port='6789'/&gt;
  &lt;/source&gt;
  &lt;target dev='vda' bus='virtio'/&gt;
  &lt;address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/&gt;
&lt;/disk&gt;
</code></pre>

<p>to this:</p>

<pre><code>&lt;disk type='network' device='disk'&gt;
  &lt;driver name='qemu' type='raw' cache='writeback' discard='unmap'/&gt;
  &lt;auth username='cinder'&gt;
    &lt;secret type='ceph' uuid='482b83f9-be95-448e-87cc-9fa602196590'/&gt;
  &lt;/auth&gt;
  &lt;source protocol='rbd' name='vms/c504ea8b-18e6-491e-9470-41c60aa50b81_disk'&gt;
    &lt;host name='192.168.20.105' port='6789'/&gt;
  &lt;/source&gt;
  &lt;target dev='vda' bus='scsi'/&gt;
  &lt;controller type='scsi' model='virtio-scsi' index='0'/&gt;
&lt;/disk&gt;
</code></pre>

<p>The software versions are:</p>

<p><strong>qemu</strong> 2.5.1</p>

<p><strong>libvirt</strong> 1.2.2</p>

<p><strong>kernel</strong> 3.18.0-031800-generic #201412071935 SMP Mon Dec 8 00:36:34 UTC 2014 x86_64 (a Ubuntu 14.04 kernel)</p>

<p>And the hypervisor is KVM.</p>

<p>I don't think the performance difference could be that large between virtio-scsi and virtio-blk. So please point out what I did wrong, and how to achieve a reasonable performance. </p>

<p>A constraint is that I want a solution that works for OpenStack (ideal if works for Juno) without many patching or coding around. E.g., I heard of virtio-scsi + vhost-scsi + scsi-mq, but that seems not available in OpenStack right now.</p>