python libusb1: asynchronous TRANSFER_NO_DEVICE status just after successful syncronous transfers
<h1>Story</h1>

<p>I'm programming a driver for a scientific camera. It uses the Cypress FX3 usb peripheral controller. In order to communicate with it I'm using libusb1 for python, specifically the module usb1. My OS is Ubuntu 16.04 LTS.</p>

<p>The communication has two steps:</p>

<ul>
<li><p>The camera is configured. The computer synchronously sends instructions to program the camera and after each instruction the camera responds a status word, which is read synchronously.</p></li>
<li><p>A photo is taken. The computer synchronously sends a single instruction and the camera starts streaming data. The computer reads this data in a asynchronous manner.</p></li>
</ul>

<p>The asynchronous communication is done in the main thread. So even if the communication itself is asynchronous, the operation is blocking.</p>

<h1>Problem</h1>

<p>I'm getting TRANSFER_NO_DEVICE status for each asynchronous transfer, which is strange given that I have just communicated with the camera in the configuration step. I've a similar code in C# in Windows using the cypress library and it works correctly, so I can rule out the camera. Also, part of the image data appears in the FX3 buffer after trying to take a photo, which I can recover using a example application provided by cypress.</p>

<p>I've built a minimum example script. Notice the configure and take_picture functions:</p>

<pre><code>#!/usr/bin/env python
# -*- coding: UTF-8 -*-
#
# StackOverflow.py

import usb1 as usb1    # Libusb, needed to provide a usb context

import GuideCamLib.binary as binary # Handles bytecode formatting
import GuideCamLib.ccd as ccd       # Generates configuration information
import GuideCamLib.usb as usb       # Manages communication

# Camera usb parameters
vid = 0x04B4;
pid = 0x00F1;

read_addr = 0x81;
write_addr = 0x01;

interface = 0;

success = [0x55, 0x55, 0x55, 0x55] + [0]*(512 - 4); # A successful response

# Objects that generate instructions for the camera
ccd0 = ccd.CCD_47_10();
formatter = binary.ByteCode();

def configure(context):
  # Generating bytes to transfer, outputs a list of int lists
  bytecode_lines = ccd0.create_configuration_bytecode(formatter);

  # Opens device
  with usb.Device(vid=vid, pid=pid, context= context) as dev:

    # Opens read / write ports
    port_write = dev.open_port(write_addr);
    port_read = dev.open_port(read_addr);

    print(" Start configuration...")
    # Transfer synchronously
    for line in bytecode_lines:
      written_bytes = port_write.write_sync(line);
      response = port_read.read_sync(512);
      if(response != success):
        raise RuntimeError(" Configuration failed. (" + str(response) + ")");
    print(" End configuration")

def take_picture(context):
  # Generating bytes to transfer, generates a list of ints
  take_photo_bytecode = formatter.seq_take_photo(ccd0.get_take_photo_mode_address());

  # Opens device
  with usb.Device(vid=vid, pid=pid, context= context) as dev:

    # Opens read / write ports
    port_write = dev.open_port(write_addr);
    port_read = dev.open_port(read_addr, 10000); # 10 sec timeout

    # Prepare asynchronous read
    print(" Prepare read")
    with port_read.read_async(512) as data_collector:
      print(" Writing")
      written_bytes = port_write.write_sync(take_photo_bytecode); # Write synchronously
      print(" Reading...")
      recieved_image = data_collector(); # Read asynchronously (but in a blocking manner)

  print " Recieved: " + str(len(recieved_image)) + " bytes.";

with usb1.USBContext() as context:
  print "Configuring camera:"
  configure(context);      # Configure camera
  print "Taking picture:"
  take_picture(context);   # Take picture
  print "Done."
</code></pre>

<p>Here is GuideCamLib/usb.py for the needed contextualization. The class _TransferCollector does most of the work, while _AsyncReader is just a function with state. Port and Device are just helper classes, to reduce boilerplate code in each transfer:</p>

<pre><code>#!/usr/bin/env python
# -*- coding: UTF-8 -*-
#
# GuideCamLib/usb.py

import usb1 as usb
import six as six

import traceback

# For human-readable printing
transfer_status_dict = \
{ \
  usb.TRANSFER_COMPLETED : "TRANSFER_COMPLETED",
  usb.TRANSFER_ERROR     : "TRANSFER_ERROR",
  usb.TRANSFER_TIMED_OUT : "TRANSFER_TIMED_OUT",
  usb.TRANSFER_CANCELLED : "TRANSFER_CANCELLED",
  usb.TRANSFER_STALL     : "TRANSFER_STALL",
  usb.TRANSFER_NO_DEVICE : "TRANSFER_NO_DEVICE",
  usb.TRANSFER_OVERFLOW  : "TRANSFER_OVERFLOW" \
};

# Callback to accumulate succesive transfer calls
class _AsyncReader:
  def __init__(self):
    self.transfers = [];

  def __call__(self, transfer):
    print "Status: " + transfer_status_dict[transfer.getStatus()]; # Prints the status of the transfer
    if(transfer.getStatus() != usb.TRANSFER_COMPLETED):
      return;
    else:
      self.transfers.append(transfer.getBuffer()[:transfer.getActualLength()]);
      transfer.submit();

# A collector of asyncronous transfer's data.
# Stops collection after port.timeout time of recieving the last transfer.
class _TransferCollector:
  # Prepare data collection
  def __init__(self, transfer_size, pararell_transfers, port):
    self.interface_handle = port.device.dev.claimInterface(port.interface);
    self.reader = _AsyncReader();
    self.port = port;
    transfers = [];

    # Queue transfers
    for ii in range(pararell_transfers):
      transfer = port.device.dev.getTransfer();
      transfer.setBulk(
        port.address,
        transfer_size,
        callback=self.reader,
        timeout=port.timeout );
      transfer.submit();
      transfers.append(transfer);
    self.transfers = transfers;

  def __enter__(self):
    self.interface_handle.__enter__();
    return self;

  def __exit__(self, exception_type, exception_value, traceback):
    self.interface_handle.__exit__(exception_type, exception_value, traceback);

  # Activate data collection
  def __call__(self):
    # Collect tranfers with _AsyncReader while there are active transfers.
    while any(x.isSubmitted() for x in self.transfers):
      try:
        self.port.device.context.handleEvents();
      except usb.USBErrorInterrupted:
        pass;
    return [six.byte2int(d) for data in self.reader.transfers for d in data];

# Port class for creating syncronous / asyncronous transfers
class Port:
  def __init__(self, device, address, timeout = None):
    self.device = device;
    self.address = address;
    self.interface = self.device.interface;
    self.timeout = timeout;
    if(timeout is None):
      self.timeout = 0;

  def read_sync(self, length):
    with self.device.dev.claimInterface(self.interface):
      data = self.device.dev.bulkRead(self.address, length, timeout=self.timeout);
      return [six.byte2int(d) for d in data];

  def write_sync(self, data):
    data = [six.int2byte(d) for d in data];
    with self.device.dev.claimInterface(self.interface):
      return self.device.dev.bulkWrite(self.address, data, timeout=self.timeout);

  # Make asyncronous transfers blocking. Collects data as long as the device
  # sends data more frecuently than self.timeout or all the transfers fails
  def read_async(self, length, pararell_transfers = 32):
    return _TransferCollector(length, pararell_transfers, self);

# Device class for creating ports
class Device:
  def __init__(self, vid = None, pid = None, context = None, interface = 0):

    if(not context):
      self.backend = usb.USBContext();
      context = self.backend.__enter__();

    self.context = context;
    self.interface = interface;

    self.dev = context.openByVendorIDAndProductID(vid, pid, skip_on_error = False);    
    if self.dev is None:
      raise RuntimeError('Device not found');

  def __enter__(self):
    return self;

  def __exit__(self, exception_type, exception_value, traceback):
    if(hasattr(self, "backend")):
      self.backend.__exit__(exception_type, exception_value, traceback);

  def open_port(self, address, timeout = None):
    return Port(self, address, timeout);
</code></pre>

<p>The script outputs the following, which clearly shows the synchronous transfers are successful but each of the queued asynchronous transfers fail with a NO_DEVICE:</p>

<pre><code>&gt;&gt;&gt; python StackOverflow.py 
Configuring camera:
 Start configuration...
 End configuration
Taking picture:
 Prepare read
 Writing
 Reading...
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Status: TRANSFER_NO_DEVICE
Traceback (most recent call last):
  File "StackOverflow.py", line 70, in &lt;module&gt;
    take_picture(context);   # Take picture
  File "StackOverflow.py", line 62, in take_picture
    recieved_image = data_collector();
  File "/media/jabozzo/Data/user_data/jabozzo/desktop/sigmamin/code/workspace_Python/USB/USB wxglade libusb1/GuideCamLib/usb.py", line 62, in __exit__
    self.interface_handle.__exit__(exception_type, exception_value, traceback);
  File "/home/jabozzo/.local/lib/python2.7/site-packages/usb1.py", line 1036, in __exit__
    self._handle.releaseInterface(self._interface)
  File "/home/jabozzo/.local/lib/python2.7/site-packages/usb1.py", line 1171, in releaseInterface
    libusb1.libusb_release_interface(self.__handle, interface),
  File "/home/jabozzo/.local/lib/python2.7/site-packages/usb1.py", line 121, in mayRaiseUSBError
    raiseUSBError(value)
  File "/home/jabozzo/.local/lib/python2.7/site-packages/usb1.py", line 117, in raiseUSBError
    raise STATUS_TO_EXCEPTION_DICT.get(value, USBError)(value)
usb1.USBErrorNotFound: LIBUSB_ERROR_NOT_FOUND [-5]
</code></pre>

<h1>Update</h1>

<p>I've changed the Device and Port classes so the interface is opened when the device is openned. That way the interface is only openned (and closed) once, independently of the number of ports openned:</p>

<pre><code># Port class for creating syncronous / asyncronous transfers
class Port:
  def __init__(self, device, address, timeout = None):
    self.device = device;
    self.address = address;
    self.timeout = timeout;
    if(timeout is None):
      self.timeout = 0;

  def read_sync(self, length):
    data = self.device.dev.bulkRead(self.address, length, timeout=self.timeout);
    return [six.byte2int(d) for d in data];

  def write_sync(self, data):
    data = [six.int2byte(d) for d in data];
    return self.device.dev.bulkWrite(self.address, data, timeout=self.timeout);

  # Make asyncronous transfers blocking. Collects data as long as the device
  # sends data more frecuently than self.timeout or all the transfers fails
  def read_async(self, length, pararell_transfers = 32):
    return _TransferCollector(length, pararell_transfers, self);

# Device class for creating ports
class Device:
  def __init__(self, vid = None, pid = None, context = None, interface = 0):

    if(not context):
      self.backend = usb.USBContext();
      context = self.backend.__enter__();

    self.context = context;
    self.interface = interface;

    self.dev = context.openByVendorIDAndProductID(vid, pid, skip_on_error = False);    
    if self.dev is None:
      raise RuntimeError('Device not found');

    self.interface_handle = self.dev.claimInterface(self.interface);

  def __enter__(self):
    self.interface_handle.__enter__();
    return self;

  def __exit__(self, exception_type, exception_value, traceback):
    self.interface_handle.__exit__(exception_type, exception_value, traceback);
    if(hasattr(self, "backend")):
      self.backend.__exit__(exception_type, exception_value, traceback);

  def open_port(self, address, timeout = None):
    return Port(self, address, timeout);
</code></pre>

<p>I still have the same error. But the printing shows me it fails earlier, at the read preparation:</p>

<pre><code>&gt;&gt;&gt; python StackOverflow.py 
Configuring camera:
 Start configuration...
 End configuration
Taking picture:
 Prepare read
Traceback (most recent call last):
...
</code></pre>

<p>I'm beginning to suspect I don't need to open an interface in order to perform asynchronous transfers. </p>