Create a numpy array of mixed column types from data using genfromtxt
<p>My question is how to create a numpy (np) array from a csv file which has columns of type int, and string. I found np.genfromtxt to be documented as the function of choice for this [1,2]. I am using python 3.5.1 and numpy 1.11.0. However, the latest numpy documentation I found is for 1.10.0 [3]. As I get an numpy error further down, this might be of interest.</p>

<p>Let me start with what I have</p>

<pre><code>import numpy as np
from io import BytesIO

# Define the input
input = "1,3,Hello\n2,4,World"

# Create a structured np.array from input by reading from BytesIO.
output = np.genfromtxt(BytesIO(input.encode()),
                       delimiter=',',
                       dtype=None)

# output.dtype.names -&gt; ('f0', 'f1', 'f2')
</code></pre>

<p>Here, the columns f0 and f1 are of type int, and f2 is a byte array. Thus</p>

<pre><code>output['f2'] == 'Hello'  # -&gt; False
</code></pre>

<p>is false, as the type differs. A proper comparison must be written as</p>

<pre><code>output['f2'] == b'Hello' # -&gt; [True, False]
</code></pre>

<p>I would prefere to compare to a string and not a byte array. Thus, I want f2 to be of type str. The solutions should be to state the types of each column explicitley. According to [1], this should be possible by setting the genfromtxt argument</p>

<pre><code>dtype=(int, int, str)
</code></pre>

<p>such that the minimal example now reads</p>

<pre><code>import numpy as np
from io import BytesIO

# Define the input
input = "1,3,Hello\n2,4,World"

# Create a structured np.array from input by reading from BytesIO.
output = np.genfromtxt(BytesIO(input.encode()),
                       delimiter=',',
                       dtype=(int, int, str))
</code></pre>

<p>However, this results in a <code>TypeError: data type not understood</code>. Maybe, something has changed between numpy version 1.10.0 and 1.11.0. In any case, I can not get this to work.</p>

<p>Therefore, I tried a second approach using the converters argument of genfromtxt. With this argument, values can be transformed by a function. The example now reads</p>

<pre><code>import numpy as np
from io import BytesIO

# Define the input
input = "1,3,Hello\n2,4,World"

# Create a structured np.array from input by reading from BytesIO.
output = np.genfromtxt(BytesIO(input.encode()),
                       delimiter=',',
                       dtype=None,
                       converters={2: lambda x: str(x, encoding='utf-8')})
</code></pre>

<p>By doing so, f2 is in deed of type <code>&lt;U</code>, which I interpret as utf-8 in little-endian encoding, but only an empty string <code>''</code> is present is each row of f2.</p>

<p>So, how can I read in the given data such that f0 and f1 are int and f2 is str?</p>

<p>[1] <a href="http://docs.scipy.org/doc/numpy-1.10.1/user/basics.io.genfromtxt.html" rel="nofollow">http://docs.scipy.org/doc/numpy-1.10.1/user/basics.io.genfromtxt.html</a></p>

<p>[2] <a href="http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.genfromtxt.html" rel="nofollow">http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.genfromtxt.html</a></p>

<p>[3] <a href="http://docs.scipy.org/doc/numpy/" rel="nofollow">http://docs.scipy.org/doc/numpy/</a></p>