How to efficiently read, encode and store data from the serial port in Python?
<p>The purpose of the program: Read serial data from the serial port, convert them to int and store them into a list. This data will be later processed and printed in real-time using matplotlib.</p>

<p>The problems: </p>

<ol>
<li>The list is taking too long to be created (several seconds, maybe
minutes).<br>
ps: the size o the list is n = 1024.</li>
<li><p>After all that minutes to create the list I get this:<br>
"(...)<br>
File "C:\Users\ianmc\AppData\Local\Programs\Python\Python35-32\lib\site-packages\matplotlib\lines.py", line 645, in recache<br>
raise RuntimeError('xdata and ydata must be the same length')<br>
RuntimeError: xdata and ydata must be the same length  "</p>

<p>Probably the plot arguments got different lengths but why? </p></li>
</ol>

<p>The code:</p>

<pre><code>import serial
import math
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation


dt = 0.01
Fs = 44000.0                     # sample rate
timestep = 1.0/Fs                # sample spacing (1/sample rate)
n = 1024                         # size of the array data (BUFFER SIZE)
t = np.arange(0, (n/100), dt)    # t range
w = 10000                        # frequency of the input

#initialize data list
data = [0] * n

#initialize magnitude list
magnitude = [0] * (n//2)

# open and prepare serial port
ser = serial.Serial('COM4', 9600, timeout=8,
                    parity=serial.PARITY_EVEN, rtscts=1)


# calculates the fft dB
def createMagnitudeDB(magnitude):
    magnitudeDB = 20 * (math.log(magnitude,10))
    return magnitudeDB

def update(data):
    # update the curves with the incoming data
    timeCurve.set_ydata(data[0])
    freqCurve.set_ydata(data[1])
    return timeCurve, freqCurve,

def generateData():
    # simulate new data coming in
    while True:

        # THE EFFICIENCY PROBLEM IS IN THIS FOR
        # Read data from arduino
        for i in range(n):     #while you are taking data
            PreData = ser.readline()
            data.append(int(PreData))


        # fft computing and normalization
        magnitude = np.fft.fft(data)/n
        magnitude = np.abs(magnitude[range(n//2)])

        # calculates the fft dB
        magnitude_dB = [createMagnitudeDB(x) for x in magnitude]

        yield (data, magnitude)


fig = plt.figure()

# create time graph axis
timeGraph = plt.subplot(2, 1, 1)
timeGraph.set_ylim(-200, 200)
timeGraph.set_xlim([0, n])
timeGraph.set_xlabel('Time')
timeGraph.set_ylabel('Amplitude')

# create frequency graph axis
freqGraph = plt.subplot(2, 1, 2)
freqGraph.set_ylim([0, 70])
freqGraph.set_xlim([0, (Fs/2)])
freqGraph.set_xlabel('Freq (Hz)')
freqGraph.set_ylabel('Magnitude')


#get frequency range
k = np.arange(n)
T = n/Fs
freq = k/T               # two sides frequency range
freq = freq[range(n//2)] # one side frequency range

# plot the curves
timeCurve, = timeGraph.plot(np.linspace(0, 1, n),'b')
freqCurve, = freqGraph.plot(freq, magnitude,'g')

# animate the curves
ani = animation.FuncAnimation(fig, update, generateData,
                              interval = 10, blit=True)

# open window
plt.show()

# close serial connection
ser.close()
</code></pre>

<p>Another piece of information: If instead of using the serial port I generate my own data the program works fine, here is it:  </p>

<pre><code>def generateData():
    # simulate new data coming in
    while True:

        nse = np.random.randn(len(t))
        r = np.exp(-t/0.05)
        cnse = np.convolve(nse, r)*dt
        cnse = cnse[:len(t)]
        data =  100*np.sin(2*np.pi*t) + 500*cnse


        # fft computing and normalization
        magnitude = np.fft.fft(data)/n
        magnitude = np.abs(magnitude[range(n//2)])

        # calculates the fft dB
        magnitude_dB = [createMagnitudeDB(x) for x in magnitude]

        yield (data, magnitude)
</code></pre>

<p>The only difference is the generateData function and not using the open/close serial commands, thats why I don't understand the "xdata and ydata must be the same length" error on the other code (data has the same length on both codes)</p>

<p><strong>EDIT :</strong></p>

<p>The program is receiving data via bluetooth from an arduino that is reading an analog pin. I measured the reading time of the arduino and it's taking "0.015ms" to read each value. To read and print each value it's taking "3.1ms". The reading time of the computer for each value is "100ms".  </p>

<p>Which means it will take n*100(ms) to create the list, being n the number of elements of the list. The minimum 'n' I could use is 256, this would take 25.6s to create the list. Thats just too long for a real-time graph.</p>