Storing and analysing experimental data in efficient way (SQL, PYTHON)
<p>I am currently employed as an undergraduate intern in a neuroscience research lab at my university where I am the dedicated "coder/math guy". I know this description is cringey but I think this is how how I am perceived. </p>

<p>I have a number of tasks including updating legacy code which has led me to take a step back and try to conceptualize the best way to do this. I am looking for advice on the best way to structure my process here.</p>

<p>The data collected is currently from two forms. One is from a 3d based capture equipment which gathers data on different markers, most notably their xyz position, it can export this data in a CSV. However there is also a weird old API written in C which I could use to fetch the data manually.
There is also an eye tracker which stores its data in a proprietary binary format with a MEX file provided to extract relevant info in Matlab, however that compiled MEX file seems to be completely black boxed.</p>

<p>I was thinking of trying to store all that data in a relational database and using the python plugin to go fetch the needed one and structure in certain dataframes for analysis. We are talking of about 100,000 * 10 of 64floating for the 3d motion capture.</p>

<p>Is it worth it to look into SQL and NOSQL schema or should I just store everything in HDF5 or JSON in structured directories? </p>