Keeping large files loaded in Flask
<p>I'm writing a "webapp" for my own personal use, meant to be run with my own computer as a server. It's basically a nice interface for data visualization. This app needs to manipulate large matrices - about 100MB - in Python, and return the computation results to the browser for visualization. Currently I'm just storing the data in a csv file and loading it into pandas every time I want to use it, but this is very slow (about 15 seconds). Is there a way to have this object (a pandas.DataFrame) persist in memory, or does this not make sense?</p>

<p>I tried memcached and I do not think it is appropriate here. I also tried using Redis but reading from the cache is actually the same speed as reading the file if I store each matrix row under its own key, and if I store it all in a string under the same key then reconstructing the array from the string is just as slow as reading it from the csv file. So no gains either way.</p>