How to serialize/deserialize Pandas DataFrame to and from ProtoBuf/Gzip in a RESTful Flask App?
<p>I have a <code>pandas dataframe</code> to be returned as a <code>Flask Response</code> object in a <code>flask</code> application. Currently I am converting it to a <code>JSON Object</code>,</p>

<pre><code>df = df.to_json()
return Response(df, status=200, mimetype='application/json') 
</code></pre>

<p>The dataframe size is really huge of the magnitude, probably 5000000 X 10.
On the client side when I deserialize it as,</p>

<pre><code>df = response.read_json()
</code></pre>

<p>As my number of <code>URL request parameters</code> grow, the <code>dataframe</code> grows as well. Deserialization time grows at a <code>linear factor</code> as compared to serialization, which I would want to avoid. e.g: <code>Serialization</code> takes 15-20 seconds, <code>deserialization</code> takes 60-70 seconds.</p>

<p>Is there a way that <code>protobuf</code> can help in this case to convert pandas dataframe to a protobuf object. Also is there a way that I can send this <code>JSON</code> as <code>Gunzipped</code> mimetype through flask? I believe there's a comparable timing and efficiency between <code>protobuf</code> and <code>gunzip</code>. </p>

<p>What's the best solution in such a scenario?</p>

<p>Thanks in advance.</p>