pandas.read_csv vs other csv libraries for loading CSV into a Postgres Database
<p>I am a relatively new user of Python. What is the best way of parsing and processing a CSV and loading it into a local Postgres Database (in Python)?</p>

<p>It was recommended to me to use the CSV library to parse and process the CSV. In particular, the task at hand says:</p>

<blockquote>
  <p>The data might have errors (some rows may be not be parseable), the
  data might be duplicated, the data might be really large.</p>
</blockquote>

<p>Is there a reason why I wouldn't be able to just use <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html" rel="nofollow noreferrer">pandas.read_csv</a> here? Does using the CSV library make parsing and loading it into a local Postgres database easier? In particular, if I just use pandas will I run into problems if rows are unparseable, if the data is really big, or if data is duplicated? (For the last bit, I know that pandas offers some relatively clean solutions for de-dupping). </p>

<p>I feel like pandas.read_csv and <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_sql.html" rel="nofollow noreferrer">pandas.to_sql</a> can do a lot of work for me here, but I'm not sure if using the CSV library offers other advantages.</p>

<p>Just in terms of speed, this post: <a href="https://softwarerecs.stackexchange.com/questions/7463/fastest-python-library-to-read-a-csv-file">https://softwarerecs.stackexchange.com/questions/7463/fastest-python-library-to-read-a-csv-file</a> seems to suggest that pandas.read_csv performs the best?</p>