convert em-dash to hyphen in python
<p>I'm converting csv files into python Dataframe. And in the original file, one of the column has characters em-dash. I want it replaced by hyphen "-".</p>

<p>Partial original file from csv:</p>

<pre><code> NoDemande     NoUsager     Sens    IdVehicule     NoConduteur     HeureDebutTrajet    HeureArriveeSurSite    HeureEffective'
42192001801   42192002715    —        157Véh       42192000153    ...
42192000003   42192002021    +        157Véh       42192000002    ...
42192001833   42192000485    —      324My3FVéh     42192000157    ...
</code></pre>

<p>My code:</p>

<pre><code>#coding=latin-1
import pandas as pd
import glob

pd.set_option('expand_frame_repr', False)

path = r'D:\Python27\mypfe\data_test'
allFiles = glob.glob(path + "/*.csv")
frame = pd.DataFrame()
list_ = []
for file_ in allFiles:
    df = pd.read_csv(file_,index_col=None,header=0,sep=';',parse_dates=['HeureDebutTrajet','HeureArriveeSurSite','HeureEffective'],
                      dayfirst=True)
    df['Sens'].replace(u"\u2014","-",inplace=True,regex=True)
    list_.append(df)
</code></pre>

<p>And it doesn't work at all, every time it only convert them into <code>?</code>, which looks like it:</p>

<pre><code>42191001122  42191002244    ?            181Véh   42191000114  ...
42191001293  42191001203    ?         319M9pVéh   42191000125  ...
42191000700  42191000272    ?            183Véh   42191000072  ...
</code></pre>

<p>And because I have french characters in the file, I'm using <code>latin-1</code> instead of <code>utf-8</code>. If I delete the first line and write like this:</p>

<pre><code>df = pd.read_csv(file_,index_col=None,header=0,sep=';',encoding='windows-1252',parse_dates=['HeureDebutTrajet','HeureArriveeSurSite','HeureEffective'],
                          dayfirst=True)
</code></pre>

<p>The result will be:</p>

<pre><code>42191001122  42191002244  â??           181VÃ©h   42191000114   ...
42191001293  42191001203  â??        319M9pVÃ©h   42191000125   ...
42191000700  42191000272  â??           183VÃ©h   42191000072   ...
</code></pre>

<p>How can I make all the em-dash <code>—</code> replaced by <code>-</code>?</p>

<p>I added the part about <code>repr</code>:</p>

<pre><code>for line in open(file_):
    print repr(line)
</code></pre>

<p>And the result turns out:</p>

<pre><code>'"42191002384";"42191000118";"\xe2\x80\x94";"";"42191000182";...
'"42191002464";"42191001671";"+";"";"42191000182";...
'"42191000045";"42191000176";"\xe2\x80\x94";"620M9pV\xc3\xa9h";"42191000003";...
'"42191001305";"42191000823";"\xe2\x80\x94";"310V7pV\xc3\xa9h";"42191000126";...
</code></pre>