Converting PDF to text: remove word breaks
<p>For my PhD project I am doing some corpus preparation, which mainly consists of cleaning up my text files. I have a corpus of 170 Dutch novels, most of which I have in epub format and which I can easily convert to txt format using Calibre. </p>

<p>The problem is that some novels are in PDF format, which include word breaks at the end of some lines. When I convert these PDF files to txt, the word breaks are still there. For instance:</p>

<p>De reden van alle beroering was niet moeilijk te raden. Adri-aan bleef even staan bij een gezelschap jongerejaars om te ho-ren welke uitkomst de discussie kreeg. Maar terwijl hij stond
te luisteren naar meningen over de acute kwestie Nieuw-Gui-nea, overviel hem de herinnering aan een zonovergoten mid-
dag begin september 1939 in dezelfde hal toen hij, student gene-eskunde, met jaargenoten het zojuist ontketende Duitse of-fensief in Polen besprak.</p>

<p>I would like to know if there is a fast way to get rid of those word breaks. There is no online tool that does the trick. I am slightly familiar with python, so solutions involving python could be welcome. Perhaps an idea is to use a regular expression for the purpose of removing al '-' characters that are at the end of the line in a txt-file?</p>

<p>This is what I tried, using the comments below:</p>

<pre><code>import re

with open('pdf_test.txt','r+', encoding='utf-8') as f:
    text = f.read()
    for line in text:
       if line.endswith('-'):
          line = re.sub('-',' ',line)
          f.seek(0)
          f.write(line)
          f.truncate()
</code></pre>

<p>However, this doesn't work...</p>