Reading UTF-8 file with universal newlines in Python 2
<p>I previously used <code>os.open()</code> to read and write text files. Now I changed to use <code>codecs.open()</code> because I wanted the UTF-8 support. This works well, but this method returned a different result in Windows, because the source files uses <code>\r\n</code> line breaks. It seems to me, that with <code>codecs.open()</code> universal line breaks are not available <a href="https://docs.python.org/2/library/codecs.html#codecs.open" rel="nofollow">because it uses binary mode</a>.</p>

<p>My understanding of the problem is that <code>os.open()</code> and <code>codecs.open()</code> have each mutually exclusive features. <code>os.open()</code> in text mode has the nice feature of <a href="https://www.python.org/dev/peps/pep-0278/" rel="nofollow">universal newline mode</a> (which in the case of reading means that it substitutes any form of line break into <code>\n</code>), whereas <code>codecs.open()</code> provides the UTF-8 support.</p>

<p>My goal is to read (and preferrably write) UTF-8 encoded files into a unicode string with universal line breaks. This implies that if I read two files with different line breaks the resulting strings should be identical. I want to do this using only core libraries with Python 2.6 compatibility. How do I do this in the most elegant way?</p>