Limiting amount read using readline
<p>I'm trying to read the first 100 lines of large text files. Simple code for doing this is shown below. The challenge, though, is that I have to guard against the case of corrupt or otherwise screwy files that don't have any line breaks (yes, people somehow figure out ways to generate these). In those cases I'd still like to read in data (because I need to see what's going on in there) but limit it to, say, n bytes.</p>
<p>The only way I can think of to do this is to read the file char by char. Other than being slow (probably not an issue for only 100 lines) I am worried that I'll run into trouble when I encounter a file using non-ASCII encoding.</p>
<p>Is it possible to limit the bytes read using readline()? Or is there a more elegant way to handle this?</p>
<pre><code>line_count = 0
with open(filepath, 'r') as f:
    for line in f:
        line_count += 1
        print('{0}: {1}'.format(line_count, line))
        if line_count == 100:
            break
</code></pre>
<p><strong>EDIT:</strong></p>
<p>As @Fredrik correctly pointed out, readline() accepts an arg that limits the number of chars read (I'd thought it was a buffer size param). So, for my purposes, the following works quite well:</p>
<pre><code>max_bytes = 1024*1024
bytes_read = 0

fo = open(filepath, &quot;r&quot;)
line = fo.readline(max_bytes)
bytes_read += len(line)
line_count = 0
while line != '':
    line_count += 1
    print('{0}: {1}'.format(line_count, line))
    if (line_count == 100) or (bytes-read &gt;= max_bytes):
        break
    else:
        line = fo.readline(max_bytes - bytes_read)
        bytes_read += len(line)
</code></pre>