

Join them; it only takes a minute:

Sign up
Join the Stack Overflow community to:
  1. Ask programming questions
  2. Answer and help your peers
  3. Get recognized for your expertise

When piping the output of a Python program, the Python interpreter gets confused about encoding and sets it to None. This means a program like this:

# -*- coding: utf-8 -*-
print u"åäö"

will work fine when run normally, but fail with:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 0: ordinal not in range(128)

when used in a pipe sequence.

What is the best way to make this work when piping? Can I just tell it to use whatever encoding the shell/filesystem/whatever is using?

The suggestions I have seen thus far is to modify your directly, or hardcoding the defaultencoding using this hack:

# -*- coding: utf-8 -*-
import sys
print u"åäö"

Is there a better way to make piping work?

share|improve this question
See also… – ShreevatsaR Oct 29 '13 at 6:13

10 Answers 10

up vote 100 down vote accepted

Your code works when run in an script because Python encodes the output to whatever encoding your terminal application is using. If you are piping you must encode it yourself.

A rule of thumb is: Always use Unicode internally. Decode what you receive, and encode what you send.

# -*- coding: utf-8 -*-
print u"åäö".encode('utf-8')

Another didactic example is a Python program to convert between ISO-8859-1 and UTF-8, making everything uppercase in between.

import sys
for line in sys.stdin:
    # Decode what you receive:
    line = line.decode('iso8859-1')

    # Work with Unicode internally:
    line = line.upper()

    # Encode what you send:
    line = line.encode('utf-8')

Setting the system default encoding is a bad idea, because some modules and libraries you use can rely on the fact it is ASCII. Don't do it.

share|improve this answer
The problem is that the user doesn't want to specify encoding explicitly. He wants just use Unicode for IO. And the encoding he uses should be an encoding specified in locale settings, not in terminal application settings. AFAIK, Python 3 uses a locale encoding in this case. Changing sys.stdout seems like a more pleasant way. – Andrey Vlasovskikh Apr 2 '10 at 22:01
Encoding / decoding every string excplictly is bound to cause bugs when a encode or decode call is missing or added once to much somewhere. The output encoding can be set when output is a terminal, so it can be set when output is not a terminal. There is even a standard LC_CTYPE environment to specify it. It is a but in python that it doesn't respect this. – Rasmus Kaj May 31 '10 at 15:34
This answer is wrong. You should not be manually converting on each input and output of your program; that's brittle and completely unmaintainable. – Glenn Maynard Apr 23 '12 at 23:29
@Glenn Maynard : so what is IYO the right answer? It's more helpful to tell us than just say 'This answer is wrong' – smci Sep 18 '12 at 11:10
@ErikJohansson: it is not about stdout accepting whatever encoding. sys.getdefaultencoding() is used in many places e.g., "а" + u"a" expression uses it. Changing sys.getdefaultencoding() may introduce data-dependent bugs that might corrupt your data silently. – J.F. Sebastian Mar 21 '14 at 7:37

First, regarding this solution:

# -*- coding: utf-8 -*-

It's not practical to explicitly print with a given encoding every time. That would be repetitive and error-prone.

A better solution is to change sys.stdout at the start of your program, to encode with a selected encoding. Here is one solution I found on Python: How is sys.stdout.encoding chosen?, in particular a comment by "toka":

import sys
import codecs
sys.stdout = codecs.getwriter('utf8')(sys.stdout)
share|improve this answer
unfortunately, changing sys.stdout to accept only unicode breaks a lot of libraries that expect it to accept encoded bytestrings. – nosklo Dec 4 '09 at 19:14
nosklo: Then how can it work reliably and automaticly when output is a terminal? – Rasmus Kaj May 31 '10 at 15:36
@Rasmus Kaj: just define your own unicode printing function and use it every time you want to print unicode: def myprint(unicodeobj): print unicodeobj.encode('utf-8') -- you automatically detect terminal encoding by inspecting sys.stdout.encoding, but you should consider the case where it is None (i.e. when redirecting output to a file) so you need a separate function anyway. – nosklo May 31 '10 at 20:46
@nosklo: This does not make sys.stdout accept only Unicode. You can pass both str and unicode to a StreamWriter. – Glenn Maynard Apr 23 '12 at 23:30
This trick will screw up the prompt of IPython. – John Chain Sep 26 '14 at 19:36

You may want to try changing the environment variable "PYTHONIOENCODING" to "utf_8." I have written a page on my ordeal with this problem.

Tl;dr of the blog post:

import sys, locale, os
print(chr(246), chr(9786), chr(9787))

gives you

ö ☺ ☻
share|improve this answer
Thanks. Now this solves from a user perspective. – Daniel Ribeiro Sep 23 '11 at 18:15
Changing sys.stdout.encoding maybe does not work, but changing sys.stdout does work: sys.stdout = codecs.getwriter(encoding)(sys.stdout). This can be done from within the python program, so the user is not forced to set an env variable. – delavnog Oct 31 '13 at 7:43
@jeckyll2hide: PYTHONIOENCODING does work. How bytes are interpreted as a text is defined by user environment. Your script shouldn't be assuming and dictate the user environment what character encoding to use. If Python doesn't pick up the settings automatically then PYTHONIOENCODING can be set for your script. You shouldn't need it unless the output is redirected to a file/pipe. – J.F. Sebastian Mar 21 '14 at 7:50
+1. Honestly I think it's a Python bug. When I redirect output I want those same bytes that would be on the terminal, but in a file. Maybe it's not for everyone but it's a good default. Crashing hard with no explanation on a trivial operation that usually "just works" is a bad default. – SnakE Dec 12 '15 at 18:09
@SnakE: the only way I can rationalize why Python's implementation intentionally would enforce an iron-clad and permanent choice of encoding on stdout at startup time, might be in order to prevent any badly encoded stuff coming out later on. Or changing it is just an unimplemented feature, in which case allowing the user to change it later on would be a reasonable Python feature request. – daveagp Dec 13 '15 at 19:09

do the job, but can't set it on python itself ...

what we can do is verify if isn't setting and tell the user to set it before call script with :

if __name__ == '__main__':
    if (sys.stdout.encoding is None):
share|improve this answer

I had a similar issue last week. It was easy to fix in my IDE (PyCharm).

Here was my fix:

Starting from PyCharm menu bar: File -> Settings... -> Editor -> File Encodings, then set: "IDE Encoding", "Project Encoding" and "Default encoding for properties files" ALL to UTF-8 and she now works like a charm.

Hope this helps!

share|improve this answer

I could "automate" it with a call to:

def __fix_io_encoding(last_resort_default='UTF-8'):
  import sys
  if [x for x in (sys.stdin,sys.stdout,sys.stderr) if x.encoding is None] :
      import os
      defEnc = None
      if defEnc is None :
          import locale
          defEnc = locale.getpreferredencoding()
        except: pass
      if defEnc is None :
        try: defEnc = sys.getfilesystemencoding()
        except: pass
      if defEnc is None :
        try: defEnc = sys.stdin.encoding
        except: pass
      if defEnc is None :
        defEnc = last_resort_default
      os.environ['PYTHONIOENCODING'] = os.environ.get("PYTHONIOENCODING",defEnc)
__fix_io_encoding() ; del __fix_io_encoding

Yes, it's possible to get an infinite loop here if this "setenv" fails.

share|improve this answer
interesting, but a pipe doesn't seem to be happy about this – n611x007 Dec 15 '12 at 23:52

I just thought I'd mention something here which I had to spent a long time experimenting with before I finally realised what was going on. This may be so obvious to everyone here that they haven't bothered mentioning it. But it would've helped me if they had, so on that principle...!

NB: I am using Jython specifically, v 2.7, so just possibly this may not apply to CPython...

NB2: the first two lines of my .py file here are:

# -*- coding: utf-8 -*-
from __future__ import print_function

The "%" (AKA "interpolation operator") string construction mechanism causes ADDITIONAL problems too... If the default encoding of the "environment" is ASCII and you try to do something like

print( "bonjour, %s" % "fréd" )  # Call this "print A"

You will have no difficulty running in Eclipse... In a Windows CLI (DOS window) you will find that the encoding is code page 850 (my Windows 7 OS) or something similar, which can handle European accented characters at least, so it'll work.

print( u"bonjour, %s" % "fréd" ) # Call this "print B"

will also work.

If, OTOH, you direct to a file from the CLI, the stdout encoding will be None, which will default to ASCII (on my OS anyway), which will not be able to handle either of the above prints... (dreaded encoding error).

So then you might think of redirecting your stdout by using

sys.stdout = codecs.getwriter('utf8')(sys.stdout)

and try running in the CLI piping to a file... Very oddly, print A above will work... But print B above will throw the encoding error! The following will however work OK:

print( u"bonjour, " + "fréd" ) # Call this "print C"

The conclusion I have come to (provisionally) is that if a string which is specified to be a Unicode string using the "u" prefix is submitted to the %-handling mechanism it appears to involve the use of the default environment encoding, regardless of whether you have set stdout to redirect!

How people deal with this is a matter of choice. I would welcome a Unicode expert to say why this happens, whether I've got it wrong in some way, what the preferred solution to this, whether it also applies to CPython, whether it happens in Python 3, etc., etc.

share|improve this answer
That's not odd, that's because "fréd" is a byte sequence and not a Unicode string, so the codecs.getwriter wrapper will leave it alone. You need a leading u, or from __future__ import unicode_literals. – Matthias Urlichs Nov 16 '14 at 6:33

On Ubuntu 12.10 and GNOME Terminal, no error is produced when the program is printing to stdout or hooked to a pipe for other programs. Both file encoding and terminal encoding is UTF-8.

$ cat
# -*- coding: utf-8 -*-
print "åäö"
$ python
$ python | tee out

What OS and terminal emulator are you using? I heard some of my colleagues have similar problems when using iTerm 2 and OS X; iTerm 2 may be the culprit.

Update: This answer is wrong - see comments for details

share|improve this answer
Correct, the example code was wrong, I have updated it now. – Joakim Lundborg Mar 24 '14 at 16:05
I'm pretty sure this only works because you are using a non-unicode string, unlike the original poster. Try prefixing the string with an u, and watch the code fail when you try to pipe or redirect. – Torkel Bjørnson-Langen Feb 14 '15 at 0:49
@TorkelBjørnson-Langen, yes, you are right, and my answer is wrong. – fossilet Feb 15 '15 at 2:42

An arguable sanitized version of Craig McQueen's answer.

import sys, codecs
class EncodedOut:
    def __init__(self, enc):
        self.enc = enc
        self.stdout = sys.stdout
    def __enter__(self):
        if sys.stdout.encoding is None:
            w = codecs.getwriter(self.enc)
            sys.stdout = w(sys.stdout)
    def __exit__(self, exc_ty, exc_val, tb):
        sys.stdout = self.stdout


with EncodedOut('utf-8'):
    print u'ÅÄÖåäö'
share|improve this answer

There's one simple answer: switch to Python 3!

share|improve this answer
IIRC, Python 3 is different enough from Python 2 that this could cause more problems than it solves. – cHao Mar 21 '13 at 19:39
Does this really solve the problem? Does Python 3 assume UTF-8 rather than ASCII when no encoding is specified? – Mark Ransom Mar 21 '13 at 21:08
@MarkRansom: It seems Python 3 uses UTF-8 encoding by default. (Well, to be fair, it treats strings as Unicode by default, and UTF-8 is one of the few sensible ways to bridge the gap on platforms that natively work with 8-bit char types.) – cHao Mar 22 '13 at 21:10
@cHao, when I execute import sys;print(sys.stdout.encoding) in Python3.2 on Windows with the output redirected, I get cp1252. I doubt that's the desired result; it's certainly not utf-8 as much as that might make sense and as much as we might wish it to be so. – Mark Ransom Mar 23 '13 at 2:21
@MarkRansom: You might get cp1252. This is what the OP requsted: "Can I just tell it to use whatever encoding the shell/filesystem/whatever is using?" Python 3 tries not to assume, but is told what the encoding is. Stdin/out are special, and system configuration gives the encoding, but the principle still holds. It was told that the input is in some specific encoding (UTF-8 usually, maybe something different on Windows), and will encode strings stored internally as Unicode to this specific encoding. Better then the old approach of hoping that the string encoding happens to be the one required. – zbyszek Mar 27 '13 at 20:13

Your Answer


