Math Functions in Python | Set 2 (logarithmic and power functions)

log | logarithm | power | Python Methods and Functions

# Python code to demonstrate how it works
# exp () and log ()

# import & quot; mathematics '' for math operations

import math

# return experience 4

print ( "The e ** 4 value is:" , end = "")

print (math.exp ( 4 ))

# return 2,3 log

print ( "The value of log 2 with base 3 is:" , end = "")

print (math.log ( 2 , 3 ))


 The e ** 4 value is: 54.598150033144236 The value of log 2 with base 3 is: 0.6309297535714574 

3. log2 (a) : — This function calculates the value of log a with base 2 . This value is more accurate than the function above.

4. log10 (a) : — This function calculates the value of log a with base 10 . This value is more precise than the function above.

# Python code to demonstrate how it works
# log2 () and log10 ()

# import & quot; math & quot; for math operations

import math

# return log2 of 16

print ( "The value of log2 of 16 is:" , end = "")

print (math.log2 ( 16 ))

# return log10 of 10000

print ( "The value of log10 of 10000 is:" , end = "")

print (math.log10 ( 10000 ))


 The value of log2 of 16 is: 4.0 The value of log10 of 10000 is: 4.0 

5 ... pow (a, b) : — This function is used to calculate the value raised to the power b (a ** b) .

6. sqrt () : — This function returns the square root of a number.

# Python code to demonstrate how it works
# pow () and sqrt ()

# import & quot; math & quot; for math operations

import math

# returns 3 ** 2

print ( "The value of 3 to the power 2 is:" , end = "")

print (math. pow ( 3 , 2 ))

# returns the square root of 25

print ( " The value of square root of 25: " , end = "")

print (math.sqrt ( 25 ))


 The value of 3 to the power 2 is: 9.0 The value of square root of 25 : 5.0 

This article is contributing to Manjeet Singh If you love Python.Engineering and would like to contribute, you can also write an article using or post the article [email protected] See my article appearing on the Python.Engineering homepage and help other geeks.


Math Functions in Python | Set 2 (logarithmic and power functions): StackOverflow Questions

Python"s equivalent of && (logical-and) in an if-statement

Question by delete

Here"s my code:

def front_back(a, b):
  # +++your code here+++
  if len(a) % 2 == 0 && len(b) % 2 == 0:
    return a[:(len(a)/2)] + b[:(len(b)/2)] + a[(len(a)/2):] + b[(len(b)/2):] 
    #todo! Not yet done. :P

I"m getting an error in the IF conditional.
What am I doing wrong?

How do you get the logical xor of two variables in Python?

Question by Zach Hirsch


For example, I have two variables that I expect to be strings. I want to test that only one of them contains a True value (is not None or the empty string):

str1 = raw_input("Enter string one:")
str2 = raw_input("Enter string two:")
if logical_xor(str1, str2):
    print "ok"
    print "bad"

The ^ operator seems to be bitwise, and not defined on all objects:

>>> 1 ^ 1
>>> 2 ^ 1
>>> "abc" ^ ""
Traceback (most recent call last):
TypeError: unsupported operand type(s) for ^: "str" and "str"

How do I log a Python error with debug information?

I am printing Python exception messages to a log file with logging.error:

import logging
except ZeroDivisionError as e:
    logging.error(e)  # ERROR:root:division by zero

Is it possible to print more detailed information about the exception and the code that generated it than just the exception string? Things like line numbers or stack traces would be great.

Making Python loggers output all messages to stdout in addition to log file

Question by user248237

Is there a way to make Python logging using the logging module automatically output things to stdout in addition to the log file where they are supposed to go? For example, I"d like all calls to logger.warning, logger.critical, logger.error to go to their intended places but in addition always be copied to stdout. This is to avoid duplicating messages like:

mylogger.critical("something failed")
print "something failed"

Separation of business logic and data access in django

I am writing a project in Django and I see that 80% of the code is in the file This code is confusing and, after a certain time, I cease to understand what is really happening.

Here is what bothers me:



class User(db.Models):

    def get_present_name(self):
        return or "Anonymous"

    def activate(self):
        self.status = "activated"

Over time, it turned into this:

class User(db.Models):

    def get_present_name(self): 
        # property became non-deterministic in terms of database
        # data is taken from another service by api
        return remote_api.request_user_name(self.uid) or "Anonymous" 

    def activate(self):
        self.status = "activated"
        send_mail("Your account is activated!", "‚Ä¶", [])

What I want is to separate entities in my code:

  1. Entities of my database, persistence level: What data does my application keep?
  2. Entities of my application, business logic level: What does my application do?


Plot logarithmic axes with matplotlib in python

Question by Jim

I want to plot a graph with one logarithmic axis using matplotlib.

I"ve been reading the docs, but can"t figure out the syntax. I know that it"s probably something simple like "scale=linear" in the plot arguments, but I can"t seem to get it right

Sample program:

import pylab
import matplotlib.pyplot as plt
a = [pow(10, i) for i in range(10)]
fig = plt.figure()
ax = fig.add_subplot(2, 1, 1)

line, = ax.plot(a, color="blue", lw=2)

logger configuration to log to file and print to stdout

I"m using Python"s logging module to log some debug strings to a file which works pretty well. Now in addition, I"d like to use this module to also print the strings out to stdout. How do I do this? In order to log my strings to a file I use following code:

import logging
import logging.handlers
logger = logging.getLogger("")
handler = logging.handlers.RotatingFileHandler(
    LOGFILE, maxBytes=(1048576*5), backupCount=7

and then call a logger function like

logger.debug("I am written to the file")

Thank you for some help here!

What are logits? What is the difference between softmax and softmax_cross_entropy_with_logits?

In the tensorflow API docs they use a keyword called logits. What is it? A lot of methods are written like:

tf.nn.softmax(logits, name=None)

If logits is just a generic Tensor input, why is it named logits?

Secondly, what is the difference between the following two methods?

tf.nn.softmax(logits, name=None)
tf.nn.softmax_cross_entropy_with_logits(logits, labels, name=None)

I know what tf.nn.softmax does, but not the other. An example would be really helpful.

How can I color Python logging output?

Question by airmind

Some time ago, I saw a Mono application with colored output, presumably because of its log system (because all the messages were standardized).

Now, Python has the logging module, which lets you specify a lot of options to customize output. So, I"m imagining something similar would be possible with Python, but I can‚Äôt find out how to do this anywhere.

Is there any way to make the Python logging module output in color?

What I want (for instance) errors in red, debug messages in blue or yellow, and so on.

Of course this would probably require a compatible terminal (most modern terminals are); but I could fallback to the original logging output if color isn"t supported.

Any ideas how I can get colored output with the logging module?

How do I disable log messages from the Requests library?



I"m usually not interested in these messages, and would like to disable them. What would be the best way to silence those messages or decrease Requests" verbosity?

Answer #1



From the range() object documentation:



class my_range:
        if stop is None:
            start, stop = 0, start
        self.start, self.stop, self.step = start, stop, step
        if step < 0:
            lo, hi, step = stop, start, -step
            lo, hi = start, stop

    def __iter__(self):
        current = self.start
        if self.step < 0:
            while current > self.stop:
                yield current
                current += self.step
            while current < self.stop:
                yield current
                current += self.step

    def __len__(self):
        return self.length

    def __getitem__(self, i):
        if i < 0:
            i += self.length
        if 0 <= i < self.length:
            return self.start + i * self.step
        raise IndexError("my_range object index out of range")

    def __contains__(self, num):
        if self.step < 0:
            if not (self.stop < num <= self.start):
                return False
            if not (self.start <= num < self.stop):
                return False
        return (num - self.start) % self.step == 0




Answer #2



from multiprocessing.dummy import Pool as ThreadPool
pool = ThreadPool(4)
results =, my_array)

Which is the multithreaded version of:

results = []
for item in my_array:




Enter image description here





import urllib2
from multiprocessing.dummy import Pool as ThreadPool

urls = [

# Make the Pool of workers
pool = ThreadPool(4)

# Open the URLs in their own threads
# and return the results
results =, urls)

# Close the pool and wait for the work to finish

And the timing results:

Single thread:   14.4 seconds
       4 Pool:   3.1 seconds
       8 Pool:   1.4 seconds
      13 Pool:   1.3 seconds


To pass multiple arrays:


Or to pass a constant and an array:



(Thanks to user136036 for the helpful comment.)

Answer #3

How to iterate over rows in a DataFrame in Pandas?

Answer: DON"T*!




  1. Vectorization
  2. Cython routines
  3. List Comprehensions (vanilla for loop)
  5. DataFrame.itertuples() and iteritems()
  6. DataFrame.iterrows()


Appeal to Authority




Faster than Looping: Vectorization, Cython



Next Best Thing: List Comprehensions*


The formula is simple,

# Iterating over two columns, use `zip`
# Iterating over multiple columns - same data type
# Iterating over multiple columns - differing data type






An Obvious Example




Further Reading


Why I Wrote this Answer



Answer #4




  1. faster attribute access.
  2. space savings in memory.

The space savings is from

  1. Storing value references in slots instead of __dict__.

Quick Caveats


class Base:
    __slots__ = "foo", "bar"

class Right(Base):
    __slots__ = "baz", 

class Wrong(Base):


>>> from sys import getsizeof
(56, 72)


>>> w = Wrong()
>>> = "foo"
Traceback (most recent call last):
AttributeError: foo



See section on multiple inheritance below for an example.





Why use __slots__: Faster attribute access.


It is trivial to demonstrate measurably significant faster access:

import timeit

class Foo(object): __slots__ = "foo",

class Bar(object): pass

slotted = Foo()
not_slotted = Bar()

def get_set_delete_fn(obj):
    def get_set_delete(): = "foo"
    return get_set_delete




>>> 0.3664822799983085 / 0.2846834529991611


Why use __slots__: Memory Savings



The space saved over using __dict__ can be significant.

SQLAlchemy attributes a lot of memory savings to __slots__.




       Python 2.7             Python 3.6
one    48         56 + 272    48         56 + 112
two    56         56 + 272    56         56 + 112
six    88         56 + 1040   88         56 + 152
11     128        56 + 1040   128        56 + 240
22     216        56 + 3344   216        56 + 408     
43     384        56 + 3344   384        56 + 752



<member "foo" of "Foo" objects>
>>> type(
<class "member_descriptor">
>>> getsizeof(

Demonstration of __slots__:


class Base(object): 
    __slots__ = ()


>>> b = Base()
>>> b.a = "a"
Traceback (most recent call last):
    b.a = "a"
AttributeError: "Base" object has no attribute "a"

Or subclass another class that defines __slots__

class Child(Base):
    __slots__ = ("a",)

and now:

c = Child()
c.a = "a"


>>> c.b = "b"
Traceback (most recent call last):
    c.b = "b"
AttributeError: "Child" object has no attribute "b"


class SlottedWithDict(Child): 
    __slots__ = ("__dict__", "b")

swd = SlottedWithDict()
swd.a = "a"
swd.b = "b"
swd.c = "c"


>>> swd.__dict__
{"c": "c"}


class NoSlots(Child): pass
ns = NoSlots()
ns.a = "a"
ns.b = "b"


>>> ns.__dict__
{"b": "b"}

However, __slots__ may cause problems for multiple inheritance:

class BaseA(object): 
    __slots__ = ("a",)

class BaseB(object): 
    __slots__ = ("b",)


Traceback (most recent call last):
    class Child(BaseA, BaseB): __slots__ = ()
TypeError: Error when calling the metaclass bases
    multiple bases have instance lay-out conflict


from abc import ABC

class AbstractA(ABC):
    __slots__ = ()

class BaseA(AbstractA): 
    __slots__ = ("a",)

class AbstractB(ABC):
    __slots__ = ()

class BaseB(AbstractB): 
    __slots__ = ("b",)

class Child(AbstractA, AbstractB): 
    __slots__ = ("a", "b")

c = Child() # no problem!

Add "__dict__" to __slots__ to get dynamic assignment:

class Foo(object):

and now:

>>> foo = Foo()
>>> foo.boink = "boink"





Set to empty tuple when subclassing a namedtuple:


from collections import namedtuple
    __slots__ = ()


>>> nt.baz


>>> nt.quux = "quux"
Traceback (most recent call last):
AttributeError: "MyNT" object has no attribute "quux"


Biggest Caveat: Multiple inheritance


class Foo(object): 
    __slots__ = "foo", "bar"
class Bar(object):

>>> class Baz(Foo, Bar): pass
Traceback (most recent call last):
TypeError: Error when calling the metaclass bases
    multiple bases have instance lay-out conflict


class Foo(object): __slots__ = ()
class Bar(object): __slots__ = ()




class AbstractBase:
    __slots__ = ()
    def __init__(self, a, b):
        self.a = a
        self.b = b
    def __repr__(self):


class Foo(AbstractBase):
    __slots__ = "a", "b"


class AbstractBaseC:
    __slots__ = ()
    def c(self):
        print("getting c!")
        return self._c
    def c(self, arg):
        print("setting c!")
        self._c = arg


class Concretion(AbstractBase, AbstractBaseC):
    __slots__ = "a b _c".split()


>>> c.c = c
setting c!
>>> c.c
getting c!
Concretion("a", "b")
>>> c.d = "d"
Traceback (most recent call last):
AttributeError: "Concretion" object has no attribute "d"

Other cases to avoid slots:



Critiques of other answers


Do not "only use __slots__ when instantiating lots of objects"

I quote:





__slots__ contributes to reusability when creating interfaces or mixins.


__slots__ doesn"t break pickling


>>> pickle.loads(pickle.dumps(f))


<__main__.Foo object at 0x1129C770>

in Python 2.7:

>>> pickle.loads(pickle.dumps(f, 2))
<__main__.Foo object at 0x1129C770>

in Python 3.6

>>> pickle.loads(pickle.dumps(f, 4))
<__main__.Foo object at 0x1129C770>


Critique of the (until Oct 2, 2016) accepted answer



The second half is wishful thinking, and off the mark:







Memory usage evidence

Create some normal objects and slotted objects:

>>> class Foo(object): pass

Instantiate a million of them:


Inspect with guppy.hpy().heap():

>>> guppy.hpy().heap()
     0 1000000  49 64000000  64  64000000  64 __main__.Foo
     1     169   0 16281480  16  80281480  80 list
     2 1000000  49 16000000  16  96281480  97 __main__.Bar
     3   12284   1   987472   1  97268952  97 str

Access the regular objects and their __dict__ and inspect again:

>>> for f in foos:
...     f.__dict__
>>> guppy.hpy().heap()
     0 1000000  33 280000000  74 280000000  74 dict of __main__.Foo
     1 1000000  33  64000000  17 344000000  91 __main__.Foo
     2     169   0  16281480   4 360281480  95 list
     3 1000000  33  16000000   4 376281480  99 __main__.Bar
     4   12284   0    987472   0 377268952  99 str



Answer #5

os.listdir() - list in the current directory


 import os
 arr = os.listdir()

Looking in a directory

arr = os.listdir("c:\files")

glob from glob


import glob

txtfiles = []
for file in glob.glob("*.txt"):

glob in a list comprehension


get the full path of only files in the current directory

import os
from os import listdir
from os.path import isfile, join

cwd = os.getcwd()
os.path.isfile(os.path.join(cwd, f))]


Getting the full path name with os.path.abspath

You get the full path in return

 import os

Walk: going through sub directories


import os

# Getting the current work directory (cwd)
thisdir = os.getcwd()

# r=root, d=directories, f = files
for r, d, f in os.walk(thisdir):
    for file in f:
        if file.endswith(".docx"):
            print(os.path.join(r, file))



 import os
 arr = os.listdir(".")

To go up in the directory tree

# Method 1
x = os.listdir("..")

# Method 2
x= os.listdir("/")


 import os
 arr = os.listdir("F:\python")

Get files of a particular subdirectory with os.listdir()

import os

x = os.listdir("./content")

os.walk(".") - current directory

 import os


 import os
 arr = []
     for file in f:

 for f in arr:

>>> F:\_python\
>>> F:\_python\programmi.txt






os.listdir() - get only txt files

 >>> ["work.txt", "3ebooks.txt"]

Using glob to get the full path of the files

If I should need the absolute path of the files:

from path import path
from glob import glob
for f in x:

>>> F:acquistionline.txt
>>> F:acquisti_2018.txt
>>> F:ootstrap_jquery_ecc.txt

Using os.path.isfile to avoid directories in the list

import os.path


Using pathlib from Python 3.4

import pathlib

flist = []
    if p.is_file():

 >>> error.PNG
 >>> exemaker.bat
 >>> guiprova.mp3
 >>> thumb.PNG

With list comprehension:



Use glob method in pathlib.Path()

import pathlib

for file in py:


Get all and only files with os.walk

import os
for t in x:
    for f in t:


Get only files with next and walk in a directory

 import os
 >>> ["calculator.bat",""]

Get only directories with next and walk in a directory

 import os
 >>> ["python3","others"]

Get all the subdir names with walk

    for dirs in d:

>>> .vscode
>>> pyexcel
>>> subtitles
>>> _metaprogramming
>>> .ipynb_checkpoints

os.scandir() from Python 3.5 and greater

import os

>>> ["calculator.bat",""]

# Another example with scandir (a little variation from
# This one is more efficient than os.listdir.
# where the script is executed.

import os
with os.scandir() as i:
    for entry in i:
        if entry.is_file():

>>> error.PNG
>>> exemaker.bat
>>> guiprova.mp3
>>> thumb.PNG




import os

def count(dir, counter=0):
    "returns number of files in dir and subdirs"
    for pack in os.walk(dir):
        for f in pack[2]:
            counter += 1





import os
import shutil
from path import path

destination = "F:\file_copied"
# os.makedirs(destination)

    for pack in os.walk(dir):
        for f in pack[2]:
            if f.endswith(filetype):
                fullpath = pack[0] + "\" + f
                shutil.copy(fullpath, destination)
                counter += 1
    if counter > 0:
        print("-" * 30)

for dir in os.listdir():
    "searches for folders that starts with `_`"
    if dir[0] == "_":
        # copyfile(dir, filetype="pdf")
        copyfile(dir, filetype="txt")

>>> _compiti18Compito Contabilit√† 1conti.txt
>>> _compiti18Compito Contabilit√† 1modula4.txt
>>> _compiti18Compito Contabilit√† 1moduloa4.txt



import os
mylist = ""
    for eachfile in os.listdir():
        mylist += eachfile + "

Example: txt with all the files of an hard drive

We will use the function walk()

import os

# see all the methods of os
listafile = []
percorso = []
        for file in files:
            percorso.append(root + "\" + file)
            testo.write(file + "
    for file in listafile:
        testo_ordinato.write(file + "

    for file in percorso:
        file_percorso.write(file + "


All the file of C: in one text file


import os

        for file in f:
            filewrite.write(f"{r + file}



import os

        for r, d, f in os.walk(folder):
            for file in f:
                if file.endswith(extension):
                    filewrite.write(f"{r + file}


>>> H:4bs_18Dolphins5.png
>>> H:4bs_18Dolphins6.png
>>> H:4bs_18Dolphins7.png
>>> H:5_18marketing htmlassetsimageslogo2.png
>>> H:7z001.png
>>> H:7z002.png

(New) Find all files and open them with tkinter GUI


import tkinter as tk
import os

    "insert all files in the listbox"
    for r, d, f in os.walk(folder):
        for file in f:
            if file.endswith(extension):
                lb.insert(0, r + "\" + file)

def open_file():

root = tk.Tk()
lb = tk.Listbox(root)
lb.pack(fill="both", expand=1)

Answer #6


There are seven possible actions to take when such event occurs:


Source: Django documentation

See also the documentation of PostgreSQL for instance.


Additional note to clarify cascade direction




I hope it makes cascade direction clearer. :)




Answer #7

You can also use the option_context, with one or more options:

with pd.option_context("display.max_rows", None, "display.max_columns", None):  # more options can be specified also

This will automatically return the options to their previous values.

If you are working on jupyter-notebook, using display(df) instead of print(df) will use jupyter rich display logic (like so).

Answer #8

The or and and python statements require truth-values. For pandas these are considered ambiguous so you should use "bitwise" | (or) or & (and) operations:

result = result[(result["var"]>0.25) | (result["var"]<-0.25)]

These are overloaded for these kind of datastructures to yield the element-wise or (or and).

Just to add some more explanation to this statement:

The exception is thrown when you want to get the bool of a pandas.Series:

>>> import pandas as pd
>>> x = pd.Series([1])
>>> bool(x)
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

What you hit was a place where the operator implicitly converted the operands to bool (you used or but it also happens for and, if and while):

>>> x or x
>>> x and x
>>> if x:
...     print("fun")
>>> while x:
...     print("fun")

Besides these 4 statements there are several python functions that hide some bool calls (like any, all, filter, ...) these are normally not problematic with pandas.Series but for completeness I wanted to mention these.

In your case the exception isn"t really helpful, because it doesn"t mention the right alternatives. For and and or you can use (if you want element-wise comparisons):

  • numpy.logical_or:

    >>> import numpy as np
    >>> np.logical_or(x, y)

    or simply the | operator:

    >>> x | y
  • numpy.logical_and:

    >>> np.logical_and(x, y)

    or simply the & operator:

    >>> x & y

If you"re using the operators then make sure you set your parenthesis correctly because of the operator precedence.

There are several logical numpy functions which should work on pandas.Series.

The alternatives mentioned in the Exception are more suited if you encountered it when doing if or while. I"ll shortly explain each of these:

  • If you want to check if your Series is empty:

    >>> x = pd.Series([])
    >>> x.empty
    >>> x = pd.Series([1])
    >>> x.empty

    Python normally interprets the length of containers (like list, tuple, ...) as truth-value if it has no explicit boolean interpretation. So if you want the python-like check, you could do: if x.size or if not x.empty instead of if x.

  • If your Series contains one and only one boolean value:

    >>> x = pd.Series([100])
    >>> (x > 50).bool()
    >>> (x < 50).bool()
  • If you want to check the first and only item of your Series (like .bool() but works even for not boolean contents):

    >>> x = pd.Series([100])
    >>> x.item()
  • If you want to check if all or any item is not-zero, not-empty or not-False:

    >>> x = pd.Series([0, 1, 2])
    >>> x.all()   # because one element is zero
    >>> x.any()   # because one (or more) elements are non-zero

Answer #9

If you like ascii art:

  • "VALID" = without padding:

       inputs:         1  2  3  4  5  6  7  8  9  10 11 (12 13)
                      |________________|                dropped
  • "SAME" = with zero padding:

                   pad|                                      |pad
       inputs:      0 |1  2  3  4  5  6  7  8  9  10 11 12 13|0  0

In this example:

  • Input width = 13
  • Filter width = 6
  • Stride = 5


  • "VALID" only ever drops the right-most columns (or bottom-most rows).
  • "SAME" tries to pad evenly left and right, but if the amount of columns to be added is odd, it will add the extra column to the right, as is the case in this example (the same logic applies vertically: there may be an extra row of zeros at the bottom).


About the name:

  • With "SAME" padding, if you use a stride of 1, the layer"s outputs will have the same spatial dimensions as its inputs.
  • With "VALID" padding, there"s no "made-up" padding inputs. The layer only uses valid input data.

Answer #10


All you have to do is:

sudo easy_install pip





python --version



sudo easy_install pip






easy_install has been deprecated. Please use instead.

First of all download the get-pip file

curl -o

Now run this file to install pip


That should do it.

Another gif you said? Here ya go!


Plot logarithmic axes with matplotlib in python

Question by Jim



Sample program:

import pylab
import matplotlib.pyplot as plt
fig = plt.figure()
ax = fig.add_subplot(2, 1, 1)


How to do exponential and logarithmic curve fitting in Python? I found only polynomial fitting

I have a set of data and I want to compare which line describes it best (polynomials of different orders, exponential or logarithmic).

I use Python and Numpy and for polynomial fitting there is a function polyfit(). But I found no such functions for exponential and logarithmic fitting.

Are there any? Or how to solve it otherwise?

Answer #1

There are lots of things I have seen make a model diverge.

  1. Too high of a learning rate. You can often tell if this is the case if the loss begins to increase and then diverges to infinity.

  2. I am not to familiar with the DNNClassifier but I am guessing it uses the categorical cross entropy cost function. This involves taking the log of the prediction which diverges as the prediction approaches zero. That is why people usually add a small epsilon value to the prediction to prevent this divergence. I am guessing the DNNClassifier probably does this or uses the tensorflow opp for it. Probably not the issue.

  3. Other numerical stability issues can exist such as division by zero where adding the epsilon can help. Another less obvious one if the square root who"s derivative can diverge if not properly simplified when dealing with finite precision numbers. Yet again I doubt this is the issue in the case of the DNNClassifier.

  4. You may have an issue with the input data. Try calling assert not np.any(np.isnan(x)) on the input data to make sure you are not introducing the nan. Also make sure all of the target values are valid. Finally, make sure the data is properly normalized. You probably want to have the pixels in the range [-1, 1] and not [0, 255].

  5. The labels must be in the domain of the loss function, so if using a logarithmic-based loss function all labels must be non-negative (as noted by evan pu and the comments below).

Answer #2


In Python 3.8, the prod function was added to the math module. See:

Older info: Python 3.7 and prior

The function you"re looking for would be called prod() or product() but Python doesn"t have that function. So, you need to write your own (which is easy).

Pronouncement on prod()

Yes, that"s right. Guido rejected the idea for a built-in prod() function because he thought it was rarely needed.

Alternative with reduce()


from functools import reduce  # Required in Python 3
import operator
def prod(iterable):
    return reduce(operator.mul, iterable, 1)

>>> prod(range(1, 5))

Note, in Python 3, the reduce() function was moved to the functools module.

Specific case: Factorials

As a side note, the primary motivating use case for prod() is to compute factorials. We already have support for that in the math module:

>>> import math

>>> math.factorial(10)

Alternative with logarithms

If your data consists of floats, you can compute a product using sum() with exponents and logarithms:

>>> from math import log, exp

>>> data = [1.2, 1.5, 2.5, 0.9, 14.2, 3.8]
>>> exp(sum(map(log, data)))

>>> 1.2 * 1.5 * 2.5 * 0.9 * 14.2 * 3.8

Note, the use of log() requires that all the inputs are positive.

Answer #3

Based on a feature mentioned in this answer to another question I have found a very generally applicable solution for placing labels on a bar chart.

Other solutions unfortunately do not work in many cases, because the spacing between label and bar is either given in absolute units of the bars or is scaled by the height of the bar. The former only works for a narrow range of values and the latter gives inconsistent spacing within one plot. Neither works well with logarithmic axes.

The solution I propose works independent of scale (i.e. for small and large numbers) and even correctly places labels for negative values and with logarithmic scales because it uses the visual unit points for offsets.

I have added a negative number to showcase the correct placement of labels in such a case.

The value of the height of each bar is used as a label for it. Other labels can easily be used with Simon"s for rect, label in zip(rects, labels) snippet.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Bring some raw data.
frequencies = [6, -16, 75, 160, 244, 260, 145, 73, 16, 4, 1]

# In my original code I create a series and run on that,
# so for consistency I create a series from the list.
freq_series = pd.Series.from_array(frequencies)

x_labels = [108300.0, 110540.0, 112780.0, 115020.0, 117260.0, 119500.0,
            121740.0, 123980.0, 126220.0, 128460.0, 130700.0]

# Plot the figure.
plt.figure(figsize=(12, 8))
ax = freq_series.plot(kind="bar")
ax.set_title("Amount Frequency")
ax.set_xlabel("Amount ($)")

def add_value_labels(ax, spacing=5):
    """Add labels to the end of each bar in a bar chart.

        ax (matplotlib.axes.Axes): The matplotlib object containing the axes
            of the plot to annotate.
        spacing (int): The distance between the labels and the bars.

    # For each bar: Place a label
    for rect in ax.patches:
        # Get X and Y placement of label from rect.
        y_value = rect.get_height()
        x_value = rect.get_x() + rect.get_width() / 2

        # Number of points between bar and label. Change to your liking.
        space = spacing
        # Vertical alignment for positive values
        va = "bottom"

        # If value of bar is negative: Place label below bar
        if y_value < 0:
            # Invert space to place label below
            space *= -1
            # Vertically align label at top
            va = "top"

        # Use Y value as label and format number with one decimal place
        label = "{:.1f}".format(y_value)

        # Create annotation
            label,                      # Use `label` as label
            (x_value, y_value),         # Place label at end of the bar
            xytext=(0, space),          # Vertically shift label by `space`
            textcoords="offset points", # Interpret `xytext` as offset in points
            ha="center",                # Horizontally center label
            va=va)                      # Vertically align label differently for
                                        # positive and negative values.

# Call the function above. All the magic happens there.


Edit: I have extracted the relevant functionality in a function, as suggested by barnhillec.

This produces the following output:

Bar chart with automatically placed labels on each bar

And with logarithmic scale (and some adjustment to the input data to showcase logarithmic scaling), this is the result:

Bar chart with logarithmic scale with automatically placed labels on each bar

Answer #4

if you want to change the base of logarithm, just add:


Before Matplotlib 3.3, you would have to use basex/basey as the bases of log

Answer #5

Short answer: use not set(a).isdisjoint(b), it"s generally the fastest.

There are four common ways to test if two lists a and b share any items. The first option is to convert both to sets and check their intersection, as such:

bool(set(a) & set(b))

Because sets are stored using a hash table in Python, searching them is O(1) (see here for more information about complexity of operators in Python). Theoretically, this is O(n+m) on average for n and m objects in lists a and b. But 1) it must first create sets out of the lists, which can take a non-negligible amount of time, and 2) it supposes that hashing collisions are sparse among your data.

The second way to do it is using a generator expression performing iteration on the lists, such as:

any(i in a for i in b)

This allows to search in-place, so no new memory is allocated for intermediary variables. It also bails out on the first find. But the in operator is always O(n) on lists (see here).

Another proposed option is an hybridto iterate through one of the list, convert the other one in a set and test for membership on this set, like so:

a = set(a); any(i in a for i in b)

A fourth approach is to take advantage of the isdisjoint() method of the (frozen)sets (see here), for example:

not set(a).isdisjoint(b)

If the elements you search are near the beginning of an array (e.g. it is sorted), the generator expression is favored, as the sets intersection method have to allocate new memory for the intermediary variables:

from timeit import timeit
>>> timeit("bool(set(a) & set(b))", setup="a=list(range(1000));b=list(range(1000))", number=100000)
>>> timeit("any(i in a for i in b)", setup="a=list(range(1000));b=list(range(1000))", number=100000)

Here"s a graph of the execution time for this example in function of list size:

Element sharing test execution time when shared at the beginning

Note that both axes are logarithmic. This represents the best case for the generator expression. As can be seen, the isdisjoint() method is better for very small list sizes, whereas the generator expression is better for larger list sizes.

On the other hand, as the search begins with the beginning for the hybrid and generator expression, if the shared element are systematically at the end of the array (or both lists does not share any values), the disjoint and set intersection approaches are then way faster than the generator expression and the hybrid approach.

>>> timeit("any(i in a for i in b)", setup="a=list(range(1000));b=[x+998 for x in range(999,0,-1)]", number=1000))

Element sharing test execution time when shared at the end

It is interesting to note that the generator expression is way slower for bigger list sizes. This is only for 1000 repetitions, instead of the 100000 for the previous figure. This setup also approximates well when when no elements are shared, and is the best case for the disjoint and set intersection approaches.

Here are two analysis using random numbers (instead of rigging the setup to favor one technique or another):

Element sharing test execution time for randomly generated data with high chance of sharing Element sharing test execution time for randomly generated data with high chance of sharing

High chance of sharing: elements are randomly taken from [1, 2*len(a)]. Low chance of sharing: elements are randomly taken from [1, 1000*len(a)].

Up to now, this analysis supposed both lists are of the same size. In case of two lists of different sizes, for example a is much smaller, isdisjoint() is always faster:

Element sharing test execution time on two differently sized lists when shared at the beginning Element sharing test execution time on two differently sized lists when shared at the end

Make sure that the a list is the smaller, otherwise the performance decreases. In this experiment, the a list size was set constant to 5.

In summary:

  • If the lists are very small (< 10 elements), not set(a).isdisjoint(b) is always the fastest.
  • If the elements in the lists are sorted or have a regular structure that you can take advantage of, the generator expression any(i in a for i in b) is the fastest on large list sizes;
  • Test the set intersection with not set(a).isdisjoint(b), which is always faster than bool(set(a) & set(b)).
  • The hybrid "iterate through list, test on set" a = set(a); any(i in a for i in b) is generally slower than other methods.
  • The generator expression and the hybrid are much slower than the two other approaches when it comes to lists without sharing elements.

In most cases, using the isdisjoint() method is the best approach as the generator expression will take much longer to execute, as it is very inefficient when no elements are shared.

Answer #6

You can do this with the code below, and the code in your question was actually very close to what you needed, all you have to do is call the cmap object you have.

import matplotlib

cmap ="Spectral")

rgba = cmap(0.5)
print(rgba) # (0.99807766255210428, 0.99923106502084169, 0.74602077638401709, 1.0)

For values outside of the range [0.0, 1.0] it will return the under and over colour (respectively). This, by default, is the minimum and maximum colour within the range (so 0.0 and 1.0). This default can be changed with cmap.set_under() and cmap.set_over().

For "special" numbers such as np.nan and np.inf the default is to use the 0.0 value, this can be changed using cmap.set_bad() similarly to under and over as above.

Finally it may be necessary for you to normalize your data such that it conforms to the range [0.0, 1.0]. This can be done using matplotlib.colors.Normalize simply as shown in the small example below where the arguments vmin and vmax describe what numbers should be mapped to 0.0 and 1.0 respectively.

import matplotlib

norm = matplotlib.colors.Normalize(vmin=10.0, vmax=20.0)

print(norm(15.0)) # 0.5

A logarithmic normaliser (matplotlib.colors.LogNorm) is also available for data ranges with a large range of values.

(Thanks to both Joe Kington and tcaswell for suggestions on how to improve the answer.)

Answer #7

For fitting y = A + B log x, just fit y against (log x).

>>> x = numpy.array([1, 7, 20, 50, 79])
>>> y = numpy.array([10, 19, 30, 35, 51])
>>> numpy.polyfit(numpy.log(x), y, 1)
array([ 8.46295607,  6.61867463])
# y ‚âà 8.46 log(x) + 6.62

For fitting y = AeBx, take the logarithm of both side gives log y = log A + Bx. So fit (log y) against x.

Note that fitting (log y) as if it is linear will emphasize small values of y, causing large deviation for large y. This is because polyfit (linear regression) works by minimizing ‚àëi (ŒîY)2 = ‚àëi (Yi≈∂i)2. When Yi = log yi, the residues ŒîYi = Œî(log yi) ‚âà Œîyi / |yi|. So even if polyfit makes a very bad decision for large y, the "divide-by-|y|" factor will compensate for it, causing polyfit favors small values.

This could be alleviated by giving each entry a "weight" proportional to y. polyfit supports weighted-least-squares via the w keyword argument.

>>> x = numpy.array([10, 19, 30, 35, 51])
>>> y = numpy.array([1, 7, 20, 50, 79])
>>> numpy.polyfit(x, numpy.log(y), 1)
array([ 0.10502711, -0.40116352])
#    y ‚âà exp(-0.401) * exp(0.105 * x) = 0.670 * exp(0.105 * x)
# (^ biased towards small values)
>>> numpy.polyfit(x, numpy.log(y), 1, w=numpy.sqrt(y))
array([ 0.06009446,  1.41648096])
#    y ‚âà exp(1.42) * exp(0.0601 * x) = 4.12 * exp(0.0601 * x)
# (^ not so biased)

Note that Excel, LibreOffice and most scientific calculators typically use the unweighted (biased) formula for the exponential regression / trend lines. If you want your results to be compatible with these platforms, do not include the weights even if it provides better results.

Now, if you can use scipy, you could use scipy.optimize.curve_fit to fit any model without transformations.

For y = A + B log x the result is the same as the transformation method:

>>> scipy.optimize.curve_fit(lambda t,a,b: a+b*numpy.log(t),  x,  y)
(array([ 6.61867467,  8.46295606]), 
 array([[ 28.15948002,  -7.89609542],
        [ -7.89609542,   2.9857172 ]]))
# y ‚âà 6.62 + 8.46 log(x)

For y = AeBx, however, we can get a better fit since it computes Œî(log y) directly. But we need to provide an initialize guess so curve_fit can reach the desired local minimum.

>>> scipy.optimize.curve_fit(lambda t,a,b: a*numpy.exp(b*t),  x,  y)
(array([  5.60728326e-21,   9.99993501e-01]),
 array([[  4.14809412e-27,  -1.45078961e-08],
        [ -1.45078961e-08,   5.07411462e+10]]))
# oops, definitely wrong.
>>> scipy.optimize.curve_fit(lambda t,a,b: a*numpy.exp(b*t),  x,  y,  p0=(4, 0.1))
(array([ 4.88003249,  0.05531256]),
 array([[  1.01261314e+01,  -4.31940132e-02],
        [ -4.31940132e-02,   1.91188656e-04]]))
# y ‚âà 4.88 exp(0.0553 x). much better.

comparison of exponential regression

Answer #8

It"s good to know that

log_b(a) = log(a)/log(b)

but also know that math.log takes an optional second argument which allows you to specify the base:

In [22]: import math

In [23]: math.log?
Type:       builtin_function_or_method
Base Class: <type "builtin_function_or_method">
String Form:    <built-in function log>
Namespace:  Interactive
    log(x[, base]) -> the logarithm of x to the given base.
    If the base not specified, returns the natural logarithm (base e) of x.

In [25]: math.log(8,2)
Out[25]: 3.0

Answer #9

A naive algorithm won"t give good results when applied to real-world data. Here is a 20-line algorithm that exploits relative word frequency to give accurate results for real-word text.

(If you want an answer to your original question which does not use word frequency, you need to refine what exactly is meant by "longest word": is it better to have a 20-letter word and ten 3-letter words, or is it better to have five 10-letter words? Once you settle on a precise definition, you just have to change the line defining wordcost to reflect the intended meaning.)

The idea

The best way to proceed is to model the distribution of the output. A good first approximation is to assume all words are independently distributed. Then you only need to know the relative frequency of all words. It is reasonable to assume that they follow Zipf"s law, that is the word with rank n in the list of words has probability roughly 1/(n log N) where N is the number of words in the dictionary.

Once you have fixed the model, you can use dynamic programming to infer the position of the spaces. The most likely sentence is the one that maximizes the product of the probability of each individual word, and it"s easy to compute it with dynamic programming. Instead of directly using the probability we use a cost defined as the logarithm of the inverse of the probability to avoid overflows.

The code

from math import log

# Build a cost dictionary, assuming Zipf"s law and cost = -math.log(probability).
words = open("words-by-frequency.txt").read().split()
wordcost = dict((k, log((i+1)*log(len(words)))) for i,k in enumerate(words))
maxword = max(len(x) for x in words)

def infer_spaces(s):
    """Uses dynamic programming to infer the location of spaces in a string
    without spaces."""

    # Find the best match for the i first characters, assuming cost has
    # been built for the i-1 first characters.
    # Returns a pair (match_cost, match_length).
    def best_match(i):
        candidates = enumerate(reversed(cost[max(0, i-maxword):i]))
        return min((c + wordcost.get(s[i-k-1:i], 9e999), k+1) for k,c in candidates)

    # Build the cost array.
    cost = [0]
        c,k = best_match(i)

    # Backtrack to recover the minimal-cost string.
    out = []
    i = len(s)
    while i>0:
        c,k = best_match(i)
        assert c == cost[i]
        i -= k

    return " ".join(reversed(out))

which you can use with

s = "thumbgreenappleactiveassignmentweeklymetaphor"

The results

I am using this quick-and-dirty 125k-word dictionary I put together from a small subset of Wikipedia.

Before: thumbgreenappleactiveassignmentweeklymetaphor.
After: thumb green apple active assignment weekly metaphor.

Before: thereismassesoftextinformationofpeoplescommentswhichisparsedfromhtmlbuttherearen odelimitedcharactersinthemforexamplethumbgreenappleactiveassignmentweeklymetapho rapparentlytherearethumbgreenappleetcinthestringialsohavealargedictionarytoquery whetherthewordisreasonablesowhatsthefastestwayofextractionthxalot.

After: there is masses of text information of peoples comments which is parsed from html but there are no delimited characters in them for example thumb green apple active assignment weekly metaphor apparently there are thumb green apple etc in the string i also have a large dictionary to query whether the word is reasonable so what s the fastest way of extraction thx a lot.

Before: itwasadarkandstormynighttherainfellintorrentsexceptatoccasionalintervalswhenitwascheckedbyaviolentgustofwindwhichsweptupthestreetsforitisinlondonthatoursceneliesrattlingalongthehousetopsandfiercelyagitatingthescantyflameofthelampsthatstruggledagainstthedarkness.

After: it was a dark and stormy night the rain fell in torrents except at occasional intervals when it was checked by a violent gust of wind which swept up the streets for it is in london that our scene lies rattling along the housetops and fiercely agitating the scanty flame of the lamps that struggled against the darkness.

As you can see it is essentially flawless. The most important part is to make sure your word list was trained to a corpus similar to what you will actually encounter, otherwise the results will be very bad.


The implementation consumes a linear amount of time and memory, so it is reasonably efficient. If you need further speedups, you can build a suffix tree from the word list to reduce the size of the set of candidates.

If you need to process a very large consecutive string it would be reasonable to split the string to avoid excessive memory usage. For example you could process the text in blocks of 10000 characters plus a margin of 1000 characters on either side to avoid boundary effects. This will keep memory usage to a minimum and will have almost certainly no effect on the quality.

Answer #10

I finally found some time to do some experiments in order to understand the difference between them. Here"s what I discovered:

  • log only allows positive values, and lets you choose how to handle negative ones (mask or clip).
  • symlog means symmetrical log, and allows positive and negative values.
  • symlog allows to set a range around zero within the plot will be linear instead of logarithmic.

I think everything will get a lot easier to understand with graphics and examples, so let"s try them:

import numpy
from matplotlib import pyplot

# Enable interactive mode

# Draw the grid lines

# Numbers from -50 to 50, with 0.1 as step
xdomain = numpy.arange(-50,50, 0.1)

# Plots a simple linear function "f(x) = x"
pyplot.plot(xdomain, xdomain)
# Plots "sin(x)"
pyplot.plot(xdomain, numpy.sin(xdomain))

# "linear" is the default mode, so this next line is redundant:

A graph using 'linear' scaling

# How to treat negative values?
# "mask" will treat negative values as invalid
# "mask" is the default, so the next two lines are equivalent
pyplot.xscale("log", nonposx="mask")

A graph using 'log' scaling and nonposx='mask'

# "clip" will map all negative values a very small positive one
pyplot.xscale("log", nonposx="clip")

A graph using 'log' scaling and nonposx='clip'

# "symlog" scaling, however, handles negative values nicely

A graph using 'symlog' scaling

# And you can even set a linear range around zero
pyplot.xscale("symlog", linthreshx=20)

A graph using 'symlog' scaling, but linear within (-20,20)

Just for completeness, I"ve used the following code to save each figure:

# Default dpi is 80
pyplot.savefig("matplotlib_xscale_linear.png", dpi=50, bbox_inches="tight")

Remember you can change the figure size using:

fig = pyplot.gcf()
fig.set_size_inches([4., 3.])
# Default size: [8., 6.]

(If you are unsure about me answering my own question, read this)


How to get all subsets of a set? (powerset)

Given a set

{0, 1, 2, 3}

How can I produce the subsets:

 {0, 1},
 {0, 2},
 {0, 3},
 {1, 2},
 {1, 3},
 {2, 3},
 {0, 1, 2},
 {0, 1, 3},
 {0, 2, 3},
 {1, 2, 3},
 {0, 1, 2, 3}]

Answer #1

You have four main options for converting types in pandas:






1. to_numeric()



Basic usage


>>> s
0      8
1      6
2    7.5
3      3
4    0.9
dtype: object

0    8.0
1    6.0
2    7.5
3    3.0
4    0.9
dtype: float64


# convert Series
my_series = pd.to_numeric(my_series)

# convert column "a" of a DataFrame


# convert all columns of DataFrame
df = df.apply(pd.to_numeric) # convert all columns of DataFrame

# convert just columns "a" and "b"


Error handling




>>> s
0         1
1         2
2       4.7
3    pandas
4        10
dtype: object


ValueError: Unable to parse string


0     1.0
1     2.0
2     4.7
3     NaN
4    10.0
dtype: float64


# the original Series is returned untouched


df.apply(pd.to_numeric, errors="ignore")






>>> s
0    1
1    2
2   -7
dtype: int64


0    1
1    2
2   -7
dtype: int8


0    1.0
1    2.0
2   -7.0
dtype: float32

2. astype()


Basic usage



# convert all DataFrame columns to the int64 dtype
df = df.astype(int)


# convert Series to float16 type
s = s.astype(np.float16)

# convert Series to Python strings
s = s.astype(str)

# convert Series to categorical type - see docs for more details
s = s.astype("category")



Be careful


>>> s
0    1
1    2
2   -7
dtype: int64


>>> s.astype(np.uint8)
0      1
1      2
2    249
dtype: uint8



3. infer_objects()



>>> df.dtypes
a    object
b    object
dtype: object


>>> df = df.infer_objects()
>>> df.dtypes
a     int64
b    object
dtype: object


4. convert_dtypes()



With our object DataFrame df, we get the following result:

>>> df.convert_dtypes().dtypes                                             
a     Int64
b    string
dtype: object




>>> df.convert_dtypes(infer_objects=False).dtypes                          
a    object
b    string
dtype: object


Answer #2

There are two types of site-packages directories, global and per user.

  1. Global site-packages ("dist-packages") directories are listed in sys.path when you run:

    python -m site

    For a more concise list run getsitepackages from the site module in Python code:

    python -c "import site; print(site.getsitepackages())"

    Note: With virtualenvs getsitepackages is not available, sys.path from above will list the virtualenv"s site-packages directory correctly, though. In Python 3, you may use the sysconfig module instead:

    python3 -c "import sysconfig; print(sysconfig.get_paths()["purelib"])"
  2. The per user site-packages directory (PEP 370) is where Python installs your local packages:

    python -m site --user-site

    If this points to a non-existing directory check the exit status of Python and see python -m site --help for explanations.

    Hint: Running pip list --user or pip freeze --user gives you a list of all installed per user site-packages.

Practical Tips

  • <package>.__path__ lets you identify the location(s) of a specific package: (details)

    $ python -c "import setuptools as _; print(_.__path__)"
  • <module>.__file__ lets you identify the location of a specific module: (difference)

    $ python3 -c "import os as _; print(_.__file__)"
  • Run pip show <package> to show Debian-style package information:

    $ pip show pytest
    Name: pytest
    Version: 3.8.2
    Summary: pytest: simple powerful testing with Python
    Author: Holger Krekel, Bruno Oliveira, Ronny Pfannschmidt, Floris Bruynooghe, Brianna Laugher, Florian Bruhin and others
    Author-email: None
    License: MIT license
    Location: /home/peter/.local/lib/python3.4/site-packages
    Requires: more-itertools, atomicwrites, setuptools, attrs, pathlib2, six, py, pluggy

Answer #3

TL;DR version:

For the simple case of:

  • I have a text column with a delimiter and I want two columns

The simplest solution is:

df[["A", "B"]] = df["AB"].str.split(" ", 1, expand=True)

You must use expand=True if your strings have a non-uniform number of splits and you want None to replace the missing values.

Notice how, in either case, the .tolist() method is not necessary. Neither is zip().

In detail:

Andy Hayden"s solution is most excellent in demonstrating the power of the str.extract() method.

But for a simple split over a known separator (like, splitting by dashes, or splitting by whitespace), the .str.split() method is enough1. It operates on a column (Series) of strings, and returns a column (Series) of lists:

>>> import pandas as pd
>>> df = pd.DataFrame({"AB": ["A1-B1", "A2-B2"]})
>>> df

0  A1-B1
1  A2-B2
>>> df["AB_split"] = df["AB"].str.split("-")
>>> df

      AB  AB_split
0  A1-B1  [A1, B1]
1  A2-B2  [A2, B2]

1: If you"re unsure what the first two parameters of .str.split() do, I recommend the docs for the plain Python version of the method.

But how do you go from:

  • a column containing two-element lists


  • two columns, each containing the respective element of the lists?

Well, we need to take a closer look at the .str attribute of a column.

It"s a magical object that is used to collect methods that treat each element in a column as a string, and then apply the respective method in each element as efficient as possible:

>>> upper_lower_df = pd.DataFrame({"U": ["A", "B", "C"]})
>>> upper_lower_df

0  A
1  B
2  C
>>> upper_lower_df["L"] = upper_lower_df["U"].str.lower()
>>> upper_lower_df

   U  L
0  A  a
1  B  b
2  C  c

But it also has an "indexing" interface for getting each element of a string by its index:

>>> df["AB"].str[0]

0    A
1    A
Name: AB, dtype: object

>>> df["AB"].str[1]

0    1
1    2
Name: AB, dtype: object

Of course, this indexing interface of .str doesn"t really care if each element it"s indexing is actually a string, as long as it can be indexed, so:

>>> df["AB"].str.split("-", 1).str[0]

0    A1
1    A2
Name: AB, dtype: object


0    B1
1    B2
Name: AB, dtype: object

Then, it"s a simple matter of taking advantage of the Python tuple unpacking of iterables to do

>>> df["A"], df["B"] = df["AB"].str.split("-", 1).str
>>> df

      AB  AB_split   A   B
0  A1-B1  [A1, B1]  A1  B1
1  A2-B2  [A2, B2]  A2  B2

Of course, getting a DataFrame out of splitting a column of strings is so useful that the .str.split() method can do it for you with the expand=True parameter:

>>> df["AB"].str.split("-", 1, expand=True)

    0   1
0  A1  B1
1  A2  B2

So, another way of accomplishing what we wanted is to do:

>>> df = df[["AB"]]
>>> df

0  A1-B1
1  A2-B2

>>> df.join(df["AB"].str.split("-", 1, expand=True).rename(columns={0:"A", 1:"B"}))

      AB   A   B
0  A1-B1  A1  B1
1  A2-B2  A2  B2

The expand=True version, although longer, has a distinct advantage over the tuple unpacking method. Tuple unpacking doesn"t deal well with splits of different lengths:

>>> df = pd.DataFrame({"AB": ["A1-B1", "A2-B2", "A3-B3-C3"]})
>>> df
0     A1-B1
1     A2-B2
2  A3-B3-C3
>>> df["A"], df["B"], df["C"] = df["AB"].str.split("-")
Traceback (most recent call last):
ValueError: Length of values does not match length of index

But expand=True handles it nicely by placing None in the columns for which there aren"t enough "splits":

>>> df.join(
...     df["AB"].str.split("-", expand=True).rename(
...         columns={0:"A", 1:"B", 2:"C"}
...     )
... )
         AB   A   B     C
0     A1-B1  A1  B1  None
1     A2-B2  A2  B2  None
2  A3-B3-C3  A3  B3    C3

Answer #4

If you haven"t got python installed along with all the node-gyp dependencies, simply open Powershell or Git Bash with administrator privileges and execute:


and then to install the package:

npm install --global node-gyp

once installed, you will have all the node-gyp dependencies downloaded, but you still need the environment variable. Validate Python is indeed found in the correct folder:


*Note - it uses python 2.7 not 3.x as it is not supported*

If it doesn"t moan, go ahead and create your (user) environment variable:

setx PYTHON ""

restart cmd, and verify the variable exists via set PYTHON which should return the variable ($env:PYTHON if using Powershell)

Lastly re-apply npm install <module>

Answer #5

How do I determine the size of an object in Python?

The answer, "Just use sys.getsizeof", is not a complete answer.

That answer does work for builtin objects directly, but it does not account for what those objects may contain, specifically, what types, such as custom objects, tuples, lists, dicts, and sets contain. They can contain instances each other, as well as numbers, strings and other objects.

A More Complete Answer

Using 64-bit Python 3.6 from the Anaconda distribution, with sys.getsizeof, I have determined the minimum size of the following objects, and note that sets and dicts preallocate space so empty ones don"t grow again until after a set amount (which may vary by implementation of the language):

Python 3:

Bytes  type        scaling notes
28     int         +4 bytes about every 30 powers of 2
37     bytes       +1 byte per additional byte
49     str         +1-4 per additional character (depending on max width)
48     tuple       +8 per additional item
64     list        +8 for each additional
224    set         5th increases to 736; 21nd, 2272; 85th, 8416; 341, 32992
240    dict        6th increases to 368; 22nd, 1184; 43rd, 2280; 86th, 4704; 171st, 9320
136    func def    does not include default args and other attrs
1056   class def   no slots 
56     class inst  has a __dict__ attr, same scaling as dict above
888    class def   with slots
16     __slots__   seems to store in mutable tuple-like structure
                   first slot grows to 48, and so on.

How do you interpret this? Well say you have a set with 10 items in it. If each item is 100 bytes each, how big is the whole data structure? The set is 736 itself because it has sized up one time to 736 bytes. Then you add the size of the items, so that"s 1736 bytes in total

Some caveats for function and class definitions:

Note each class definition has a proxy __dict__ (48 bytes) structure for class attrs. Each slot has a descriptor (like a property) in the class definition.

Slotted instances start out with 48 bytes on their first element, and increase by 8 each additional. Only empty slotted objects have 16 bytes, and an instance with no data makes very little sense.

Also, each function definition has code objects, maybe docstrings, and other possible attributes, even a __dict__.

Also note that we use sys.getsizeof() because we care about the marginal space usage, which includes the garbage collection overhead for the object, from the docs:

getsizeof() calls the object‚Äôs __sizeof__ method and adds an additional garbage collector overhead if the object is managed by the garbage collector.

Also note that resizing lists (e.g. repetitively appending to them) causes them to preallocate space, similarly to sets and dicts. From the listobj.c source code:

    /* This over-allocates proportional to the list size, making room
     * for additional growth.  The over-allocation is mild, but is
     * enough to give linear-time amortized behavior over a long
     * sequence of appends() in the presence of a poorly-performing
     * system realloc().
     * The growth pattern is:  0, 4, 8, 16, 25, 35, 46, 58, 72, 88, ...
     * Note: new_allocated won"t overflow because the largest possible value
     *       is PY_SSIZE_T_MAX * (9 / 8) + 6 which always fits in a size_t.
    new_allocated = (size_t)newsize + (newsize >> 3) + (newsize < 9 ? 3 : 6);

Historical data

Python 2.7 analysis, confirmed with guppy.hpy and sys.getsizeof:

Bytes  type        empty + scaling notes
24     int         NA
28     long        NA
37     str         + 1 byte per additional character
52     unicode     + 4 bytes per additional character
56     tuple       + 8 bytes per additional item
72     list        + 32 for first, 8 for each additional
232    set         sixth item increases to 744; 22nd, 2280; 86th, 8424
280    dict        sixth item increases to 1048; 22nd, 3352; 86th, 12568 *
120    func def    does not include default args and other attrs
64     class inst  has a __dict__ attr, same scaling as dict above
16     __slots__   class with slots has no dict, seems to store in 
                    mutable tuple-like structure.
904    class def   has a proxy __dict__ structure for class attrs
104    old class   makes sense, less stuff, has real dict though.

Note that dictionaries (but not sets) got a more compact representation in Python 3.6

I think 8 bytes per additional item to reference makes a lot of sense on a 64 bit machine. Those 8 bytes point to the place in memory the contained item is at. The 4 bytes are fixed width for unicode in Python 2, if I recall correctly, but in Python 3, str becomes a unicode of width equal to the max width of the characters.

And for more on slots, see this answer.

A More Complete Function

We want a function that searches the elements in lists, tuples, sets, dicts, obj.__dict__"s, and obj.__slots__, as well as other things we may not have yet thought of.

We want to rely on gc.get_referents to do this search because it works at the C level (making it very fast). The downside is that get_referents can return redundant members, so we need to ensure we don"t double count.

Classes, modules, and functions are singletons - they exist one time in memory. We"re not so interested in their size, as there"s not much we can do about them - they"re a part of the program. So we"ll avoid counting them if they happen to be referenced.

We"re going to use a blacklist of types so we don"t include the entire program in our size count.

import sys
from types import ModuleType, FunctionType
from gc import get_referents

# Custom objects know their class.
# Function objects seem to know way too much, including modules.
# Exclude modules as well.
BLACKLIST = type, ModuleType, FunctionType

def getsize(obj):
    """sum size of object & members."""
    if isinstance(obj, BLACKLIST):
        raise TypeError("getsize() does not take argument of type: "+ str(type(obj)))
    seen_ids = set()
    size = 0
    objects = [obj]
    while objects:
        need_referents = []
        for obj in objects:
            if not isinstance(obj, BLACKLIST) and id(obj) not in seen_ids:
                size += sys.getsizeof(obj)
        objects = get_referents(*need_referents)
    return size

To contrast this with the following whitelisted function, most objects know how to traverse themselves for the purposes of garbage collection (which is approximately what we"re looking for when we want to know how expensive in memory certain objects are. This functionality is used by gc.get_referents.) However, this measure is going to be much more expansive in scope than we intended if we are not careful.

For example, functions know quite a lot about the modules they are created in.

Another point of contrast is that strings that are keys in dictionaries are usually interned so they are not duplicated. Checking for id(key) will also allow us to avoid counting duplicates, which we do in the next section. The blacklist solution skips counting keys that are strings altogether.

Whitelisted Types, Recursive visitor

To cover most of these types myself, instead of relying on the gc module, I wrote this recursive function to try to estimate the size of most Python objects, including most builtins, types in the collections module, and custom types (slotted and otherwise).

This sort of function gives much more fine-grained control over the types we"re going to count for memory usage, but has the danger of leaving important types out:

import sys
from numbers import Number
from collections import deque
from import Set, Mapping

ZERO_DEPTH_BASES = (str, bytes, Number, range, bytearray)

def getsize(obj_0):
    """Recursively iterate to sum size of object & members."""
    _seen_ids = set()
    def inner(obj):
        obj_id = id(obj)
        if obj_id in _seen_ids:
            return 0
        size = sys.getsizeof(obj)
        if isinstance(obj, ZERO_DEPTH_BASES):
            pass # bypass remaining control flow and return
        elif isinstance(obj, (tuple, list, Set, deque)):
            size += sum(inner(i) for i in obj)
        elif isinstance(obj, Mapping) or hasattr(obj, "items"):
            size += sum(inner(k) + inner(v) for k, v in getattr(obj, "items")())
        # Check for custom object instances - may subclass above too
        if hasattr(obj, "__dict__"):
            size += inner(vars(obj))
        if hasattr(obj, "__slots__"): # can have __slots__ with __dict__
            size += sum(inner(getattr(obj, s)) for s in obj.__slots__ if hasattr(obj, s))
        return size
    return inner(obj_0)

And I tested it rather casually (I should unittest it):

>>> getsize(["a", tuple("bcd"), Foo()])
>>> getsize(Foo())
>>> getsize(tuple("bcd"))
>>> getsize(["a", tuple("bcd"), Foo(), {"foo": "bar", "baz": "bar"}])
>>> getsize({"foo": "bar", "baz": "bar"})
>>> getsize({})
>>> getsize({"foo":"bar"})
>>> getsize("foo")
>>> class Bar():
...     def baz():
...         pass
>>> getsize(Bar())
>>> getsize(Bar().__dict__)
>>> sys.getsizeof(Bar())
>>> getsize(Bar.__dict__)
>>> sys.getsizeof(Bar.__dict__)

This implementation breaks down on class definitions and function definitions because we don"t go after all of their attributes, but since they should only exist once in memory for the process, their size really doesn"t matter too much.

Answer #6

To somewhat expand on the earlier answers here, there are a number of details which are commonly overlooked.

  • Prefer over subprocess.check_call() and friends over over subprocess.Popen() over os.system() over os.popen()
  • Understand and probably use text=True, aka universal_newlines=True.
  • Understand the meaning of shell=True or shell=False and how it changes quoting and the availability of shell conveniences.
  • Understand differences between sh and Bash
  • Understand how a subprocess is separate from its parent, and generally cannot change the parent.
  • Avoid running the Python interpreter as a subprocess of Python.

These topics are covered in some more detail below.

Prefer or subprocess.check_call()

The subprocess.Popen() function is a low-level workhorse but it is tricky to use correctly and you end up copy/pasting multiple lines of code ... which conveniently already exist in the standard library as a set of higher-level wrapper functions for various purposes, which are presented in more detail in the following.

Here"s a paragraph from the documentation:


Unfortunately, the availability of these wrapper functions differs between Python versions.

  • was officially introduced in Python 3.5. It is meant to replace all of the following.
  • subprocess.check_output() was introduced in Python 2.7 / 3.1. It is basically equivalent to, check=True, stdout=subprocess.PIPE).stdout
  • subprocess.check_call() was introduced in Python 2.5. It is basically equivalent to, check=True)
  • was introduced in Python 2.4 in the original subprocess module (PEP-324). It is basically equivalent to

High-level API vs subprocess.Popen()

The refactored and extended is more logical and more versatile than the older legacy functions it replaces. It returns a CompletedProcess object which has various methods which allow you to retrieve the exit status, the standard output, and a few other results and status indicators from the finished subprocess. is the way to go if you simply need a program to run and return control to Python. For more involved scenarios (background processes, perhaps with interactive I/O with the Python parent program) you still need to use subprocess.Popen() and take care of all the plumbing yourself. This requires a fairly intricate understanding of all the moving parts and should not be undertaken lightly. The simpler Popen object represents the (possibly still-running) process which needs to be managed from your code for the remainder of the lifetime of the subprocess.

It should perhaps be emphasized that just subprocess.Popen() merely creates a process. If you leave it at that, you have a subprocess running concurrently alongside with Python, so a "background" process. If it doesn"t need to do input or output or otherwise coordinate with you, it can do useful work in parallel with your Python program.

Avoid os.system() and os.popen()

Since time eternal (well, since Python 2.5) the os module documentation has contained the recommendation to prefer subprocess over os.system():


The problems with system() are that it"s obviously system-dependent and doesn"t offer ways to interact with the subprocess. It simply runs, with standard output and standard error outside of Python"s reach. The only information Python receives back is the exit status of the command (zero means success, though the meaning of non-zero values is also somewhat system-dependent).

PEP-324 (which was already mentioned above) contains a more detailed rationale for why os.system is problematic and how subprocess attempts to solve those issues.

os.popen() used to be even more strongly discouraged:

Deprecated since version 2.6: This function is obsolete. Use the subprocess module.

However, since sometime in Python 3, it has been reimplemented to simply use subprocess, and redirects to the subprocess.Popen() documentation for details.

Understand and usually use check=True

You"ll also notice that has many of the same limitations as os.system(). In regular use, you should generally check whether the process finished successfully, which subprocess.check_call() and subprocess.check_output() do (where the latter also returns the standard output of the finished subprocess). Similarly, you should usually use check=True with unless you specifically need to allow the subprocess to return an error status.

In practice, with check=True or subprocess.check_*, Python will throw a CalledProcessError exception if the subprocess returns a nonzero exit status.

A common error with is to omit check=True and be surprised when downstream code fails if the subprocess failed.

On the other hand, a common problem with check_call() and check_output() was that users who blindly used these functions were surprised when the exception was raised e.g. when grep did not find a match. (You should probably replace grep with native Python code anyway, as outlined below.)

All things counted, you need to understand how shell commands return an exit code, and under what conditions they will return a non-zero (error) exit code, and make a conscious decision how exactly it should be handled.

Understand and probably use text=True aka universal_newlines=True

Since Python 3, strings internal to Python are Unicode strings. But there is no guarantee that a subprocess generates Unicode output, or strings at all.

(If the differences are not immediately obvious, Ned Batchelder"s Pragmatic Unicode is recommended, if not outright obligatory, reading. There is a 36-minute video presentation behind the link if you prefer, though reading the page yourself will probably take significantly less time.)

Deep down, Python has to fetch a bytes buffer and interpret it somehow. If it contains a blob of binary data, it shouldn"t be decoded into a Unicode string, because that"s error-prone and bug-inducing behavior - precisely the sort of pesky behavior which riddled many Python 2 scripts, before there was a way to properly distinguish between encoded text and binary data.

With text=True, you tell Python that you, in fact, expect back textual data in the system"s default encoding, and that it should be decoded into a Python (Unicode) string to the best of Python"s ability (usually UTF-8 on any moderately up to date system, except perhaps Windows?)

If that"s not what you request back, Python will just give you bytes strings in the stdout and stderr strings. Maybe at some later point you do know that they were text strings after all, and you know their encoding. Then, you can decode them.

normal =[external, arg],
    stdout=subprocess.PIPE, stderr=subprocess.PIPE,

convoluted =[external, arg],
    stdout=subprocess.PIPE, stderr=subprocess.PIPE,
# You have to know (or guess) the encoding

Python 3.7 introduced the shorter and more descriptive and understandable alias text for the keyword argument which was previously somewhat misleadingly called universal_newlines.

Understand shell=True vs shell=False

With shell=True you pass a single string to your shell, and the shell takes it from there.

With shell=False you pass a list of arguments to the OS, bypassing the shell.

When you don"t have a shell, you save a process and get rid of a fairly substantial amount of hidden complexity, which may or may not harbor bugs or even security problems.

On the other hand, when you don"t have a shell, you don"t have redirection, wildcard expansion, job control, and a large number of other shell features.

A common mistake is to use shell=True and then still pass Python a list of tokens, or vice versa. This happens to work in some cases, but is really ill-defined and could break in interesting ways.

buggy ="dig +short")

broken =["dig", "+short", ""],

pathological =["dig +short"],

correct =["dig", "+short", ""],
    # Probably don"t forget these, too
    check=True, text=True)

# XXX Probably better avoid shell=True
# but this is nominally correct
fixed_but_fugly ="dig +short",
    # Probably don"t forget these, too
    check=True, text=True)

The common retort "but it works for me" is not a useful rebuttal unless you understand exactly under what circumstances it could stop working.

Refactoring Example

Very often, the features of the shell can be replaced with native Python code. Simple Awk or sed scripts should probably simply be translated to Python instead.

To partially illustrate this, here is a typical but slightly silly example which involves many shell features.

cmd = """while read -r x;
   do ping -c 3 "$x" | grep "round-trip min/avg/max"
   done <hosts.txt"""

# Trivial but horrible
results =
    cmd, shell=True, universal_newlines=True, check=True)

# Reimplement with shell=False
with open("hosts.txt") as hosts:
    for host in hosts:
        host = host.rstrip("
")  # drop newline
        ping =
             ["ping", "-c", "3", host],
        for line in ping.stdout.split("
             if "round-trip min/avg/max" in line:
                 print("{}: {}".format(host, line))

Some things to note here:

  • With shell=False you don"t need the quoting that the shell requires around strings. Putting quotes anyway is probably an error.
  • It often makes sense to run as little code as possible in a subprocess. This gives you more control over execution from within your Python code.
  • Having said that, complex shell pipelines are tedious and sometimes challenging to reimplement in Python.

The refactored code also illustrates just how much the shell really does for you with a very terse syntax -- for better or for worse. Python says explicit is better than implicit but the Python code is rather verbose and arguably looks more complex than this really is. On the other hand, it offers a number of points where you can grab control in the middle of something else, as trivially exemplified by the enhancement that we can easily include the host name along with the shell command output. (This is by no means challenging to do in the shell, either, but at the expense of yet another diversion and perhaps another process.)

Common Shell Constructs

For completeness, here are brief explanations of some of these shell features, and some notes on how they can perhaps be replaced with native Python facilities.

  • Globbing aka wildcard expansion can be replaced with glob.glob() or very often with simple Python string comparisons like for file in os.listdir("."): if not file.endswith(".png"): continue. Bash has various other expansion facilities like .{png,jpg} brace expansion and {1..100} as well as tilde expansion (~ expands to your home directory, and more generally ~account to the home directory of another user)
  • Shell variables like $SHELL or $my_exported_var can sometimes simply be replaced with Python variables. Exported shell variables are available as e.g. os.environ["SHELL"] (the meaning of export is to make the variable available to subprocesses -- a variable which is not available to subprocesses will obviously not be available to Python running as a subprocess of the shell, or vice versa. The env= keyword argument to subprocess methods allows you to define the environment of the subprocess as a dictionary, so that"s one way to make a Python variable visible to a subprocess). With shell=False you will need to understand how to remove any quotes; for example, cd "$HOME" is equivalent to os.chdir(os.environ["HOME"]) without quotes around the directory name. (Very often cd is not useful or necessary anyway, and many beginners omit the double quotes around the variable and get away with it until one day ...)
  • Redirection allows you to read from a file as your standard input, and write your standard output to a file. grep "foo" <inputfile >outputfile opens outputfile for writing and inputfile for reading, and passes its contents as standard input to grep, whose standard output then lands in outputfile. This is not generally hard to replace with native Python code.
  • Pipelines are a form of redirection. echo foo | nl runs two subprocesses, where the standard output of echo is the standard input of nl (on the OS level, in Unix-like systems, this is a single file handle). If you cannot replace one or both ends of the pipeline with native Python code, perhaps think about using a shell after all, especially if the pipeline has more than two or three processes (though look at the pipes module in the Python standard library or a number of more modern and versatile third-party competitors).
  • Job control lets you interrupt jobs, run them in the background, return them to the foreground, etc. The basic Unix signals to stop and continue a process are of course available from Python, too. But jobs are a higher-level abstraction in the shell which involve process groups etc which you have to understand if you want to do something like this from Python.
  • Quoting in the shell is potentially confusing until you understand that everything is basically a string. So ls -l / is equivalent to "ls" "-l" "/" but the quoting around literals is completely optional. Unquoted strings which contain shell metacharacters undergo parameter expansion, whitespace tokenization and wildcard expansion; double quotes prevent whitespace tokenization and wildcard expansion but allow parameter expansions (variable substitution, command substitution, and backslash processing). This is simple in theory but can get bewildering, especially when there are several layers of interpretation (a remote shell command, for example).

Understand differences between sh and Bash

subprocess runs your shell commands with /bin/sh unless you specifically request otherwise (except of course on Windows, where it uses the value of the COMSPEC variable). This means that various Bash-only features like arrays, [[ etc are not available.

If you need to use Bash-only syntax, you can pass in the path to the shell as executable="/bin/bash" (where of course if your Bash is installed somewhere else, you need to adjust the path)."""
    # This for loop syntax is Bash only
        # Arrays are Bash-only
    shell=True, check=True,

A subprocess is separate from its parent, and cannot change it

A somewhat common mistake is doing something like"cd /tmp", shell=True)"pwd", shell=True)  # Oops, doesn"t print /tmp

The same thing will happen if the first subprocess tries to set an environment variable, which of course will have disappeared when you run another subprocess, etc.

A child process runs completely separate from Python, and when it finishes, Python has no idea what it did (apart from the vague indicators that it can infer from the exit status and output from the child process). A child generally cannot change the parent"s environment; it cannot set a variable, change the working directory, or, in so many words, communicate with its parent without cooperation from the parent.

The immediate fix in this particular case is to run both commands in a single subprocess;"cd /tmp; pwd", shell=True)

though obviously this particular use case isn"t very useful; instead, use the cwd keyword argument, or simply os.chdir() before running the subprocess. Similarly, for setting a variable, you can manipulate the environment of the current process (and thus also its children) via

os.environ["foo"] = "bar"

or pass an environment setting to a child process with"echo "$foo"", shell=True, env={"foo": "bar"})

(not to mention the obvious refactoring["echo", "bar"]); but echo is a poor example of something to run in a subprocess in the first place, of course).

Don"t run Python from Python

This is slightly dubious advice; there are certainly situations where it does make sense or is even an absolute requirement to run the Python interpreter as a subprocess from a Python script. But very frequently, the correct approach is simply to import the other Python module into your calling script and call its functions directly.

If the other Python script is under your control, and it isn"t a module, consider turning it into one. (This answer is too long already so I will not delve into details here.)

If you need parallelism, you can run Python functions in subprocesses with the multiprocessing module. There is also threading which runs multiple tasks in a single process (which is more lightweight and gives you more control, but also more constrained in that threads within a process are tightly coupled, and bound to a single GIL.)

Answer #7

To summarize and complement the existing answers:

  • python.exe is a console (terminal) application for launching CLI-type scripts (console applications).

    • Unless run from an existing console window, python.exe opens a new console window.

    • Standard streams sys.stdin, sys.stdout and sys.stderr are connected to the console window.

    • Execution is synchronous when launched from a cmd.exe or PowerShell console window: See eryksun"s 1st comment below.

      • If a new console window was created, it stays open until the script terminates.
      • When invoked from an existing console window, the prompt is blocked until the script terminates.
  • pythonw.exe is a GUI app for launching GUI/no-UI-at-all scripts.

    • NO console window is opened.
    • Execution is asynchronous:
      • When invoked from a console window, the script is merely launched and the prompt returns right away, whether the script is still running or not.
    • Standard streams sys.stdin, sys.stdout and sys.stderr are NOT available.
      • Caution: Unless you take extra steps, this has potentially unexpected side effects:
        • Unhandled exceptions cause the script to abort silently.
        • In Python 2.x, simply trying to use print() can cause that to happen (in 3.x, print() simply has no effect).
          • To prevent that from within your script, and to learn more, see this answer of mine.
          • Ad-hoc, you can use output redirection:Thanks, @handle.
            pythonw.exe yourScript.pyw 1>stdout.txt 2>stderr.txt
            (from PowerShell:
            cmd /c pythonw.exe yourScript.pyw 1>stdout.txt 2>stderr.txt) to capture stdout and stderr output in files.
            If you"re confident that use of print() is the only reason your script fails silently with pythonw.exe, and you"re not interested in stdout output, use @handle"s command from the comments:
            pythonw.exe yourScript.pyw 1>NUL 2>&1
            Caveat: This output redirection technique does not work when invoking *.pyw scripts directly (as opposed to by passing the script file path to pythonw.exe). See eryksun"s 2nd comment and its follow-ups below.

You can control which of the executables runs your script by default - such as when opened from Explorer - by choosing the right filename extension:

  • *.py files are by default associated (invoked) with python.exe
  • *.pyw files are by default associated (invoked) with pythonw.exe

Answer #8

The simple answer is because 3*0.1 != 0.3 due to quantization (roundoff) error (whereas 4*0.1 == 0.4 because multiplying by a power of two is usually an "exact" operation). Python tries to find the shortest string that would round to the desired value, so it can display 4*0.1 as 0.4 as these are equal, but it cannot display 3*0.1 as 0.3 because these are not equal.

You can use the .hex method in Python to view the internal representation of a number (basically, the exact binary floating point value, rather than the base-10 approximation). This can help to explain what"s going on under the hood.

>>> (0.1).hex()
>>> (0.3).hex()
>>> (0.1*3).hex()
>>> (0.4).hex()
>>> (0.1*4).hex()

0.1 is 0x1.999999999999a times 2^-4. The "a" at the end means the digit 10 - in other words, 0.1 in binary floating point is very slightly larger than the "exact" value of 0.1 (because the final 0x0.99 is rounded up to 0x0.a). When you multiply this by 4, a power of two, the exponent shifts up (from 2^-4 to 2^-2) but the number is otherwise unchanged, so 4*0.1 == 0.4.

However, when you multiply by 3, the tiny little difference between 0x0.99 and 0x0.a0 (0x0.07) magnifies into a 0x0.15 error, which shows up as a one-digit error in the last position. This causes 0.1*3 to be very slightly larger than the rounded value of 0.3.

Python 3"s float repr is designed to be round-trippable, that is, the value shown should be exactly convertible into the original value (float(repr(f)) == f for all floats f). Therefore, it cannot display 0.3 and 0.1*3 exactly the same way, or the two different numbers would end up the same after round-tripping. Consequently, Python 3"s repr engine chooses to display one with a slight apparent error.

Answer #9

I recommend anytree (I am the author).


from anytree import Node, RenderTree

udo = Node("Udo")
marc = Node("Marc", parent=udo)
lian = Node("Lian", parent=marc)
dan = Node("Dan", parent=udo)
jet = Node("Jet", parent=dan)
jan = Node("Jan", parent=dan)
joe = Node("Joe", parent=dan)


for pre, fill, node in RenderTree(udo):
    print("%s%s" % (pre,
‚îú‚îÄ‚îÄ Marc
‚îÇ   ‚îî‚îÄ‚îÄ Lian
‚îî‚îÄ‚îÄ Dan
    ‚îú‚îÄ‚îÄ Jet
    ‚îú‚îÄ‚îÄ Jan
    ‚îî‚îÄ‚îÄ Joe

(Node("/Udo/Dan/Jet"), Node("/Udo/Dan/Jan"), Node("/Udo/Dan/Joe"))

anytree has also a powerful API with:

  • simple tree creation
  • simple tree modification
  • pre-order tree iteration
  • post-order tree iteration
  • resolve relative and absolute node paths
  • walking from one node to an other.
  • tree rendering (see example above)
  • node attach/detach hookups

Answer #10

Distribution Fitting with Sum of Square Error (SSE)

This is an update and modification to Saullo"s answer, that uses the full list of the current scipy.stats distributions and returns the distribution with the least SSE between the distribution"s histogram and the data"s histogram.

Example Fitting

Using the El Ni√±o dataset from statsmodels, the distributions are fit and error is determined. The distribution with the least error is returned.

All Distributions

All Fitted Distributions

Best Fit Distribution

Best Fit Distribution

Example Code

%matplotlib inline

import warnings
import numpy as np
import pandas as pd
import scipy.stats as st
import statsmodels.api as sm
from scipy.stats._continuous_distns import _distn_names
import matplotlib
import matplotlib.pyplot as plt

matplotlib.rcParams["figure.figsize"] = (16.0, 12.0)"ggplot")

# Create models from data
def best_fit_distribution(data, bins=200, ax=None):
    """Model data by finding best fit distribution to data"""
    # Get histogram of original data
    y, x = np.histogram(data, bins=bins, density=True)
    x = (x + np.roll(x, -1))[:-1] / 2.0

    # Best holders
    best_distributions = []

    # Estimate distribution parameters from data
    for ii, distribution in enumerate([d for d in _distn_names if not d in ["levy_stable", "studentized_range"]]):

        print("{:>3} / {:<3}: {}".format( ii+1, len(_distn_names), distribution ))

        distribution = getattr(st, distribution)

        # Try to fit the distribution
            # Ignore warnings from data that can"t be fit
            with warnings.catch_warnings():
                # fit dist to data
                params =

                # Separate parts of parameters
                arg = params[:-2]
                loc = params[-2]
                scale = params[-1]
                # Calculate fitted PDF and error with fit in distribution
                pdf = distribution.pdf(x, loc=loc, scale=scale, *arg)
                sse = np.sum(np.power(y - pdf, 2.0))
                # if axis pass in add to plot
                    if ax:
                        pd.Series(pdf, x).plot(ax=ax)
                except Exception:

                # identify if this distribution is better
                best_distributions.append((distribution, params, sse))
        except Exception:

    return sorted(best_distributions, key=lambda x:x[2])

def make_pdf(dist, params, size=10000):
    """Generate distributions"s Probability Distribution Function """

    # Separate parts of parameters
    arg = params[:-2]
    loc = params[-2]
    scale = params[-1]

    # Get sane start and end points of distribution
    start = dist.ppf(0.01, *arg, loc=loc, scale=scale) if arg else dist.ppf(0.01, loc=loc, scale=scale)
    end = dist.ppf(0.99, *arg, loc=loc, scale=scale) if arg else dist.ppf(0.99, loc=loc, scale=scale)

    # Build PDF and turn into pandas Series
    x = np.linspace(start, end, size)
    y = dist.pdf(x, loc=loc, scale=scale, *arg)
    pdf = pd.Series(y, x)

    return pdf

# Load data from statsmodels datasets
data = pd.Series(sm.datasets.elnino.load_pandas().data.set_index("YEAR").values.ravel())

# Plot for comparison
ax = data.plot(kind="hist", bins=50, density=True, alpha=0.5, color=list(matplotlib.rcParams["axes.prop_cycle"])[1]["color"])

# Save plot limits
dataYLim = ax.get_ylim()

# Find best fit distribution
best_distibutions = best_fit_distribution(data, 200, ax)
best_dist = best_distibutions[0]

# Update plots
ax.set_title(u"El Ni√±o sea temp.
 All Fitted Distributions")
ax.set_xlabel(u"Temp (¬∞C)")

# Make PDF with best params 
pdf = make_pdf(best_dist[0], best_dist[1])

# Display
ax = pdf.plot(lw=2, label="PDF", legend=True)
data.plot(kind="hist", bins=50, density=True, alpha=0.5, label="Data", legend=True, ax=ax)

param_names = (best_dist[0].shapes + ", loc, scale").split(", ") if best_dist[0].shapes else ["loc", "scale"]
param_str = ", ".join(["{}={:0.2f}".format(k,v) for k,v in zip(param_names, best_dist[1])])
dist_str = "{}({})".format(best_dist[0].name, param_str)

ax.set_title(u"El Ni√±o sea temp. with best fit distribution 
" + dist_str)
ax.set_xlabel(u"Temp. (¬∞C)")