def listified_tokenizer ( source ) : io_obj = io . StringIO ( source ) return [ list ( a ) for a in tokenize . generate_tokens ( io_obj . readline ) ]
tokenize in python to get sentences
Tokenizes * source * and returns the tokens as a list of lists .
def listified_tokenizer(source):
    """Tokenizes *source* and returns the tokens as a list of lists."""
    io_obj = io.StringIO(source)
    return [list(a) for a in tokenize.generate_tokens(io_obj.readline)]