Logging from pyspark application to a local or hdfs file
<p>I have an application in pyspark includes closure functions that contain logging statements, I don't know how to log messages to local/hdfs file in pyspark.
I tried something as below but doesn't work: </p>

<pre><code>import json
from pyspark import SparkContext
import logging

def parse_json(text_line):
    try:
        return(json.loads(text_line))
    except ValueError:
        # here I need to log a warning message to a local file or even to default spark logs
        logger.warn("invalid json structure" + text_line)
        return({})

if __name__ == "__main__":
    my_data = ['{"id": "111", "name": "aaa"}',
               '{"wrong json", "name": "bbb"}',
               '{"id": "333", "name": "ccc"}']
    sc = SparkContext()
    logger = logging.getLogger('py4j')
    lines = sc.parallelize(my_data)
    my_data_json = lines.map(parse_json).filter(lambda x: x)
    print(my_data_json.collect())
</code></pre>

<p>Any help please!</p>