Getting log output from spark workers in google cloud
<p>I am running a spark job in a google cluster and I am trying to acquire some logging information during an RDD map process. Quick example:</p>

<pre><code>object LoggerWrapper extends Serializable{
    @transient lazy val logger=Logger.getLogger("myLogger")
}
object Processing{
 ...
    rdd.map(x=&gt;{
       LoggerWrapper.logger.info("processing:"+x)
       foo(x)
    })
   ...
  sparkContext.stop
 }
</code></pre>

<p>I am following the methodology described <a href="https://www.mapr.com/blog/how-log-apache-spark" rel="nofollow">here</a> combined with the directions found <a href="http://spark.apache.org/docs/latest/running-on-yarn.html" rel="nofollow">in the Spark webpage</a>. The resulting log4j.properties is the one shown at the end. The file is uploaded using the <code>--files</code> flag of the of <code>gcloud</code> command (shown below).  I have also updated the yarn-site.xml file so that the property <code>yarn.log-aggregation-enable</code> is set to <code>true</code>.</p>

<p>My first question is that when I run from the master node <code>yarn logs -application &lt;applicationID&gt;</code> I always get the error message "Log aggregation has not completed or is not enabled." Is there anything else that is necessary to be done to collect the messages.</p>

<p>The second question is if it is possible to get the log messages of all workers in the console output as the the process is running. For instance if the spark job is a streaming job, then I would like to get the messages while the job is running.</p>

<p>log4j.properties:</p>

<pre><code>log4j.appender.myConsoleAppender=org.apache.log4j.ConsoleAppender
log4j.appender.myConsoleAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.myConsoleAppender.layout.ConversionPattern=%d [%t] %-5p %c - %m%n
log4j.appender.RollingAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RollingAppender.File=${spark.yarn.app.container.log.dir}/spark.log
log4j.appender.RollingAppender.DatePattern='.'yyyy-MM-dd
log4j.appender.RollingAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.RollingAppender.layout.ConversionPattern=[%p] %d %c %M - %m%n

log4j.appender.RollingAppenderU=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RollingAppenderU.File=${spark.yarn.app.container.log.dir}/sparkU.log
log4j.appender.RollingAppenderU.DatePattern='.'yyyy-MM-dd
log4j.appender.RollingAppenderU.layout=org.apache.log4j.PatternLayout
log4j.appender.RollingAppenderU.layout.ConversionPattern=[%p] %d %c %M - %m%n


# By default, everything goes to console and file
log4j.rootLogger=INFO, RollingAppender, myConsoleAppender

# My custom logging goes to another file
log4j.logger.myLogger=INFO, RollingAppenderU, myConsoleAppender

# The noisier spark logs go to file only
log4j.logger.spark.storage=INFO, RollingAppender
log4j.additivity.spark.storage=false
log4j.logger.spark.scheduler=INFO, RollingAppender
log4j.additivity.spark.scheduler=false
log4j.logger.spark.CacheTracker=INFO, RollingAppender
log4j.additivity.spark.CacheTracker=false
log4j.logger.spark.CacheTrackerActor=INFO, RollingAppender
log4j.additivity.spark.CacheTrackerActor=false
log4j.logger.spark.MapOutputTrackerActor=INFO, RollingAppender
log4j.additivity.spark.MapOutputTrackerActor=false
log4j.logger.spark.MapOutputTracker=INFO, RollingAppender
log4j.additivty.spark.MapOutputTracker=false
</code></pre>

<p>gcloud command:
<code>gcloud dataproc jobs submit spark --cluster myCluster
--properties spark.driver.memory=1000m,spark.driver.maxResult=512m,spark.executor.memory=1000m --jars gs://path/to/jar/myJar.jar --files /absolute/path/to/local/file/log4j.properties 
--class contextual.wikidata.spark.jobs.$1 &lt;application-arguments&gt;</code></p>