{"action":"created","issue":{"url":"https://api.github.com/repos/snowplow/snowplow/issues/1954","labels_url":"https://api.github.com/repos/snowplow/snowplow/issues/1954/labels{/name}","comments_url":"https://api.github.com/repos/snowplow/snowplow/issues/1954/comments","events_url":"https://api.github.com/repos/snowplow/snowplow/issues/1954/events","html_url":"https://github.com/snowplow/snowplow/issues/1954","id":100567122,"number":1954,"title":"Scala Common Enrich: catch all exceptions in EtlPipeline","user":{"login":"fblundun","id":5892914,"avatar_url":"https://avatars.githubusercontent.com/u/5892914?v=3","gravatar_id":"","url":"https://api.github.com/users/fblundun","html_url":"https://github.com/fblundun","followers_url":"https://api.github.com/users/fblundun/followers","following_url":"https://api.github.com/users/fblundun/following{/other_user}","gists_url":"https://api.github.com/users/fblundun/gists{/gist_id}","starred_url":"https://api.github.com/users/fblundun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fblundun/subscriptions","organizations_url":"https://api.github.com/users/fblundun/orgs","repos_url":"https://api.github.com/users/fblundun/repos","events_url":"https://api.github.com/users/fblundun/events{/privacy}","received_events_url":"https://api.github.com/users/fblundun/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/snowplow/snowplow/labels/3.%20ETL","name":"3. ETL","color":"c3d69b"},{"url":"https://api.github.com/repos/snowplow/snowplow/labels/enhancement","name":"enhancement","color":"DDDDDD"}],"state":"open","locked":false,"assignee":{"login":"fblundun","id":5892914,"avatar_url":"https://avatars.githubusercontent.com/u/5892914?v=3","gravatar_id":"","url":"https://api.github.com/users/fblundun","html_url":"https://github.com/fblundun","followers_url":"https://api.github.com/users/fblundun/followers","following_url":"https://api.github.com/users/fblundun/following{/other_user}","gists_url":"https://api.github.com/users/fblundun/gists{/gist_id}","starred_url":"https://api.github.com/users/fblundun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fblundun/subscriptions","organizations_url":"https://api.github.com/users/fblundun/orgs","repos_url":"https://api.github.com/users/fblundun/repos","events_url":"https://api.github.com/users/fblundun/events{/privacy}","received_events_url":"https://api.github.com/users/fblundun/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/snowplow/snowplow/milestones/90","html_url":"https://github.com/snowplow/snowplow/milestones/Release%2071%20[HAD]%20Bird%20TBC","labels_url":"https://api.github.com/repos/snowplow/snowplow/milestones/90/labels","id":1173631,"number":90,"title":"Release 71 [HAD] Bird TBC","description":"","creator":{"login":"alexanderdean","id":200511,"avatar_url":"https://avatars.githubusercontent.com/u/200511?v=3","gravatar_id":"","url":"https://api.github.com/users/alexanderdean","html_url":"https://github.com/alexanderdean","followers_url":"https://api.github.com/users/alexanderdean/followers","following_url":"https://api.github.com/users/alexanderdean/following{/other_user}","gists_url":"https://api.github.com/users/alexanderdean/gists{/gist_id}","starred_url":"https://api.github.com/users/alexanderdean/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexanderdean/subscriptions","organizations_url":"https://api.github.com/users/alexanderdean/orgs","repos_url":"https://api.github.com/users/alexanderdean/repos","events_url":"https://api.github.com/users/alexanderdean/events{/privacy}","received_events_url":"https://api.github.com/users/alexanderdean/received_events","type":"User","site_admin":false},"open_issues":25,"closed_issues":4,"state":"open","created_at":"2015-06-20T19:26:59Z","updated_at":"2015-08-12T15:15:57Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2015-08-12T15:14:56Z","updated_at":"2015-08-12T16:04:49Z","closed_at":null,"body":"If we put a `try {...} catch {case NonFatal(e) => ...}` around the whole of the inside of the `EtlPipeline.processEvents` method, then we can catch any unexpected exceptions which our more fine-grained error handling misses. What do you think @alexanderdean ?"},"comment":{"url":"https://api.github.com/repos/snowplow/snowplow/issues/comments/130356433","html_url":"https://github.com/snowplow/snowplow/issues/1954#issuecomment-130356433","issue_url":"https://api.github.com/repos/snowplow/snowplow/issues/1954","id":130356433,"user":{"login":"fblundun","id":5892914,"avatar_url":"https://avatars.githubusercontent.com/u/5892914?v=3","gravatar_id":"","url":"https://api.github.com/users/fblundun","html_url":"https://github.com/fblundun","followers_url":"https://api.github.com/users/fblundun/followers","following_url":"https://api.github.com/users/fblundun/following{/other_user}","gists_url":"https://api.github.com/users/fblundun/gists{/gist_id}","starred_url":"https://api.github.com/users/fblundun/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fblundun/subscriptions","organizations_url":"https://api.github.com/users/fblundun/orgs","repos_url":"https://api.github.com/users/fblundun/repos","events_url":"https://api.github.com/users/fblundun/events{/privacy}","received_events_url":"https://api.github.com/users/fblundun/received_events","type":"User","site_admin":false},"created_at":"2015-08-12T16:04:49Z","updated_at":"2015-08-12T16:04:49Z","body":"I'm adding the whole stack trace to the message to make it possible to work out the source of the exception. Sample message for an artificially thrown NumberFormatException:\r\n\r\n```\r\n{\"line\":\"...\",\"errors\":[\"error: Unexpected error processing events: java.lang.NumberFormatException\\n\\tat com.snowplowanalytics.snowplow.enrich.common.enrichments.EnrichmentManager$.enrichEvent(EnrichmentManager.scala:69)\\n\\tat com.snowplowanalytics.snowplow.enrich.common.EtlPipeline$$anonfun$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3.apply(EtlPipeline.scala:94)\\n\\tat com.snowplowanalytics.snowplow.enrich.common.EtlPipeline$$anonfun$1$$anonfun$apply$1$$anonfun$apply$2$$anonfun$apply$3.apply(EtlPipeline.scala:93)\\n\\tat scalaz.NonEmptyList$class.map(NonEmptyList.scala:23)\\n\\tat scalaz.NonEmptyListFunctions$$anon$4.map(NonEmptyList.scala:196)\\n\\tat com.snowplowanalytics.snowplow.enrich.common.EtlPipeline$$anonfun$1$$anonfun$apply$1$$anonfun$apply$2.apply(EtlPipeline.scala:93)\\n\\tat com.snowplowanalytics.snowplow.enrich.common.EtlPipeline$$anonfun$1$$anonfun$apply$1$$anonfun$apply$2.apply(EtlPipeline.scala:91)\\n\\tat scalaz.Validation$class.map(Validation.scala:114)\\n\\tat scalaz.Success.map(Validation.scala:347)\\n\\tat com.snowplowanalytics.snowplow.enrich.common.EtlPipeline$$anonfun$1$$anonfun$apply$1.apply(EtlPipeline.scala:91)\\n\\tat com.snowplowanalytics.snowplow.enrich.common.EtlPipeline$$anonfun$1$$anonfun$apply$1.apply(EtlPipeline.scala:89)\\n\\tat scala.Option.map(Option.scala:145)\\n\\tat com.snowplowanalytics.snowplow.enrich.common.EtlPipeline$$anonfun$1.apply(EtlPipeline.scala:89)\\n\\tat com.snowplowanalytics.snowplow.enrich.common.EtlPipeline$$anonfun$1.apply(EtlPipeline.scala:87)\\n\\tat scalaz.Validation$class.map(Validation.scala:114)\\n\\tat scalaz.Success.map(Validation.scala:347)\\n\\tat com.snowplowanalytics.snowplow.enrich.common.EtlPipeline$.processEvents(EtlPipeline.scala:87)\\n\\tat com.snowplowanalytics.snowplow.enrich.hadoop.EtlJob$$anonfun$7.apply(EtlJob.scala:172)\\n\\tat com.snowplowanalytics.snowplow.enrich.hadoop.EtlJob$$anonfun$7.apply(EtlJob.scala:171)\\n\\tat com.twitter.scalding.MapFunction.operate(Operations.scala:58)\\n\\tat cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)\\n\\tat cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)\\n\\tat cascading.flow.stream.FunctionEachStage$1.collect(FunctionEachStage.java:80)\\n\\tat cascading.tuple.TupleEntryCollector.safeCollect(TupleEntryCollector.java:145)\\n\\tat cascading.tuple.TupleEntryCollector.add(TupleEntryCollector.java:133)\\n\\tat com.twitter.scalding.FlatMapFunction$$anonfun$operate$2.apply(Operations.scala:48)\\n\\tat com.twitter.scalding.FlatMapFunction$$anonfun$operate$2.apply(Operations.scala:46)\\n\\tat scala.collection.Iterator$class.foreach(Iterator.scala:727)\\n\\tat scala.collection.AbstractIterator.foreach(Iterator.scala:1157)\\n\\tat com.twitter.scalding.FlatMapFunction.operate(Operations.scala:46)\\n\\tat cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:99)\\n\\tat cascading.flow.stream.FunctionEachStage.receive(FunctionEachStage.java:39)\\n\\tat cascading.flow.stream.SourceStage.map(SourceStage.java:102)\\n\\tat cascading.flow.stream.SourceStage.call(SourceStage.java:53)\\n\\tat cascading.flow.stream.SourceStage.call(SourceStage.java:38)\\n\\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\\n\\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\\n\\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\\n\\tat java.lang.Thread.run(Thread.java:744)\\n\\n    level: \\\"error\\\"\\n\"],\"failure_tstamp\":\"2015-08-12T15:57:39.244Z\"}\r\n```\r\n\r\nThe downside is that this can lead to very long messages - the one above is 3454 characters. Do we think this could cause problems later? If so I could try deleting all but the first few lines of the stack trace."}}