{"action":"created","issue":{"url":"https://api.github.com/repos/HazyResearch/deepdive/issues/381","labels_url":"https://api.github.com/repos/HazyResearch/deepdive/issues/381/labels{/name}","comments_url":"https://api.github.com/repos/HazyResearch/deepdive/issues/381/comments","events_url":"https://api.github.com/repos/HazyResearch/deepdive/issues/381/events","html_url":"https://github.com/HazyResearch/deepdive/issues/381","id":105137329,"number":381,"title":"Failure in format_converter – system idle","user":{"login":"raphaelhoffmann","id":991789,"avatar_url":"https://avatars.githubusercontent.com/u/991789?v=3","gravatar_id":"","url":"https://api.github.com/users/raphaelhoffmann","html_url":"https://github.com/raphaelhoffmann","followers_url":"https://api.github.com/users/raphaelhoffmann/followers","following_url":"https://api.github.com/users/raphaelhoffmann/following{/other_user}","gists_url":"https://api.github.com/users/raphaelhoffmann/gists{/gist_id}","starred_url":"https://api.github.com/users/raphaelhoffmann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/raphaelhoffmann/subscriptions","organizations_url":"https://api.github.com/users/raphaelhoffmann/orgs","repos_url":"https://api.github.com/users/raphaelhoffmann/repos","events_url":"https://api.github.com/users/raphaelhoffmann/events{/privacy}","received_events_url":"https://api.github.com/users/raphaelhoffmann/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-09-07T03:10:16Z","updated_at":"2015-09-07T04:16:27Z","closed_at":null,"body":"See log below. There are two issues here:\r\n\r\n1.  I am getting a java.lang.NoClassDefFoundError when running format_converter.\r\n2.  The error is only reported on the console, but not in the error log file. After that, the system goes into an idle state, where to_binary.py is no longer running; however the DD process is, and there seems to be no activity. Unfortunately, the relevant files eg. dd_variables_rates are deleted when this error occurs making it time-consuming to re-run.\r\n\r\nI've reproduced this error twice.\r\n\r\n```\r\n17:30:11 [PostgresInferenceRunnerComponent$PostgresInferenceRunner(akka://deepdive)] INFO  Converting grounding file format...\r\n17:30:11 [PostgresInferenceRunnerComponent$PostgresInferenceRunner(akka://deepdive)] DEBUG Executing: python /lfs/raiders4/1/raphaelh/dd/new/deepdive/util/tobinary.py /lfs/raide\r\nrs4/1/raphaelh/dd/new/deepdive/out /lfs/raiders4/1/raphaelh/dd/new/deepdive/util/format_converter /lfs/raiders4/1/raphaelh/dd/new/deepdive/out\r\n17:30:53 [taskManager] INFO  Memory usage: 721/2082MB (max: 27305MB)\r\n17:31:53 [taskManager] INFO  Memory usage: 721/2082MB (max: 27305MB)\r\n\r\nException: java.lang.NoClassDefFoundError thrown from the UncaughtExceptionHandler in thread \"Thread-15\"\r\n\r\nException: java.lang.NoClassDefFoundError thrown from the UncaughtExceptionHandler in thread \"Thread-14\"\r\n17:32:53 [taskManager] INFO  Memory usage: 721/2082MB (max: 27305MB)\r\n17:33:53 [taskManager] INFO  Memory usage: 721/2082MB (max: 27305MB)\r\n17:34:53 [taskManager] INFO  Memory usage: 722/2082MB (max: 27305MB)\r\n17:35:53 [taskManager] INFO  Memory usage: 722/2082MB (max: 27305MB)\r\n...\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/HazyResearch/deepdive/issues/comments/138183243","html_url":"https://github.com/HazyResearch/deepdive/issues/381#issuecomment-138183243","issue_url":"https://api.github.com/repos/HazyResearch/deepdive/issues/381","id":138183243,"user":{"login":"alldefector","id":5607717,"avatar_url":"https://avatars.githubusercontent.com/u/5607717?v=3","gravatar_id":"","url":"https://api.github.com/users/alldefector","html_url":"https://github.com/alldefector","followers_url":"https://api.github.com/users/alldefector/followers","following_url":"https://api.github.com/users/alldefector/following{/other_user}","gists_url":"https://api.github.com/users/alldefector/gists{/gist_id}","starred_url":"https://api.github.com/users/alldefector/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alldefector/subscriptions","organizations_url":"https://api.github.com/users/alldefector/orgs","repos_url":"https://api.github.com/users/alldefector/repos","events_url":"https://api.github.com/users/alldefector/events{/privacy}","received_events_url":"https://api.github.com/users/alldefector/received_events","type":"User","site_admin":false},"created_at":"2015-09-07T04:16:26Z","updated_at":"2015-09-07T04:16:26Z","body":"Based on my experience with Java, NoClassDefFoundError might occur when a\nthread is cleaned up (some sort of race condition between the user code and\nJVM's thread manager I think). The DD log file is still printing every\nminute:\n\n21:05:53.772 [default-dispatcher-3][taskManager][TaskManager] INFO  Memory\nusage: 107/2101MB (max: 27305MB)\n21:06:53.772 [default-dispatcher-3][taskManager][TaskManager] INFO  Memory\nusage: 107/2101MB (max: 27305MB)\n21:07:53.772 [default-dispatcher-3][taskManager][TaskManager] INFO  Memory\nusage: 107/2101MB (max: 27305MB)\n21:08:53.772 [default-dispatcher-3][taskManager][TaskManager] INFO  Memory\nusage: 108/2101MB (max: 27305MB)\n...\n\nBut looks like it's just akka puking -- I don't see tobinary.py or\nformat_converter running.\n\nDoes the converter or the python script print out logs that are not\nredirected by Java? Those logs would be useful. Maybe generate the input\nfile and halt DD and then run the converter manually so we can see the logs.\n\n\nOn Sun, Sep 6, 2015 at 8:10 PM Raphael Hoffmann <notifications@github.com>\nwrote:\n\n> See log below. There are two issues here:\n>\n>    1. I am getting a java.lang.NoClassDefFoundError when running\n>    format_converter.\n>    2. The error is only reported on the console, but not in the error log\n>    file. After that, the system goes into an idle state, where to_binary.py is\n>    no longer running; however the DD process is, and there seems to be no\n>    activity. Unfortunately, the relevant files eg. dd_variables_rates are\n>    deleted when this error occurs making it time-consuming to re-run.\n>\n> I've reproduced this error twice.\n>\n> 17:30:11 [PostgresInferenceRunnerComponent$PostgresInferenceRunner(akka://deepdive)] INFO  Converting grounding file format...\n> 17:30:11 [PostgresInferenceRunnerComponent$PostgresInferenceRunner(akka://deepdive)] DEBUG Executing: python /lfs/raiders4/1/raphaelh/dd/new/deepdive/util/tobinary.py /lfs/raide\n> rs4/1/raphaelh/dd/new/deepdive/out /lfs/raiders4/1/raphaelh/dd/new/deepdive/util/format_converter /lfs/raiders4/1/raphaelh/dd/new/deepdive/out\n> 17:30:53 [taskManager] INFO  Memory usage: 721/2082MB (max: 27305MB)\n> 17:31:53 [taskManager] INFO  Memory usage: 721/2082MB (max: 27305MB)\n>\n> Exception: java.lang.NoClassDefFoundError thrown from the UncaughtExceptionHandler in thread \"Thread-15\"\n>\n> Exception: java.lang.NoClassDefFoundError thrown from the UncaughtExceptionHandler in thread \"Thread-14\"\n> 17:32:53 [taskManager] INFO  Memory usage: 721/2082MB (max: 27305MB)\n> 17:33:53 [taskManager] INFO  Memory usage: 721/2082MB (max: 27305MB)\n> 17:34:53 [taskManager] INFO  Memory usage: 722/2082MB (max: 27305MB)\n> 17:35:53 [taskManager] INFO  Memory usage: 722/2082MB (max: 27305MB)\n> ...\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/HazyResearch/deepdive/issues/381>.\n>\n"}}