{"action":"created","issue":{"url":"https://api.github.com/repos/linkedin/gobblin/issues/386","labels_url":"https://api.github.com/repos/linkedin/gobblin/issues/386/labels{/name}","comments_url":"https://api.github.com/repos/linkedin/gobblin/issues/386/comments","events_url":"https://api.github.com/repos/linkedin/gobblin/issues/386/events","html_url":"https://github.com/linkedin/gobblin/issues/386","id":111560210,"number":386,"title":"Gobblin not working with KafkaSource and mapreduce","user":{"login":"kzarzycki-advertine","id":11393284,"avatar_url":"https://avatars.githubusercontent.com/u/11393284?v=3","gravatar_id":"","url":"https://api.github.com/users/kzarzycki-advertine","html_url":"https://github.com/kzarzycki-advertine","followers_url":"https://api.github.com/users/kzarzycki-advertine/followers","following_url":"https://api.github.com/users/kzarzycki-advertine/following{/other_user}","gists_url":"https://api.github.com/users/kzarzycki-advertine/gists{/gist_id}","starred_url":"https://api.github.com/users/kzarzycki-advertine/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kzarzycki-advertine/subscriptions","organizations_url":"https://api.github.com/users/kzarzycki-advertine/orgs","repos_url":"https://api.github.com/users/kzarzycki-advertine/repos","events_url":"https://api.github.com/users/kzarzycki-advertine/events{/privacy}","received_events_url":"https://api.github.com/users/kzarzycki-advertine/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2015-10-15T07:29:37Z","updated_at":"2015-11-19T18:22:04Z","closed_at":null,"body":"Hi, \r\nI'm trying to launch gobblin-mapreduce.sh on my job config, that is almost copy/paste from your wiki https://github.com/linkedin/gobblin/wiki/Kafka-HDFS-Ingestion\r\n\r\nI'm launching gobblin with command:\r\n```\r\nbin/gobblin-mapreduce.sh  --conf jobs/dump-kafka.properties --workdir work/\r\n```\r\n\r\nBut the job fails with the following repeated error in all mappers:\r\n```\r\njava.lang.NoClassDefFoundError: kafka/common/TopicAndPartition\r\n\tat gobblin.source.extractor.extract.kafka.KafkaWrapper$KafkaOldAPI.createFetchRequest(KafkaWrapper.java:401)\r\n\tat gobblin.source.extractor.extract.kafka.KafkaWrapper$KafkaOldAPI.fetchNextMessageBuffer(KafkaWrapper.java:333)\r\n\tat gobblin.source.extractor.extract.kafka.KafkaWrapper.fetchNextMessageBuffer(KafkaWrapper.java:136)\r\n\tat gobblin.source.extractor.extract.kafka.KafkaExtractor.fetchNextMessageBuffer(KafkaExtractor.java:239)\r\n\tat gobblin.source.extractor.extract.kafka.KafkaExtractor.readRecordImpl(KafkaExtractor.java:125)\r\n\tat gobblin.instrumented.extractor.InstrumentedExtractorBase.readRecord(InstrumentedExtractorBase.java:121)\r\n\tat gobblin.instrumented.extractor.InstrumentedExtractor.readRecord(InstrumentedExtractor.java:34)\r\n\tat gobblin.runtime.LimitingExtractorDecorator.readRecord(LimitingExtractorDecorator.java:69)\r\n\tat gobblin.instrumented.extractor.InstrumentedExtractorDecorator.readRecordImpl(InstrumentedExtractorDecorator.java:64)\r\n\tat gobblin.instrumented.extractor.InstrumentedExtractorDecorator.readRecord(InstrumentedExtractorDecorator.java:57)\r\n\tat gobblin.runtime.Task.run(Task.java:169)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.ClassNotFoundException: kafka.common.TopicAndPartition\r\n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:372)\r\n\tat java.net.URLClassLoader$1.run(URLClassLoader.java:361)\r\n\tat java.security.AccessController.doPrivileged(Native Method)\r\n\tat java.net.URLClassLoader.findClass(URLClassLoader.java:360)\r\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:424)\r\n\tat sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)\r\n\tat java.lang.ClassLoader.loadClass(ClassLoader.java:357)\r\n\t... 14 more\r\n```\r\n\r\nIt seems that gobblin does not include kafka (and other) jars in the mapreduce tasks's classpath. \r\nI also tried to include all the jars in lib/ directory to libjars with command:\r\n```\r\nbin/gobblin-mapreduce.sh  --conf jobs/dump-kafka.properties --workdir work/ --jars `ls lib/* | tr \"\\n\" \",\"` \r\n```\r\n\r\nBut this time, I get error of clashing guava libraries:\r\n```\r\nError: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException\r\n        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)\r\n        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)\r\n        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)\r\n        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)\r\n        at java.security.AccessController.doPrivileged(Native Method)\r\n        at javax.security.auth.Subject.doAs(Subject.java:422)\r\n        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)\r\n        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)\r\nCaused by: java.lang.reflect.InvocationTargetException\r\n        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\r\n        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\r\n        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\r\n        at java.lang.reflect.Constructor.newInstance(Constructor.java:408)\r\n        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)\r\n        ... 7 more\r\nCaused by: java.lang.NoSuchMethodError: com.google.common.collect.Sets.newConcurrentHashSet()Ljava/util/Set;\r\n        at gobblin.configuration.SourceState.<clinit>(SourceState.java:54)\r\n        at gobblin.runtime.mapreduce.MRJobLauncher$TaskRunner.<init>(MRJobLauncher.java:554)\r\n        ... 12 more\r\n```\r\n\r\nI have hadoop 2.4.0, which uses guava 11.0.2, while the one in lib/ is guava-15.0. \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/linkedin/gobblin/issues/comments/158145837","html_url":"https://github.com/linkedin/gobblin/issues/386#issuecomment-158145837","issue_url":"https://api.github.com/repos/linkedin/gobblin/issues/386","id":158145837,"user":{"login":"zliu41","id":6342538,"avatar_url":"https://avatars.githubusercontent.com/u/6342538?v=3","gravatar_id":"","url":"https://api.github.com/users/zliu41","html_url":"https://github.com/zliu41","followers_url":"https://api.github.com/users/zliu41/followers","following_url":"https://api.github.com/users/zliu41/following{/other_user}","gists_url":"https://api.github.com/users/zliu41/gists{/gist_id}","starred_url":"https://api.github.com/users/zliu41/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zliu41/subscriptions","organizations_url":"https://api.github.com/users/zliu41/orgs","repos_url":"https://api.github.com/users/zliu41/repos","events_url":"https://api.github.com/users/zliu41/events{/privacy}","received_events_url":"https://api.github.com/users/zliu41/received_events","type":"User","site_admin":false},"created_at":"2015-11-19T18:22:04Z","updated_at":"2015-11-19T18:22:04Z","body":"@gilmichlin is it still because of the Guava dependency? Based on HIVE-5733 it shouldn't be a problem with Hive 1.2.0 or later.\r\n\r\nIf so, is there any hive version that works for you?"}}