{"action":"created","issue":{"url":"https://api.github.com/repos/snowplow/snowplow/issues/1435","labels_url":"https://api.github.com/repos/snowplow/snowplow/issues/1435/labels{/name}","comments_url":"https://api.github.com/repos/snowplow/snowplow/issues/1435/comments","events_url":"https://api.github.com/repos/snowplow/snowplow/issues/1435/events","html_url":"https://github.com/snowplow/snowplow/issues/1435","id":58225927,"number":1435,"title":"kinesis-lzo-s3-sink:  Uing Apache Pig to Load S3 storage data from kinesis-lzo-s3-sink","user":{"login":"christiangda","id":1197820,"avatar_url":"https://avatars.githubusercontent.com/u/1197820?v=3","gravatar_id":"","url":"https://api.github.com/users/christiangda","html_url":"https://github.com/christiangda","followers_url":"https://api.github.com/users/christiangda/followers","following_url":"https://api.github.com/users/christiangda/following{/other_user}","gists_url":"https://api.github.com/users/christiangda/gists{/gist_id}","starred_url":"https://api.github.com/users/christiangda/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/christiangda/subscriptions","organizations_url":"https://api.github.com/users/christiangda/orgs","repos_url":"https://api.github.com/users/christiangda/repos","events_url":"https://api.github.com/users/christiangda/events{/privacy}","received_events_url":"https://api.github.com/users/christiangda/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-02-19T15:51:49Z","updated_at":"2015-02-20T09:08:32Z","closed_at":null,"body":"Hi everyone, \r\nI'm using your apps for process my data,   thats include:\r\n1-trackers/javascript-tracker \r\n2-collectors/scala-stream-collector\r\n\r\n3- Our Own solution\r\n\r\nand then \r\n4-storage/kinesis-lzo-s3-sink\r\n\r\n**once the data is stored in amazon s3 as lzo (and its index lzo.index)\r\nwe are trying to process that files using Apache Pig as follow:**\r\n\r\n#### Our pig script\r\n```\r\nregister /usr/local/pig/lib/elephant-bird-core-4.6.jar;\r\nregister /usr/local/pig/lib/elephant-bird-cascading2-4.6.jar;\r\nregister /usr/local/pig/lib/elephant-bird-pig-4.6.jar;\r\nregister /usr/local/pig/lib/elephant-bird-hadoop-compat-4.6.jar;\r\nregister /usr/local/pig/lib/snowplow-stream-collector-0.3.0.jar;\r\n\r\ndata = LOAD '/home/devuser/ltv/data/snowplow/*.lzo.index' \r\nUSING  com.twitter.elephantbird.pig.load.ThriftPigLoader('com.snowplowanalytics.snowplow.CollectorPayload.thrift.model1.CollectorPayload');\r\n\r\nDESCRIBE data;\r\ndata: {schema: chararray,ipAddress: chararray,timestamp: long,encoding: chararray,collector: chararray,userAgent: chararray,refererUri: chararray,path: chararray,querystring: chararray,body: chararray,headers: {t: (headers_tuple: chararray)},contentType: chararray,hostname: chararray,networkUserId: chararray}\r\n\r\nDUMP data;\r\n```\r\n### We have this error, after DUMP data!\r\n```\r\n2015-02-19 15:41:14,384 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN\r\n2015-02-19 15:41:14,384 [main] INFO  org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]}\r\n2015-02-19 15:41:14,392 [main] WARN  org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized\r\n2015-02-19 15:41:14,402 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1\r\n2015-02-19 15:41:14,403 [main] INFO  org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt__0001_m_000001_1' to file:/tmp/temp-1627216190/tmp-1495567355/_temporary/0/task__0001_m_000001\r\n2015-02-19 15:41:14,408 [main] WARN  org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized\r\n2015-02-19 15:41:14,414 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1\r\n2015-02-19 15:41:14,414 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1\r\n```\r\n\r\n### Then we try changing the schema to:\r\n```pig\r\ndata = LOAD '/user/devuser/ltv/data/snowplow/*.lzo.index'\r\n    USING com.twitter.elephantbird.pig.load.ThriftPigLoader('com.snowplowanalytics.snowplow.collectors.thrift.SnowplowRawEvent');\r\n```\r\n#### And we have the same error!\r\n\r\n#### It's posible to load S3 thrift lzo data storage by your kinesis-lzo-s3-sink from pig?\r\n\r\n\r\n### Thanks for your help!\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/snowplow/snowplow/issues/comments/75208563","html_url":"https://github.com/snowplow/snowplow/issues/1435#issuecomment-75208563","issue_url":"https://api.github.com/repos/snowplow/snowplow/issues/1435","id":75208563,"user":{"login":"christiangda","id":1197820,"avatar_url":"https://avatars.githubusercontent.com/u/1197820?v=3","gravatar_id":"","url":"https://api.github.com/users/christiangda","html_url":"https://github.com/christiangda","followers_url":"https://api.github.com/users/christiangda/followers","following_url":"https://api.github.com/users/christiangda/following{/other_user}","gists_url":"https://api.github.com/users/christiangda/gists{/gist_id}","starred_url":"https://api.github.com/users/christiangda/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/christiangda/subscriptions","organizations_url":"https://api.github.com/users/christiangda/orgs","repos_url":"https://api.github.com/users/christiangda/repos","events_url":"https://api.github.com/users/christiangda/events{/privacy}","received_events_url":"https://api.github.com/users/christiangda/received_events","type":"User","site_admin":false},"created_at":"2015-02-20T09:08:32Z","updated_at":"2015-02-20T09:08:32Z","body":"Hi Alexander, Thanks for your answer, you are right, I forgot mention that we don't have any error, we only have this information:\r\n```\r\ngrunt> DUMP data;\r\n2015-02-20 08:50:51,827 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN\r\n2015-02-20 08:50:51,884 [main] INFO  org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]}\r\n2015-02-20 08:50:52,463 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum\r\n2015-02-20 08:50:52,464 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS\r\n2015-02-20 08:50:52,629 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1\r\n2015-02-20 08:50:52,629 [main] INFO  org.apache.hadoop.conf.Configuration.deprecation - mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive\r\n2015-02-20 08:50:52,631 [main] INFO  org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output of task 'attempt__0001_m_000001_1' to file:/tmp/temp1145700320/tmp-1189644891/_temporary/0/task__0001_m_000001\r\n2015-02-20 08:50:52,648 [main] WARN  org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized\r\n2015-02-20 08:50:52,657 [main] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1\r\n2015-02-20 08:50:52,661 [main] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1\r\n```\r\n### We have searched in all the logs of process, system and pig, and we haven't any errors,  but when we run the command DUMP, we have no data !\r\n \r\n\r\n\r\n### Tools version\r\n- Apache Pig version 0.13.0 (r1606446) compiled Jun 29 2014, 02:27:58\r\n- Linux hadoop-dev 3.13.0-45-generic #74-Ubuntu SMP Tue Jan 13 19:36:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux\r\n- Hadoop 2.4.0\r\n>Subversion http://svn.apache.org/repos/asf/hadoop/common -r 1583262\r\n>Compiled by jenkins on 2014-03-31T08:29Z\r\n>Compiled with protoc 2.5.0\r\n>From source with checksum 375b2832a6641759c6eaf6e3e998147\r\n>This command was run using /usr/local/hadoop-2.4.0/share/hadoop/common/hadoop-common-2.4.0.jar\r\n\r\n\r\n### Scala collector log trace\r\n```\r\n08:33:31.495 [scala-stream-collector-akka.actor.default-dispatcher-13] DEBUG akka.io.TcpListener - New connection accepted\r\n08:33:31.502 [scala-stream-collector-akka.actor.default-dispatcher-12] DEBUG s.can.server.HttpServerConnection - Dispatching GET request to http://192.168.100.110:8080/i?e=ue&ue_px=eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJkYXRhIjp7InVzZXJfaWQiOjk4ODMxOSwic2Vzc2lvbl9pZCI6MTQxMDYsImFjdGlvbiI6InNlc3Npb25fZW5kIiwic29mdHdhcmUiOiJGUkFNME5CR0pLIiwidmVyc2lvbiI6InYgMS4xLjE0IiwiY2FtcGFpZ24iOiJjYW1wYWlnbiIsIm9sZF92ZXJzaW9uIjoidiAxLjEuMTQifX19&dtm=1424421211442&tid=101143&vp=1855x993&ds=1855x993&vid=1&duid=954c0aa191bae4fa&p=web&tv=js-2.0.0&fp=1941551860&lang=en-us&cs=UTF-8&tz=Europe/London&f_pdf=1&f_qt=0&f_realp=0&f_wma=0&f_dir=0&f_fla=1&f_java=1&f_gears=0&f_ag=0&res=1920x1080&cd=24&cookie=1&url=file:///home/christian/Documents/Tuguu/LTV/SnowPlow/1-tracker/web-js-tracker/tgu-track.1.html to handler Actor[akka://scala-stream-collector/system/IO-TCP/selectors/$a/24#-340979721]\r\n08:33:31.503 [scala-stream-collector-akka.actor.default-dispatcher-13] INFO  c.s.s.c.s.sinks.KinesisSink - Writing Thrift record to Kinesis: CollectorPayload(schema:iglu:com.snowplowanalytics.snowplow/CollectorPayload/thrift/1-0-0, ipAddress:192.168.100.1, timestamp:1424421211503, encoding:UTF-8, collector:ssc-0.3.0-kinesis, userAgent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36, path:/i, querystring:e=ue&ue_px=eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy91bnN0cnVjdF9ldmVudC9qc29uc2NoZW1hLzEtMC0wIiwiZGF0YSI6eyJkYXRhIjp7InVzZXJfaWQiOjk4ODMxOSwic2Vzc2lvbl9pZCI6MTQxMDYsImFjdGlvbiI6InNlc3Npb25fZW5kIiwic29mdHdhcmUiOiJGUkFNME5CR0pLIiwidmVyc2lvbiI6InYgMS4xLjE0IiwiY2FtcGFpZ24iOiJjYW1wYWlnbiIsIm9sZF92ZXJzaW9uIjoidiAxLjEuMTQifX19&dtm=1424421211442&tid=101143&vp=1855x993&ds=1855x993&vid=1&duid=954c0aa191bae4fa&p=web&tv=js-2.0.0&fp=1941551860&lang=en-us&cs=UTF-8&tz=Europe%2FLondon&f_pdf=1&f_qt=0&f_realp=0&f_wma=0&f_dir=0&f_fla=1&f_java=1&f_gears=0&f_ag=0&res=1920x1080&cd=24&cookie=1&url=file%3A%2F%2F%2Fhome%2Fchristian%2FDocuments%2FTuguu%2FLTV%2FSnowPlow%2F1-tracker%2Fweb-js-tracker%2Ftgu-track.1.html, headers:[X-Forwarded-For: 192.168.100.1, Cookie: sp=c124c4b8-9f3a-475f-9491-af2d772e045b, Accept-Language: es, en-US, en, Accept-Encoding: gzip, deflate, sdch, User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36, Accept: image/webp, */*;q=0.8, Host: 192.168.100.110:8080], hostname:192.168.100.110, networkUserId:c124c4b8-9f3a-475f-9491-af2d772e045b)\r\n08:33:31.508 [scala-stream-collector-akka.actor.default-dispatcher-9] DEBUG akka.io.TcpIncomingConnection - Closing connection due to IO error java.io.IOException: Connection reset by peer\r\n08:33:31.509 [scala-stream-collector-akka.actor.default-dispatcher-9] DEBUG s.can.server.HttpServerConnection - Connection was ErrorClosed(Connection reset by peer), awaiting TcpConnection termination...\r\n08:33:31.510 [scala-stream-collector-akka.actor.default-dispatcher-12] DEBUG s.can.server.HttpServerConnection - TcpConnection terminated, stopping\r\n08:33:31.672 [pool-1-thread-10] INFO  c.s.s.c.s.sinks.KinesisSink - Writing successful.\r\n08:33:31.672 [pool-1-thread-10] INFO  c.s.s.c.s.sinks.KinesisSink -   + ShardId: shardId-000000000000\r\n08:33:31.672 [pool-1-thread-10] INFO  c.s.s.c.s.sinks.KinesisSink -   + SequenceNumber: 49548077043996487777971171182938030608387103198025875458\r\n```\r\n### kinesis-lzo-s3-sink log trace\r\n```\r\nfeb 20, 2015 8:34:31 AM com.amazonaws.services.kinesis.metrics.impl.DefaultCWMetricsPublisher publishMetrics\r\nINFORMACIÓN: Successfully published 14 datums.\r\nfeb 20, 2015 8:34:41 AM com.amazonaws.services.kinesis.metrics.impl.DefaultCWMetricsPublisher publishMetrics\r\nINFORMACIÓN: Successfully published 12 datums.\r\nfeb 20, 2015 8:34:52 AM com.amazonaws.services.kinesis.metrics.impl.DefaultCWMetricsPublisher publishMetrics\r\nINFORMACIÓN: Successfully published 12 datums.\r\nfeb 20, 2015 8:34:57 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker$WorkerLog info\r\nINFORMACIÓN: Current stream shard assignments: shardId-000000000000\r\nfeb 20, 2015 8:34:57 AM com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker$WorkerLog info\r\n```\r\n\r\n### Data files in S3 and downloaded to my FS\r\n```\r\n-rw------- 1 devuser devuser 2877 Feb 20 08:44 2015-02-20-49548077043996487777971171182912643166175195023284371458-49548077043996487777971171182939239534206717827200581634.lzo\r\n-rw------- 1 devuser devuser    8 Feb 20 08:44 2015-02-20-49548077043996487777971171182912643166175195023284371458-49548077043996487777971171182939239534206717827200581634.lzo.index\r\n```\r\n\r\n### It is possible to do that?,  I mean ,  load \"Thrift-Lzo SnowPlow format\" from Apache Pig using Elephant-bird library?\r\n\r\n#### Thanks for your comments\r\n"}}