{"action":"created","issue":{"url":"https://api.github.com/repos/pinterest/secor/issues/154","labels_url":"https://api.github.com/repos/pinterest/secor/issues/154/labels{/name}","comments_url":"https://api.github.com/repos/pinterest/secor/issues/154/comments","events_url":"https://api.github.com/repos/pinterest/secor/issues/154/events","html_url":"https://github.com/pinterest/secor/issues/154","id":121791872,"number":154,"title":"Have trouble running run_tests.sh","user":{"login":"kchew534","id":4525289,"avatar_url":"https://avatars.githubusercontent.com/u/4525289?v=3","gravatar_id":"","url":"https://api.github.com/users/kchew534","html_url":"https://github.com/kchew534","followers_url":"https://api.github.com/users/kchew534/followers","following_url":"https://api.github.com/users/kchew534/following{/other_user}","gists_url":"https://api.github.com/users/kchew534/gists{/gist_id}","starred_url":"https://api.github.com/users/kchew534/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kchew534/subscriptions","organizations_url":"https://api.github.com/users/kchew534/orgs","repos_url":"https://api.github.com/users/kchew534/repos","events_url":"https://api.github.com/users/kchew534/events{/privacy}","received_events_url":"https://api.github.com/users/kchew534/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-12-11T21:08:27Z","updated_at":"2015-12-11T22:28:53Z","closed_at":null,"body":"Hi,\r\n\r\nI know this has been reported before. Yes I read the run_tests header but I am kind of confuse on what are the \"Hadoop native libs\" refer to? Those in lib/native (The share libraries) or something else? I am running this on mac os Yosemite and my thrift version is 0.9.3\r\n\r\n*****************************************************************************\r\nscripts/run_tests.sh: line 55: declare: -A: invalid option\r\ndeclare: usage: declare [-afFirtx] [-p] [name[=value] ...]\r\n********************************************************\r\nRunning tests for Message Type: 0 and  ReaderWriter:com.pinterest.secor.io.impl.SequenceFileReaderWriterFactory using filesystem: s3n\r\n********************************************************\r\nrunning post_and_verify_test\r\nrunning pkill -f 'com.pinterest.secor.main.ConsumerMain' || true\r\nrunning pkill -f 'kafka.Kafka' || true\r\nrunning pkill -f 'org.apache.zookeeper.server.quorum.QuorumPeerMain' || true\r\nrunning rm -r -f /tmp/secor_dev\r\nrunning s3cmd del --recursive s3://test-bucket/secor_dev\r\nscripts/run_tests.sh: line 77: s3cmd: command not found\r\nrunning s3cmd ls -r s3://test-bucket/secor_dev\r\nscripts/run_tests.sh: line 77: s3cmd: command not found\r\nrunning mkdir -p /tmp/secor_dev/logs\r\nrunning scripts/run_kafka_class.sh         org.apache.zookeeper.server.quorum.QuorumPeerMain scripts/../zookeeper.test.properties >         /tmp/secor_dev/logs/zookeeper.log 2>&1 &\r\nrunning scripts/run_kafka_class.sh kafka.Kafka scripts/../kafka.test.properties >         /tmp/secor_dev/logs/kafka_server.log 2>&1 &\r\nrunning scripts/run_kafka_class.sh kafka.admin.TopicCommand --create --zookeeper         localhost:2181 --replication-factor 1 --partitions 2 --topic test >         /tmp/secor_dev/logs/create_topic.log 2>&1\r\nrunning /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home//bin/java -server -ea -Dlog4j.configuration=log4j.dev.properties         -Dconfig=secor.test.backup.properties -Dsecor.s3.filesystem=s3n -Dsecor.file.reader.writer.factory=com.pinterest.secor.io.impl.SequenceFileReaderWriterFactory -cp *:lib/*         com.pinterest.secor.main.ConsumerMain > /tmp/secor_dev/logs/secor_backup.log 2>&1 &\r\nrunning /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home//bin/java -server -ea -Dlog4j.configuration=log4j.dev.properties         -Dconfig=secor.test.backup.properties -cp *:lib/*         com.pinterest.secor.main.TestLogMessageProducerMain -t test -m 100 -p 1 -type 0 -timeshift 0 >         /tmp/secor_dev/logs/test_log_message_producer.log 2>&1\r\nWaiting 40 sec for Secor to upload logs to s3\r\nVerifying 100 0\r\nrunning /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home//bin/java -server -ea -Dlog4j.configuration=log4j.dev.properties           -Dconfig=secor.test.backup.properties -Dsecor.s3.filesystem=s3n -Dsecor.file.reader.writer.factory=com.pinterest.secor.io.impl.SequenceFileReaderWriterFactory -cp *:lib/*           com.pinterest.secor.main.LogFileVerifierMain -t test -m 100 -q >           /tmp/secor_dev/logs/log_verifier_backup.log 2>&1\r\n\\e[1;41;97mVerification FAILED\\e[0m\r\nSee log /tmp/secor_dev/logs/log_verifier_backup.log for more details\r\n2015-12-11 12:52:33,216 [main] (org.apache.hadoop.util.NativeCodeLoader:62) WARN  Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\n2015-12-11 12:52:34,439 [main] (com.pinterest.secor.main.LogFileVerifierMain:99) ERROR Log file verifier failed\r\norg.apache.hadoop.security.AccessControlException: Permission denied: s3n://test-bucket/secor_dev/backup/test\r\n\tat org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:449)\r\n\tat org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:427)\r\n\tat org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleException(Jets3tNativeFileSystemStore.java:411)\r\n\tat org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:181)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:497)\r\n\tat org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)\r\n\tat org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)\r\n\tat org.apache.hadoop.fs.s3native.$Proxy4.retrieveMetadata(Unknown Source)\r\n\tat org.apache.hadoop.fs.s3native.NativeS3FileSystem.listStatus(NativeS3FileSystem.java:526)\r\n\tat com.pinterest.secor.util.FileUtil.list(FileUtil.java:102)\r\n\tat com.pinterest.secor.util.FileUtil.listRecursively(FileUtil.java:119)\r\n\tat com.pinterest.secor.tools.LogFileVerifier.populateTopicPartitionToOffsetToFiles(LogFileVerifier.java:58)\r\n\tat com.pinterest.secor.tools.LogFileVerifier.verifyCounts(LogFileVerifier.java:116)\r\n\tat com.pinterest.secor.main.LogFileVerifierMain.main(LogFileVerifierMain.java:93)\r\nCaused by: org.jets3t.service.impl.rest.HttpException: 403 Forbidden\r\n\tat org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:425)\r\n\tat org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:279)\r\n\tat org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:1052)\r\n\tat org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2264)\r\n\tat org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2193)\r\n\tat org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1120)\r\n\tat org.jets3t.service.StorageService.getObjectDetails(StorageService.java:575)\r\n\tat org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:174)\r\n\t... 13 more\r\nSee log /tmp/secor_dev/logs/secor_backup.log for more details\r\n<snaporama>\r\n2015-12-11 12:51:52,714 [Thread-0] (kafka.utils.Logging$class:68) INFO  Property serializer.class is overridden to kafka.serializer.DefaultEncoder\r\nException in thread \"Thread-0\" java.lang.RuntimeException: Undefined message encoding type: 0\r\n\tat com.pinterest.secor.tools.TestLogMessageProducer.run(TestLogMessageProducer.java:76)\r\n****************************************************************************************************\r\n\r\nTIA\r\n\r\nKim"},"comment":{"url":"https://api.github.com/repos/pinterest/secor/issues/comments/164066489","html_url":"https://github.com/pinterest/secor/issues/154#issuecomment-164066489","issue_url":"https://api.github.com/repos/pinterest/secor/issues/154","id":164066489,"user":{"login":"kchew534","id":4525289,"avatar_url":"https://avatars.githubusercontent.com/u/4525289?v=3","gravatar_id":"","url":"https://api.github.com/users/kchew534","html_url":"https://github.com/kchew534","followers_url":"https://api.github.com/users/kchew534/followers","following_url":"https://api.github.com/users/kchew534/following{/other_user}","gists_url":"https://api.github.com/users/kchew534/gists{/gist_id}","starred_url":"https://api.github.com/users/kchew534/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kchew534/subscriptions","organizations_url":"https://api.github.com/users/kchew534/orgs","repos_url":"https://api.github.com/users/kchew534/repos","events_url":"https://api.github.com/users/kchew534/events{/privacy}","received_events_url":"https://api.github.com/users/kchew534/received_events","type":"User","site_admin":false},"created_at":"2015-12-11T22:28:53Z","updated_at":"2015-12-11T22:28:53Z","body":"Thanks Henry. And I believe I have to install s3cmd and fakes3 as well, correct?"}}