{"action":"opened","issue":{"url":"https://api.github.com/repos/CommBank/ebenezer/issues/81","labels_url":"https://api.github.com/repos/CommBank/ebenezer/issues/81/labels{/name}","comments_url":"https://api.github.com/repos/CommBank/ebenezer/issues/81/comments","events_url":"https://api.github.com/repos/CommBank/ebenezer/issues/81/events","html_url":"https://github.com/CommBank/ebenezer/issues/81","id":57728429,"number":81,"title":"Tests for Hive using CDH5 fail with MetaException","user":{"login":"stephanh","id":376868,"avatar_url":"https://avatars.githubusercontent.com/u/376868?v=3","gravatar_id":"","url":"https://api.github.com/users/stephanh","html_url":"https://github.com/stephanh","followers_url":"https://api.github.com/users/stephanh/followers","following_url":"https://api.github.com/users/stephanh/following{/other_user}","gists_url":"https://api.github.com/users/stephanh/gists{/gist_id}","starred_url":"https://api.github.com/users/stephanh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stephanh/subscriptions","organizations_url":"https://api.github.com/users/stephanh/orgs","repos_url":"https://api.github.com/users/stephanh/repos","events_url":"https://api.github.com/users/stephanh/events{/privacy}","received_events_url":"https://api.github.com/users/stephanh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-02-15T12:16:02Z","updated_at":"2015-02-15T12:16:02Z","closed_at":null,"body":"Under some circumstances when running unit tests using the Hive monad where we:\r\n* Run a test using Hive\r\n* Followed by a test using the `Hive.createTable` method zipped and then another Hive command\r\n \r\nThe execution fails with `MetaException(message:Metastore contains multiple versions)`.\r\n\r\n\r\n###Sample code\r\n```\r\n def pre = {\r\n    executesOk(Execution.fromHive(Hive.query(\"SHOW DATABASES\")))\r\n  }\r\n\r\n  def test = {\r\n    val exec = for {\r\n      r <- Execution.fromHive(Hive.createParquetTable[SimpleHive](\"testa\", \"unpart\", List.empty, None)) zip Execution.fromHive(Hive.createParquetTable[SimpleHive](\"testb\", \"unpart\", List.empty, None))\r\n      s <- Execution.fromHive(Hive.query(\"SHOW DATABASES\")).recoverWith { case _ => Execution.from(List(\"other\")) }\r\n      _ <- Execution.fromHive(Hive.query(\"USE testa\"))\r\n    } yield s\r\n    executesSuccessfully(exec) must_== List(\"default\", \"testa\", \"testb\")\r\n  }\r\n```\r\n\r\n###Exceptions:\r\n```\r\nMetaException(message:Metastore contains multiple versions)\r\n\tat org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6373)\r\n\tat org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6335)\r\n\tat org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6294)\r\n\tat org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6282)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)\r\n\tat com.sun.proxy.$Proxy30.verifySchema(Unknown Source)\r\n\tat org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:485)\r\n\tat org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_all_databases(HiveMetaStore.java:1035)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106)\r\n\tat com.sun.proxy.$Proxy32.get_all_databases(Unknown Source)\r\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:848)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)\r\n\tat com.sun.proxy.$Proxy33.getAllDatabases(Unknown Source)\r\n\tat org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1121)\r\n\tat org.apache.hadoop.hive.ql.exec.DDLTask.showDatabases(DDLTask.java:2430)\r\n\tat org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:363)\r\n\tat org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)\r\n\tat org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)\r\n\tat org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1516)\r\n\tat org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1283)\r\n\tat org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1101)\r\n\tat org.apache.hadoop.hive.ql.Driver.run(Driver.java:924)\r\n\tat org.apache.hadoop.hive.ql.Driver.run(Driver.java:914)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.Hive$$anonfun$query$1.apply(Hive.scala:385)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.Hive$$anonfun$query$1.apply(Hive.scala:378)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.Hive.run(Hive.scala:133)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.RichExecutionObject$$anonfun$fromHive$1$$anonfun$apply$76.apply(HiveSpec.scala:315)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.RichExecutionObject$$anonfun$fromHive$1$$anonfun$apply$76.apply(HiveSpec.scala:315)\r\n\tat com.twitter.scalding.Execution$FutureConst$$anonfun$runStats$1$$anonfun$1.apply(Execution.scala:251)\r\n\tat com.twitter.scalding.Execution$FutureConst$$anonfun$runStats$1$$anonfun$1.apply(Execution.scala:251)\r\n\tat scala.util.Try$.apply(Try.scala:161)\r\n\tat com.twitter.scalding.Execution$FutureConst$$anonfun$runStats$1.apply(Execution.scala:251)\r\n\tat com.twitter.scalding.Execution$FutureConst$$anonfun$runStats$1.apply(Execution.scala:250)\r\n\tat com.twitter.scalding.Execution$MapEvalCache.getOrElseInsert(Execution.scala:224)\r\n\tat com.twitter.scalding.Execution$FutureConst.runStats(Execution.scala:250)\r\n\tat com.twitter.scalding.Execution$FlatMapped$$anonfun$runStats$2$$anonfun$3.apply(Execution.scala:268)\r\n\tat com.twitter.scalding.Execution$FlatMapped$$anonfun$runStats$2$$anonfun$3.apply(Execution.scala:266)\r\n\tat scala.util.Success$$anonfun$map$1.apply(Try.scala:206)\r\n\tat scala.util.Try$.apply(Try.scala:161)\r\n\tat scala.util.Success.map(Try.scala:206)\r\n\tat scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)\r\n\tat scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)\r\n\tat scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n```\r\n\r\n```\r\nFAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: org.apache.hadoop.hive.metastore.api.MetaException Metastore contains multiple versions)\r\nMetaException(message:Metastore contains multiple versions)\r\n\tat org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6373)\r\n\tat org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6335)\r\n\tat org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6294)\r\n\tat org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6282)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)\r\n\tat com.sun.proxy.$Proxy30.verifySchema(Unknown Source)\r\n\tat org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:485)\r\n\tat org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:805)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106)\r\n\tat com.sun.proxy.$Proxy32.get_database(Unknown Source)\r\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:960)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)\r\n\tat com.sun.proxy.$Proxy33.getDatabase(Unknown Source)\r\n\tat org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1188)\r\n\tat org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1177)\r\n\tat org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:4035)\r\n\tat org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:269)\r\n\tat org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)\r\n\tat org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)\r\n\tat org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1516)\r\n\tat org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1283)\r\n\tat org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1101)\r\n\tat org.apache.hadoop.hive.ql.Driver.run(Driver.java:924)\r\n\tat org.apache.hadoop.hive.ql.Driver.run(Driver.java:914)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.Hive$$anonfun$query$1.apply(Hive.scala:385)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.Hive$$anonfun$query$1.apply(Hive.scala:378)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.Hive.run(Hive.scala:133)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.RichExecutionObject$$anonfun$fromHive$1$$anonfun$apply$76.apply(HiveSpec.scala:315)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.RichExecutionObject$$anonfun$fromHive$1$$anonfun$apply$76.apply(HiveSpec.scala:315)\r\n\tat com.twitter.scalding.Execution$FutureConst$$anonfun$runStats$1$$anonfun$1.apply(Execution.scala:251)\r\n\tat com.twitter.scalding.Execution$FutureConst$$anonfun$runStats$1$$anonfun$1.apply(Execution.scala:251)\r\n\tat scala.util.Try$.apply(Try.scala:161)\r\n\tat com.twitter.scalding.Execution$FutureConst$$anonfun$runStats$1.apply(Execution.scala:251)\r\n\tat com.twitter.scalding.Execution$FutureConst$$anonfun$runStats$1.apply(Execution.scala:250)\r\n\tat com.twitter.scalding.Execution$MapEvalCache.getOrElseInsert(Execution.scala:224)\r\n\tat com.twitter.scalding.Execution$FutureConst.runStats(Execution.scala:250)\r\n\tat com.twitter.scalding.Execution$FlatMapped$$anonfun$runStats$2$$anonfun$3.apply(Execution.scala:268)\r\n\tat com.twitter.scalding.Execution$FlatMapped$$anonfun$runStats$2$$anonfun$3.apply(Execution.scala:266)\r\n\tat scala.util.Success$$anonfun$map$1.apply(Try.scala:206)\r\n\tat scala.util.Try$.apply(Try.scala:161)\r\n\tat scala.util.Success.map(Try.scala:206)\r\n\tat scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)\r\n\tat scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)\r\n\tat scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n\r\norg.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Metastore contains multiple versions)\r\n\tat org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1192)\r\n\tat org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1177)\r\n\tat org.apache.hadoop.hive.ql.exec.DDLTask.switchDatabase(DDLTask.java:4035)\r\n\tat org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:269)\r\n\tat org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)\r\n\tat org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)\r\n\tat org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1516)\r\n\tat org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1283)\r\n\tat org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1101)\r\n\tat org.apache.hadoop.hive.ql.Driver.run(Driver.java:924)\r\n\tat org.apache.hadoop.hive.ql.Driver.run(Driver.java:914)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.Hive$$anonfun$query$1.apply(Hive.scala:385)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.Hive$$anonfun$query$1.apply(Hive.scala:378)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.Hive.run(Hive.scala:133)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.RichExecutionObject$$anonfun$fromHive$1$$anonfun$apply$76.apply(HiveSpec.scala:315)\r\n\tat au.com.cba.omnia.ebenezer.scrooge.hive.RichExecutionObject$$anonfun$fromHive$1$$anonfun$apply$76.apply(HiveSpec.scala:315)\r\n\tat com.twitter.scalding.Execution$FutureConst$$anonfun$runStats$1$$anonfun$1.apply(Execution.scala:251)\r\n\tat com.twitter.scalding.Execution$FutureConst$$anonfun$runStats$1$$anonfun$1.apply(Execution.scala:251)\r\n\tat scala.util.Try$.apply(Try.scala:161)\r\n\tat com.twitter.scalding.Execution$FutureConst$$anonfun$runStats$1.apply(Execution.scala:251)\r\n\tat com.twitter.scalding.Execution$FutureConst$$anonfun$runStats$1.apply(Execution.scala:250)\r\n\tat com.twitter.scalding.Execution$MapEvalCache.getOrElseInsert(Execution.scala:224)\r\n\tat com.twitter.scalding.Execution$FutureConst.runStats(Execution.scala:250)\r\n\tat com.twitter.scalding.Execution$FlatMapped$$anonfun$runStats$2$$anonfun$3.apply(Execution.scala:268)\r\n\tat com.twitter.scalding.Execution$FlatMapped$$anonfun$runStats$2$$anonfun$3.apply(Execution.scala:266)\r\n\tat scala.util.Success$$anonfun$map$1.apply(Try.scala:206)\r\n\tat scala.util.Try$.apply(Try.scala:161)\r\n\tat scala.util.Success.map(Try.scala:206)\r\n\tat scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)\r\n\tat scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)\r\n\tat scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\nCaused by: MetaException(message:Metastore contains multiple versions)\r\n\tat org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6373)\r\n\tat org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6335)\r\n\tat org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6294)\r\n\tat org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6282)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)\r\n\tat com.sun.proxy.$Proxy30.verifySchema(Unknown Source)\r\n\tat org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:485)\r\n\tat org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database(HiveMetaStore.java:805)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:106)\r\n\tat com.sun.proxy.$Proxy32.get_database(Unknown Source)\r\n\tat org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getDatabase(HiveMetaStoreClient.java:960)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:90)\r\n\tat com.sun.proxy.$Proxy33.getDatabase(Unknown Source)\r\n\tat org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1188)\r\n\t... 33 more\r\n```"}}