Optimize multiple Hive QL in Oozie
<p>I am not familiar enough with hive, so here I am. We are using Oozie to chain a bunch of hive ql jobs together. I was tasked to optimize an application that has already been running in our production environment. The Business Partners don't want it to take longer than like 1.5 hours. One of the first things I noticed was that there are around 90 oozie actions within this one work flow. We also share a yarn queue with other applications. Half of those actions are hive2 actions, and each of the Hive QL actions only do one HQL statement. It seems sometimes there are delays between HiveQL actions because the Oozie launcher waits in a queue, and then the HiveQL waits in a queue. Is that normal? Are there ways around this.</p>

<p>For time sensitive Hive Queries:
1) Is Oozie the right tool we should use for chaining time sensitive HiveQL scripts together
2) What are some Alternatives ( Could using Java or Python to launch and handle the flow between HQL have a performance benefit)?
3) Is there something we can do in HQL itself? (Again, I'm new to hive, primarily experience with MapReduce/Spark and simple workflows (less than 20 actions)
4) Are there any other performance considerations that I didn't mention?</p>

<p>Thank you,</p>