Wednesday, 6 January 2016

Interview Questions & Answers on Apache Spark [Part 1]

Q1: When do you use apache spark? OR  What are the benefits of Spark over Mapreduce?
  4. In Spark, you can basically do everything using single application / console (pyspark or scala console) and get  the results immediately. Switching between 'Running something on cluster' and 'doing something locally' is fairly easy and straightforward. This also leads to less context switch of the developer and more productivity.
  5. Spark kind of equals to MapReduce and Oozie put together.

Q2: Is there are point of learning Mapreduce, then?
Ans: Yes. For the following reason: 

Q3: When running Spark on Yarn, do I need to install Spark on all nodes of Yarn Cluster?
Since spark runs on top of Yarn, it utilizes yarn for the execution of its commands over the cluster's nodes.
So, you just have to install Spark on one node.

Q4: What are the downsides of Spark?
  1. She may end up running everything on the local node instead of distributing work over to the cluster.
  2. She might hit some webservice too many times by the way of using multiple clusters.


Q5: What is a RDD?
  1. Immutable - You can operate on the rdd to produce another rdd but you can’t alter it.
  2. Partitioned / Parallel - The data located on RDD is operated in parallel. Any operation on RDD is done using multiple nodes.
  3. Resilience - If one of the node hosting the partition fails, another nodes takes its data.

RDD provides two kinds of operations: Transformations and Actions.

Q6: What is Transformations?

The example of transformations are:
  1. map() - applies the function passed to it on each element of RDD resulting in a new RDD.
  2. filter() - creates a new RDD by picking the elements from the current RDD which pass the function argument.

Q7: What are Actions?
  1. reduce() - executes the function passed again and again until only one value is left. The function should take two argument and return one value.
  2. take() - take all the values back to the local node form RDD.

Related Posts Plugin for WordPress, Blogger...