{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-zeppelin/issues/80","labels_url":"https://api.github.com/repos/apache/incubator-zeppelin/issues/80/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-zeppelin/issues/80/comments","events_url":"https://api.github.com/repos/apache/incubator-zeppelin/issues/80/events","html_url":"https://github.com/apache/incubator-zeppelin/pull/80","id":79947117,"number":80,"title":"Add Scala utility functions for display","user":{"login":"doanduyhai","id":1532977,"avatar_url":"https://avatars.githubusercontent.com/u/1532977?v=3","gravatar_id":"","url":"https://api.github.com/users/doanduyhai","html_url":"https://github.com/doanduyhai","followers_url":"https://api.github.com/users/doanduyhai/followers","following_url":"https://api.github.com/users/doanduyhai/following{/other_user}","gists_url":"https://api.github.com/users/doanduyhai/gists{/gist_id}","starred_url":"https://api.github.com/users/doanduyhai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/doanduyhai/subscriptions","organizations_url":"https://api.github.com/users/doanduyhai/orgs","repos_url":"https://api.github.com/users/doanduyhai/repos","events_url":"https://api.github.com/users/doanduyhai/events{/privacy}","received_events_url":"https://api.github.com/users/doanduyhai/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-05-23T20:36:04Z","updated_at":"2015-06-12T11:40:44Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/incubator-zeppelin/pulls/80","html_url":"https://github.com/apache/incubator-zeppelin/pull/80","diff_url":"https://github.com/apache/incubator-zeppelin/pull/80.diff","patch_url":"https://github.com/apache/incubator-zeppelin/pull/80.patch"},"body":"Until now, to display data as a table, there are 2 alternatives:\r\n\r\n1. Either use **Spark DataFrame** and Zeppeline built-in support\r\n2. Or generate manually a `println(%table ...)`. As an example of displaying an `RDD[(String,String,Int)]` representing a collection of users:\r\n\r\n```scala\r\nval data = new java.lang.StringBuilder(\"%table Login\\tName\\tAge\\n\")\r\nrdd.foreach {\r\n   case (login,name,age) => data.append(s\"$login\\t$name\\t$age\\n\")\r\n}\r\n\r\nprintln(data.toString())\r\n```\r\n\r\nMy proposal is to add a new utility function to make creating tables easier that the code example above. Of course one can always use **Spark DataFrame** but I find it quite restrictive. People using Spark versions lesser than 1.3 cannot rely on DataFrame and sometimes one does not want to transform an RDD to DataFrame for display.\r\n\r\nHow are the utility functions implemented ?\r\n\r\n1. I added a new module **spark-utils** which provide Scala code for display utility functions. This module will use the **maven-scala-plugin** to compile all the classes in package `org.apache.zeppelin.spark.utils`. \r\n\r\n2. Right now the package `org.apache.zeppelin.spark.utils` only contains 1 object `DisplayUtils` which augments RDDs/**Scala Traversable** of Tuples or case classes (all of them sub-class of trait `Product`) with the new method `displayAsTable(columnLabels: String*)`.\r\n\r\n3. The `DisplayUtils` object is imported automatically into the `SparkInterpreter` with `intp.interpret(\"import org.apache.zeppelin.spark.utils.DisplayUtils._\");`\r\n\r\n4. The Maven module **interpreter** will now have a **runtime** dependency on the module **spark-utils** so that the utility class will be loaded at runtime\r\n\r\n5. Usage of the new display utility function is:\r\n\r\n    **Paragraph1**\r\n    ```scala\r\n    case class Person(login: String, name: String, age: Int)\r\n    val rddTuples:RDD[(String,String,Int)] = sc.parallelize(List((\"jdoe\",\"John DOE\",32),(\"hsue\",\"Helen     SUE\",27))\r\n    val rddCaseClass:RDD[Person] = sc.parallelize(List(Person(\"jdoe\",\"John DOE\",32),Person(\"hsue\",\"Helen SUE\",27))\r\n    ```\r\n    **Paragraph2**\r\n    ```scala\r\n    rddTuples.displayAsTable(\"Login\",\"Name\",\"Age\")\r\n    ```\r\n    \r\n    **Paragraph3**\r\n    ```scala\r\n    rddCaseClass.displayAsTable(\"Login\",\"Name\",\"Age\")\r\n    ```\r\n\r\n6. The `displayAsTable()` method is error-proof, meaning that if the user provides **more** columns label that the number of elements in the tuples/case class, the extra column labels will ignored. If the user provides **less** column labels than expected, the method will pad missing column headers with **Column2**, **Column3** etc ...\r\n\r\n7. In addition to the `displayAsTable` methods, I added some other utility methods to make it easier to handle custom HTML and images:\r\n    a. calling `html()` will generate the string `\"%html \"`\r\n    b. calling `html(\"<p> This is a test</p>)` will generate the string `\"%html <p> This is a test</p>\"`\r\n    c. calling `img(\"http://www.google.com\")` will generate the string `\"<img src='http://www.google.com' />\"`\r\n   d. calling `img64()` will generate the string `\"%img \"`\r\n   e. calling `img64(\"ABCDE123\")` will generate the string `\"%img ABCDE123\"`\r\n\r\nOf course the `DisplayUtils` object can be extended with new other functions to support future advanced displaying features\r\n"},"comment":{"url":"https://api.github.com/repos/apache/incubator-zeppelin/issues/comments/111465199","html_url":"https://github.com/apache/incubator-zeppelin/pull/80#issuecomment-111465199","issue_url":"https://api.github.com/repos/apache/incubator-zeppelin/issues/80","id":111465199,"user":{"login":"doanduyhai","id":1532977,"avatar_url":"https://avatars.githubusercontent.com/u/1532977?v=3","gravatar_id":"","url":"https://api.github.com/users/doanduyhai","html_url":"https://github.com/doanduyhai","followers_url":"https://api.github.com/users/doanduyhai/followers","following_url":"https://api.github.com/users/doanduyhai/following{/other_user}","gists_url":"https://api.github.com/users/doanduyhai/gists{/gist_id}","starred_url":"https://api.github.com/users/doanduyhai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/doanduyhai/subscriptions","organizations_url":"https://api.github.com/users/doanduyhai/orgs","repos_url":"https://api.github.com/users/doanduyhai/repos","events_url":"https://api.github.com/users/doanduyhai/events{/privacy}","received_events_url":"https://api.github.com/users/doanduyhai/received_events","type":"User","site_admin":false},"created_at":"2015-06-12T11:40:44Z","updated_at":"2015-06-12T11:40:44Z","body":"Hello @Leemoonsoo \r\n\r\nSo,\r\n\r\n1) I've renamed `displayAsTable()` to `display()` as suggested\r\n2) I've introduce a default limit to RDD ( `rdd.take()`) using the `zeppeline.spark.maxResult` property\r\n3) User can override the default limit to pass in their own limit value in the `display()` method\r\n4) There is no default limit for **Scala** collection\r\n5) I've rebased from master branch\r\n\r\nHere is a screen capture showing the new default limit: \r\n![image](https://cloud.githubusercontent.com/assets/1532977/8129319/6fe1f298-1108-11e5-9708-f7891ea52fa1.png)\r\n\r\nFor code normalization, I would prefer we merge and close this pull request and open a new JIRA to prepare the port of ZeppelinContext class in **Scala** with appropriate unit tests.\r\n\r\nBaby steps.\r\n\r\n\r\n "}}