{"action":"closed","issue":{"url":"https://api.github.com/repos/benchflow/data-transformers/issues/1","labels_url":"https://api.github.com/repos/benchflow/data-transformers/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/benchflow/data-transformers/issues/1/comments","events_url":"https://api.github.com/repos/benchflow/data-transformers/issues/1/events","html_url":"https://github.com/benchflow/data-transformers/issues/1","id":112406444,"number":1,"title":"Develop Tasks that read a mysql dump, transform the data and store them in Cassandra","user":{"login":"VincenzoFerme","id":7163641,"avatar_url":"https://avatars.githubusercontent.com/u/7163641?v=3","gravatar_id":"","url":"https://api.github.com/users/VincenzoFerme","html_url":"https://github.com/VincenzoFerme","followers_url":"https://api.github.com/users/VincenzoFerme/followers","following_url":"https://api.github.com/users/VincenzoFerme/following{/other_user}","gists_url":"https://api.github.com/users/VincenzoFerme/gists{/gist_id}","starred_url":"https://api.github.com/users/VincenzoFerme/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VincenzoFerme/subscriptions","organizations_url":"https://api.github.com/users/VincenzoFerme/orgs","repos_url":"https://api.github.com/users/VincenzoFerme/repos","events_url":"https://api.github.com/users/VincenzoFerme/events{/privacy}","received_events_url":"https://api.github.com/users/VincenzoFerme/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/benchflow/data-transformers/labels/feature","name":"feature","color":"eb6420"}],"state":"closed","locked":false,"assignee":{"login":"Cerfoglg","id":8819678,"avatar_url":"https://avatars.githubusercontent.com/u/8819678?v=3","gravatar_id":"","url":"https://api.github.com/users/Cerfoglg","html_url":"https://github.com/Cerfoglg","followers_url":"https://api.github.com/users/Cerfoglg/followers","following_url":"https://api.github.com/users/Cerfoglg/following{/other_user}","gists_url":"https://api.github.com/users/Cerfoglg/gists{/gist_id}","starred_url":"https://api.github.com/users/Cerfoglg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Cerfoglg/subscriptions","organizations_url":"https://api.github.com/users/Cerfoglg/orgs","repos_url":"https://api.github.com/users/Cerfoglg/repos","events_url":"https://api.github.com/users/Cerfoglg/events{/privacy}","received_events_url":"https://api.github.com/users/Cerfoglg/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/benchflow/data-transformers/milestones/1","html_url":"https://github.com/benchflow/data-transformers/milestones/0.0.1","labels_url":"https://api.github.com/repos/benchflow/data-transformers/milestones/1/labels","id":1366601,"number":1,"title":"0.0.1","description":"","creator":{"login":"VincenzoFerme","id":7163641,"avatar_url":"https://avatars.githubusercontent.com/u/7163641?v=3","gravatar_id":"","url":"https://api.github.com/users/VincenzoFerme","html_url":"https://github.com/VincenzoFerme","followers_url":"https://api.github.com/users/VincenzoFerme/followers","following_url":"https://api.github.com/users/VincenzoFerme/following{/other_user}","gists_url":"https://api.github.com/users/VincenzoFerme/gists{/gist_id}","starred_url":"https://api.github.com/users/VincenzoFerme/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VincenzoFerme/subscriptions","organizations_url":"https://api.github.com/users/VincenzoFerme/orgs","repos_url":"https://api.github.com/users/VincenzoFerme/repos","events_url":"https://api.github.com/users/VincenzoFerme/events{/privacy}","received_events_url":"https://api.github.com/users/VincenzoFerme/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":3,"state":"open","created_at":"2015-10-20T16:20:08Z","updated_at":"2015-11-28T16:00:03Z","due_on":"2015-11-14T23:00:00Z","closed_at":null},"comments":2,"created_at":"2015-10-20T16:20:51Z","updated_at":"2015-11-28T16:00:03Z","closed_at":"2015-11-28T16:00:03Z","body":"Develop a task that:\r\n- ~~read a mysql dump file, obtained from Minio. Note that we store compressed data on Minio, but it should be possible to directly read compressed file in Spark;~~\r\n- ~~**extract** some data from it by specifying the: table and the columns we want to extract;~~\r\n- [Depends on #6] **transform** the data to a custom format used by BenchFlow ([Custom format reference](https://dl.dropboxusercontent.com/u/7530328/github/2015_BenchFlow_Msc_Marco_Argenti.pdf). This reference uses a MySQL database, we should port the format to Cassandra, by taking care of saving the data related to a benchmark_id in the same place. Some hints can be obtained from: http://blog.rackspace.com/cassandra-by-example/)\r\n- ~~**load** the data in a Cassandra database~~\r\n\r\n~~Give a look at [Spark SQL](https://spark.apache.org/sql/) providing a powerful tool for performing ETL on data. A nice example can be found on the following link: http://chapeau.freevariable.com/2014/10/fedmsg-and-spark.html~~\r\n\r\nThis task is a starting point to understand what we need to do in order to simplify this process for the users, by providing an already implemented library that does the most.\r\n\r\n ~~**Something to evaluate**:~~\r\n-  ~~Since Spark and Spark SQL has native support for csv file (as this nice example shows: http://stackoverflow.com/a/25366955), we can also think about performing dumps also in csv (functionalities that is largely supported by dump tools for databases) an work with them instead of sql dumps. For doing it I would use the same collector that does the mysqldump.~~\r\n\r\nPay attention to:\r\n- reuse the code whenever it is possible, to simplify the implementation of similar transformation for different database schemas"}}