{"action":"created","issue":{"url":"https://api.github.com/repos/benchflow/analysers/issues/4","labels_url":"https://api.github.com/repos/benchflow/analysers/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/benchflow/analysers/issues/4/comments","events_url":"https://api.github.com/repos/benchflow/analysers/issues/4/events","html_url":"https://github.com/benchflow/analysers/issues/4","id":121847943,"number":4,"title":"Evaluate the performance of having multiple analysers instead of fiew ones","user":{"login":"VincenzoFerme","id":7163641,"avatar_url":"https://avatars.githubusercontent.com/u/7163641?v=3","gravatar_id":"","url":"https://api.github.com/users/VincenzoFerme","html_url":"https://github.com/VincenzoFerme","followers_url":"https://api.github.com/users/VincenzoFerme/followers","following_url":"https://api.github.com/users/VincenzoFerme/following{/other_user}","gists_url":"https://api.github.com/users/VincenzoFerme/gists{/gist_id}","starred_url":"https://api.github.com/users/VincenzoFerme/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VincenzoFerme/subscriptions","organizations_url":"https://api.github.com/users/VincenzoFerme/orgs","repos_url":"https://api.github.com/users/VincenzoFerme/repos","events_url":"https://api.github.com/users/VincenzoFerme/events{/privacy}","received_events_url":"https://api.github.com/users/VincenzoFerme/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/benchflow/analysers/labels/needs%20discussion","name":"needs discussion","color":"bfe5bf"}],"state":"open","locked":false,"assignee":{"login":"Cerfoglg","id":8819678,"avatar_url":"https://avatars.githubusercontent.com/u/8819678?v=3","gravatar_id":"","url":"https://api.github.com/users/Cerfoglg","html_url":"https://github.com/Cerfoglg","followers_url":"https://api.github.com/users/Cerfoglg/followers","following_url":"https://api.github.com/users/Cerfoglg/following{/other_user}","gists_url":"https://api.github.com/users/Cerfoglg/gists{/gist_id}","starred_url":"https://api.github.com/users/Cerfoglg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Cerfoglg/subscriptions","organizations_url":"https://api.github.com/users/Cerfoglg/orgs","repos_url":"https://api.github.com/users/Cerfoglg/repos","events_url":"https://api.github.com/users/Cerfoglg/events{/privacy}","received_events_url":"https://api.github.com/users/Cerfoglg/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/benchflow/analysers/milestones/2","html_url":"https://github.com/benchflow/analysers/milestones/Next%20Tasks","labels_url":"https://api.github.com/repos/benchflow/analysers/milestones/2/labels","id":1366593,"number":2,"title":"Next Tasks","description":"Stuff we’ll probably do soon. We never change or rename this milestone.","creator":{"login":"VincenzoFerme","id":7163641,"avatar_url":"https://avatars.githubusercontent.com/u/7163641?v=3","gravatar_id":"","url":"https://api.github.com/users/VincenzoFerme","html_url":"https://github.com/VincenzoFerme","followers_url":"https://api.github.com/users/VincenzoFerme/followers","following_url":"https://api.github.com/users/VincenzoFerme/following{/other_user}","gists_url":"https://api.github.com/users/VincenzoFerme/gists{/gist_id}","starred_url":"https://api.github.com/users/VincenzoFerme/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VincenzoFerme/subscriptions","organizations_url":"https://api.github.com/users/VincenzoFerme/orgs","repos_url":"https://api.github.com/users/VincenzoFerme/repos","events_url":"https://api.github.com/users/VincenzoFerme/events{/privacy}","received_events_url":"https://api.github.com/users/VincenzoFerme/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":0,"state":"open","created_at":"2015-10-20T16:14:01Z","updated_at":"2015-12-12T10:38:27Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2015-12-12T10:38:27Z","updated_at":"2015-12-15T20:38:11Z","closed_at":null,"body":"We would like to have a large number of dedicated analysers to have more cohesive code. We should consider and evaluate the possible impact on the performance of the execution:\r\n- data must be kept in memory, once loaded by the first analyser requiring it (Spark should offer this out of the box)\r\n- we must load only the data useful for the computation\r\n- we should let Cassandra do most of the computation for us, when it is not needed to load the data and it is faster to be executed on cassandra"},"comment":{"url":"https://api.github.com/repos/benchflow/analysers/issues/comments/164889617","html_url":"https://github.com/benchflow/analysers/issues/4#issuecomment-164889617","issue_url":"https://api.github.com/repos/benchflow/analysers/issues/4","id":164889617,"user":{"login":"Cerfoglg","id":8819678,"avatar_url":"https://avatars.githubusercontent.com/u/8819678?v=3","gravatar_id":"","url":"https://api.github.com/users/Cerfoglg","html_url":"https://github.com/Cerfoglg","followers_url":"https://api.github.com/users/Cerfoglg/followers","following_url":"https://api.github.com/users/Cerfoglg/following{/other_user}","gists_url":"https://api.github.com/users/Cerfoglg/gists{/gist_id}","starred_url":"https://api.github.com/users/Cerfoglg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Cerfoglg/subscriptions","organizations_url":"https://api.github.com/users/Cerfoglg/orgs","repos_url":"https://api.github.com/users/Cerfoglg/repos","events_url":"https://api.github.com/users/Cerfoglg/events{/privacy}","received_events_url":"https://api.github.com/users/Cerfoglg/received_events","type":"User","site_admin":false},"created_at":"2015-12-15T20:38:11Z","updated_at":"2015-12-15T20:38:11Z","body":"@VincenzoFerme \r\n\r\n* I'm not entirely sure that Spark can persist data across multiple scripts. You can persist an RDD across operations as described here http://spark.apache.org/docs/latest/programming-guide.html#rdd-persistence but I believe it assumes it's within the same script, unless you look at the off heap option, but that might be problematic.\r\n* If we set up Cassandra correctly, with the right query we should only pull the data we need without issues\r\n* This, absolutely.\r\n\r\nConsidering the way Spark caches data, it appears that data does not persist across multiple scripts, as RDDs don't persist across them. This means we may have to have all metrics for a certain set of data to be computed within the same script. Of course, nothing stops us from organising our code so that we have different files for every metric, but we will need to combine them so that we launch a single script per data set as opposed to a script per singular metric."}}