{"action":"created","issue":{"url":"https://api.github.com/repos/berkeley-dsc/dlab-finance/issues/7","labels_url":"https://api.github.com/repos/berkeley-dsc/dlab-finance/issues/7/labels{/name}","comments_url":"https://api.github.com/repos/berkeley-dsc/dlab-finance/issues/7/comments","events_url":"https://api.github.com/repos/berkeley-dsc/dlab-finance/issues/7/events","html_url":"https://github.com/berkeley-dsc/dlab-finance/issues/7","id":59883274,"number":7,"title":"Step 1: Efficiently extract some set of securities","user":{"login":"davclark","id":120710,"avatar_url":"https://avatars.githubusercontent.com/u/120710?v=3","gravatar_id":"","url":"https://api.github.com/users/davclark","html_url":"https://github.com/davclark","followers_url":"https://api.github.com/users/davclark/followers","following_url":"https://api.github.com/users/davclark/following{/other_user}","gists_url":"https://api.github.com/users/davclark/gists{/gist_id}","starred_url":"https://api.github.com/users/davclark/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davclark/subscriptions","organizations_url":"https://api.github.com/users/davclark/orgs","repos_url":"https://api.github.com/users/davclark/repos","events_url":"https://api.github.com/users/davclark/events{/privacy}","received_events_url":"https://api.github.com/users/davclark/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"rsoni1","id":8722919,"avatar_url":"https://avatars.githubusercontent.com/u/8722919?v=3","gravatar_id":"","url":"https://api.github.com/users/rsoni1","html_url":"https://github.com/rsoni1","followers_url":"https://api.github.com/users/rsoni1/followers","following_url":"https://api.github.com/users/rsoni1/following{/other_user}","gists_url":"https://api.github.com/users/rsoni1/gists{/gist_id}","starred_url":"https://api.github.com/users/rsoni1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rsoni1/subscriptions","organizations_url":"https://api.github.com/users/rsoni1/orgs","repos_url":"https://api.github.com/users/rsoni1/repos","events_url":"https://api.github.com/users/rsoni1/events{/privacy}","received_events_url":"https://api.github.com/users/rsoni1/received_events","type":"User","site_admin":false},"milestone":null,"comments":9,"created_at":"2015-03-05T00:00:54Z","updated_at":"2015-03-16T08:09:02Z","closed_at":null,"body":"At this step, we can be doing a trivial computation, such as the computation of daily minimums. The structure of the TAQ data files (each security in a \"clump\", as I understand it) may allow us to do this efficiently without an indexed data format. Or, we may want to convert our data to a different format (e.g., parquet, HDF5).\r\n\r\ncc @nitalpatwa"},"comment":{"url":"https://api.github.com/repos/berkeley-dsc/dlab-finance/issues/comments/81480663","html_url":"https://github.com/berkeley-dsc/dlab-finance/issues/7#issuecomment-81480663","issue_url":"https://api.github.com/repos/berkeley-dsc/dlab-finance/issues/7","id":81480663,"user":{"login":"nitalpatwa","id":8423067,"avatar_url":"https://avatars.githubusercontent.com/u/8423067?v=3","gravatar_id":"","url":"https://api.github.com/users/nitalpatwa","html_url":"https://github.com/nitalpatwa","followers_url":"https://api.github.com/users/nitalpatwa/followers","following_url":"https://api.github.com/users/nitalpatwa/following{/other_user}","gists_url":"https://api.github.com/users/nitalpatwa/gists{/gist_id}","starred_url":"https://api.github.com/users/nitalpatwa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nitalpatwa/subscriptions","organizations_url":"https://api.github.com/users/nitalpatwa/orgs","repos_url":"https://api.github.com/users/nitalpatwa/repos","events_url":"https://api.github.com/users/nitalpatwa/events{/privacy}","received_events_url":"https://api.github.com/users/nitalpatwa/received_events","type":"User","site_admin":false},"created_at":"2015-03-16T08:09:02Z","updated_at":"2015-03-16T08:09:02Z","body":"I tried rebuilding Maven with following flags (as per notes found on spark\npage), but still running out heap space\nwhen executing example. The node I have had 16GB RAM.\n\nexport MAVEN_OPTS=\"-Xmx4g -XX:MaxPermSize=512M\n-XX:ReservedCodeCacheSize=512m\"\ncd com-cotdp-hadoop-master/\nmvn clean package\n\nIf you are familiar with Maven, is above method of building the jar okay?\n\nNital\n\n\n\n\nOn Mon, Mar 16, 2015 at 12:26 AM, Nital P. Patwa <npatwa@berkeley.edu>\nwrote:\n\n> I built the JAR with Maven, but while running in pyspark shell, ran out of\n> heap space.\n> Interestingly on the driver machine/name-node machine almost all of 16GB\n> is taken. Wonder who is taking it.\n> Nital\n>\n>\n> On Sun, Mar 15, 2015 at 7:30 PM, Nital P. Patwa <npatwa@berkeley.edu>\n> wrote:\n>\n>> Dav,\n>> So far I have been using only python and the interactive shell.\n>> 1) How do I build the hadoop-zip.jar?  The com-cotdp-hadoop github is\n>> there but not sure how to deal with it.\n>> 2) Let's say that above .jar exists, how would it work with interactive\n>> shell? Will it be similar to this?\n>>\n>> SPARK_CLASSPATH=/path/to/elasticsearch-hadoop.jar ./bin/pyspark\n>>\n>> Nital\n>>\n>>\n>> On Sun, Mar 15, 2015 at 7:00 PM, Dav Clark <notifications@github.com>\n>> wrote:\n>>\n>>> There is example code already in the repository implementing the\n>>> compression codec logic!\n>>>\n>>> Regarding splitting, I'm guessing it won't be a problem, but you'll need\n>>> more logic to recombine days that are split across files.\n>>>\n>>> D\n>>>\n>>> On Mar 15, 2015 6:49 PM, \"nitalpatwa\" <notifications@github.com> wrote:\n>>>\n>>> > I read about compressed files and hadoop/spark but the following\n>>> questions\n>>> > are unanswered:\n>>> > 1) zip file: Is it splittable? i.e. can it be handled without having to\n>>> > read it in entirety on one node?\n>>> > I am finding that the files have a very large compression ratio, so a\n>>> 6GB\n>>> > file may easily overshoot the memory,\n>>> > of my 16GB node on softlayer.\n>>> > 2) Spark talks about having hadoopFile API, and giving a compression\n>>> codec:\n>>> >\n>>> > Warning While Spark's textFile() method can handle compressed input, it\n>>> > automatically disables splittable even if the input is compressed such\n>>> that\n>>> > it could be read in a splittable way. If you find yourself needing to\n>>> read\n>>> > in a large single-file compressed input, consider skipping Spark's\n>>> wrapper\n>>> > and instead use either newAPIHadoopFile or hadoopFile and specify the\n>>> > correct compression codec.\n>>> >\n>>> > But it is not clear what exactly is the API call in python.\n>>> > Nital\n>>> >\n>>> > On Sun, Mar 15, 2015 at 5:49 AM, Ritesh Soni <notifications@github.com\n>>> >\n>>> > wrote:\n>>> >\n>>> > > ok. Although cluster issues have now been resolved and it's blazingly\n>>> > fast\n>>> > > without any crashes like the ones we experienced earlier. You can run\n>>> > Spark\n>>> > > interactively on the cluster just\n>>> > >\n>>> > > export MASTER=$SPARK_URL\n>>> > >\n>>> > > before starting pyspark.\n>>> > >\n>>> > > --\n>>> > > Reply to this email directly or view it on GitHub\n>>> > > <\n>>> >\n>>> https://github.com/berkeley-dsc/dlab-finance/issues/7#issuecomment-80997465\n>>> > >\n>>> > > .\n>>> > >\n>>> >\n>>> > --\n>>> > Reply to this email directly or view it on GitHub\n>>> > <\n>>> https://github.com/berkeley-dsc/dlab-finance/issues/7#issuecomment-81340575\n>>> >\n>>>\n>>> > .\n>>> >\n>>>\n>>> —\n>>> Reply to this email directly or view it on GitHub\n>>> <https://github.com/berkeley-dsc/dlab-finance/issues/7#issuecomment-81346610>\n>>> .\n>>>\n>>\n>>\n>\n"}}