{"action":"created","issue":{"url":"https://api.github.com/repos/lintool/warcbase/issues/170","labels_url":"https://api.github.com/repos/lintool/warcbase/issues/170/labels{/name}","comments_url":"https://api.github.com/repos/lintool/warcbase/issues/170/comments","events_url":"https://api.github.com/repos/lintool/warcbase/issues/170/events","html_url":"https://github.com/lintool/warcbase/issues/170","id":118498928,"number":170,"title":"Spark job taking too long","user":{"login":"jrwiebe","id":4625549,"avatar_url":"https://avatars.githubusercontent.com/u/4625549?v=3","gravatar_id":"","url":"https://api.github.com/users/jrwiebe","html_url":"https://github.com/jrwiebe","followers_url":"https://api.github.com/users/jrwiebe/followers","following_url":"https://api.github.com/users/jrwiebe/following{/other_user}","gists_url":"https://api.github.com/users/jrwiebe/gists{/gist_id}","starred_url":"https://api.github.com/users/jrwiebe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jrwiebe/subscriptions","organizations_url":"https://api.github.com/users/jrwiebe/orgs","repos_url":"https://api.github.com/users/jrwiebe/repos","events_url":"https://api.github.com/users/jrwiebe/events{/privacy}","received_events_url":"https://api.github.com/users/jrwiebe/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/lintool/warcbase/labels/documentation","name":"documentation","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-11-23T23:35:08Z","updated_at":"2015-11-24T20:37:42Z","closed_at":null,"body":"I've had a simple Spark job running for 7 hours now, when it should have been completed in minutes. (I didn't realize quite how slowly it was going at first; later I became curious to see how long it would take.)\r\n\r\nI ran the following from the Spark shell: \r\n```\r\nimport org.warcbase.spark.matchbox.RecordLoader\r\nimport org.warcbase.spark.rdd.RecordRDD._\r\n\r\nRecordLoader.loadArc(\"/collections/webarchives/CanadianPoliticalParties/arc\", sc)\r\n  .keepValidPages()\r\n  .keepDomains(Set(\"greenparty.ca\"))\r\n  .map(r => (r.getCrawldate, r.getDomain, r.getUrl, r.getRawBodyContent))\r\n  .saveAsTextFile(\"cpp.greenparty-arc-plaintext\")\r\n```\r\n\r\nThe monitoring web app says it's running on 2 executors, and that the \"active stage\" is saveAsTextFile. If I'm reading the stats correctly, almost all the run time has involved output.\r\n\r\nObviously something is going wrong here.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/lintool/warcbase/issues/comments/159398131","html_url":"https://github.com/lintool/warcbase/issues/170#issuecomment-159398131","issue_url":"https://api.github.com/repos/lintool/warcbase/issues/170","id":159398131,"user":{"login":"ianmilligan1","id":3834704,"avatar_url":"https://avatars.githubusercontent.com/u/3834704?v=3","gravatar_id":"","url":"https://api.github.com/users/ianmilligan1","html_url":"https://github.com/ianmilligan1","followers_url":"https://api.github.com/users/ianmilligan1/followers","following_url":"https://api.github.com/users/ianmilligan1/following{/other_user}","gists_url":"https://api.github.com/users/ianmilligan1/gists{/gist_id}","starred_url":"https://api.github.com/users/ianmilligan1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ianmilligan1/subscriptions","organizations_url":"https://api.github.com/users/ianmilligan1/orgs","repos_url":"https://api.github.com/users/ianmilligan1/repos","events_url":"https://api.github.com/users/ianmilligan1/events{/privacy}","received_events_url":"https://api.github.com/users/ianmilligan1/received_events","type":"User","site_admin":false},"created_at":"2015-11-24T20:37:42Z","updated_at":"2015-11-24T20:37:42Z","body":"Just to clarify - as I'd like to put this into the documentation - if say I'm on a 8-core EC2 instance, I could pass it as:\r\n\r\n`spark-shell --jars target/warcbase-0.1.0-SNAPSHOT-fatjar.jar --num-executors 8`\r\n\r\nIf that sounds reasonable, I say we can close this and I'll update documentation."}}