{"action":"created","issue":{"url":"https://api.github.com/repos/spotify/luigi/issues/963","labels_url":"https://api.github.com/repos/spotify/luigi/issues/963/labels{/name}","comments_url":"https://api.github.com/repos/spotify/luigi/issues/963/comments","events_url":"https://api.github.com/repos/spotify/luigi/issues/963/events","html_url":"https://github.com/spotify/luigi/issues/963","id":78775498,"number":963,"title":"Luigi memory consumption causing spark jobs to crash","user":{"login":"cqnguyen","id":1943572,"avatar_url":"https://avatars.githubusercontent.com/u/1943572?v=3","gravatar_id":"","url":"https://api.github.com/users/cqnguyen","html_url":"https://github.com/cqnguyen","followers_url":"https://api.github.com/users/cqnguyen/followers","following_url":"https://api.github.com/users/cqnguyen/following{/other_user}","gists_url":"https://api.github.com/users/cqnguyen/gists{/gist_id}","starred_url":"https://api.github.com/users/cqnguyen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cqnguyen/subscriptions","organizations_url":"https://api.github.com/users/cqnguyen/orgs","repos_url":"https://api.github.com/users/cqnguyen/repos","events_url":"https://api.github.com/users/cqnguyen/events{/privacy}","received_events_url":"https://api.github.com/users/cqnguyen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-05-21T01:36:36Z","updated_at":"2015-06-04T19:49:22Z","closed_at":null,"body":"Hey guys,\r\n\r\nJust discovered that when running luigi with spark, found my jobs were failing due to OOM errors. When running the same job without luigi in spark, the job succeeds successfully.\r\n\r\nSpark jobs with large data sets fail when running via luigi. Small spark jobs relative to cluster size succeed when running through luigi. \r\n\r\nCheers!\r\nCory"},"comment":{"url":"https://api.github.com/repos/spotify/luigi/issues/comments/109029492","html_url":"https://github.com/spotify/luigi/issues/963#issuecomment-109029492","issue_url":"https://api.github.com/repos/spotify/luigi/issues/963","id":109029492,"user":{"login":"Tarrasch","id":294349,"avatar_url":"https://avatars.githubusercontent.com/u/294349?v=3","gravatar_id":"","url":"https://api.github.com/users/Tarrasch","html_url":"https://github.com/Tarrasch","followers_url":"https://api.github.com/users/Tarrasch/followers","following_url":"https://api.github.com/users/Tarrasch/following{/other_user}","gists_url":"https://api.github.com/users/Tarrasch/gists{/gist_id}","starred_url":"https://api.github.com/users/Tarrasch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Tarrasch/subscriptions","organizations_url":"https://api.github.com/users/Tarrasch/orgs","repos_url":"https://api.github.com/users/Tarrasch/repos","events_url":"https://api.github.com/users/Tarrasch/events{/privacy}","received_events_url":"https://api.github.com/users/Tarrasch/received_events","type":"User","site_admin":false},"created_at":"2015-06-04T19:49:22Z","updated_at":"2015-06-04T19:49:22Z","body":"I continue to have a hard time seeing how luigi could relate to this. First, luigi is just running a given command line when shelling out to spark (I assume), second, the failure is on the cluster, the jobs running there don't know about luigi, right? Last, I've heard over and over again that spark is still a bit beta and that it's suddenly starts to hog memory.\r\n\r\nNow with this said, the problem can still be with luigi of course, but it wouldn't be the first place I would look at when debugging this."}}