{"action":"created","issue":{"url":"https://api.github.com/repos/BIDData/BIDMach/issues/44","labels_url":"https://api.github.com/repos/BIDData/BIDMach/issues/44/labels{/name}","comments_url":"https://api.github.com/repos/BIDData/BIDMach/issues/44/comments","events_url":"https://api.github.com/repos/BIDData/BIDMach/issues/44/events","html_url":"https://github.com/BIDData/BIDMach/issues/44","id":71365149,"number":44,"title":"java.lang.RuntimeException: Cuda error in GSMat() too many resources requested for launch","user":{"login":"ziqilau","id":1218323,"avatar_url":"https://avatars.githubusercontent.com/u/1218323?v=3","gravatar_id":"","url":"https://api.github.com/users/ziqilau","html_url":"https://github.com/ziqilau","followers_url":"https://api.github.com/users/ziqilau/followers","following_url":"https://api.github.com/users/ziqilau/following{/other_user}","gists_url":"https://api.github.com/users/ziqilau/gists{/gist_id}","starred_url":"https://api.github.com/users/ziqilau/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ziqilau/subscriptions","organizations_url":"https://api.github.com/users/ziqilau/orgs","repos_url":"https://api.github.com/users/ziqilau/repos","events_url":"https://api.github.com/users/ziqilau/events{/privacy}","received_events_url":"https://api.github.com/users/ziqilau/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-04-27T18:29:52Z","updated_at":"2015-04-27T20:33:51Z","closed_at":null,"body":"Hi all,\r\n\r\n1) I was using this AMI: http://tleyden.github.io/blog/2014/10/25/cuda-6-dot-5-on-aws-gpu-instance-running-ubuntu-14-dot-04/ for cuda environment on a aws gx2.2xlarge instance.\r\n\r\n2) after download this bidmach bundle: http://bid2.berkeley.edu/bid-data-project/BIDMach_1.0.0-linux-x86_64.tar.gz, I scripts/getdata.sh download the data.\r\n\r\n3) I stuck at nn.train method while training a sample data from movielens10M, see following:\r\n\r\n---------------------------------------------------------------------------------------------------------------------------------------\r\nWelcome to Scala version 2.11.2 (OpenJDK 64-Bit Server VM, Java 1.7.0_79).\r\nType in expressions to have them evaluated.\r\nType :help for more information.\r\n\r\nscala> val a = loadSMat(\"data/movielens/train1.smat.lz4\")\r\na: BIDMat.SMat =\r\n(    172,  14574)      4\r\n(    187,  14574)      4\r\n(    195,  14574)      4\r\n(    207,  14574)      4\r\n(    215,  14574)      4\r\n(    222,  14574)      3\r\n(    224,  14574)      3\r\n(    226,  14574)      3\r\n     ...     ...     ...\r\n\r\nscala> val (nn, opts) = NMF.learner(a)\r\nnn: BIDMach.Learner = Learner(BIDMach.datasources.MatDS@36895c35,BIDMach.models.NMF@7404b78b,null,BIDMach.updaters.IncNorm@61ae422e,BIDMach.models.NMF$xopts$4@777b0c1b)\r\nopts: BIDMach.Learner.Options with BIDMach.models.NMF.Opts with BIDMach.datasources.MatDS.Opts with BIDMach.updaters.IncNorm.Opts = BIDMach.models.NMF$xopts$4@777b0c1b\r\n\r\nscala> nn.train\r\ncorpus perplexity=65134.014613\r\npass= 0\r\ndevice is 0\r\njava.lang.RuntimeException: Cuda error in GSMat() too many resources requested for launch\r\n  at BIDMat.GSMat$.apply(GSMat.scala:325)\r\n  at BIDMat.GSMat$.newOrCheckGSMat(GSMat.scala:480)\r\n  at BIDMat.GSMat$.newOrCheckGSMat(GSMat.scala:528)\r\n  at BIDMat.GSMat$.fromSMat(GSMat.scala:409)\r\n  at BIDMat.GSMat$.apply(GSMat.scala:330)\r\n  at BIDMach.models.Model$$anonfun$copyMats$1.apply$mcVI$sp(Model.scala:106)\r\n  at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)\r\n  at BIDMach.models.Model.copyMats(Model.scala:94)\r\n  at BIDMach.models.Model.doblockg(Model.scala:73)\r\n  at BIDMach.Learner.retrain(Learner.scala:83)\r\n  at BIDMach.Learner.train(Learner.scala:49)\r\n  ... 33 elided\r\n"},"comment":{"url":"https://api.github.com/repos/BIDData/BIDMach/issues/comments/96811122","html_url":"https://github.com/BIDData/BIDMach/issues/44#issuecomment-96811122","issue_url":"https://api.github.com/repos/BIDData/BIDMach/issues/44","id":96811122,"user":{"login":"jcanny","id":1195924,"avatar_url":"https://avatars.githubusercontent.com/u/1195924?v=3","gravatar_id":"","url":"https://api.github.com/users/jcanny","html_url":"https://github.com/jcanny","followers_url":"https://api.github.com/users/jcanny/followers","following_url":"https://api.github.com/users/jcanny/following{/other_user}","gists_url":"https://api.github.com/users/jcanny/gists{/gist_id}","starred_url":"https://api.github.com/users/jcanny/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcanny/subscriptions","organizations_url":"https://api.github.com/users/jcanny/orgs","repos_url":"https://api.github.com/users/jcanny/repos","events_url":"https://api.github.com/users/jcanny/events{/privacy}","received_events_url":"https://api.github.com/users/jcanny/received_events","type":"User","site_admin":false},"created_at":"2015-04-27T20:33:51Z","updated_at":"2015-04-27T20:33:51Z","body":"I got the same error on a different machine. It turns out the movielens dataset has many samples of all zeros - the first 14575 user entries are all zeros. \r\n\r\nBIDMach takes the first minibatch of data (about 2000 samples on this dataset) and uses it to compute the density - i.e. non-zeros per column - for the dataset. That makes the assumption that hte data are stationary, which isnt true in this case.\r\n\r\nThere are a few workarounds - the first is to simply discard the first 14574 columns:\r\n\r\nval aa = a(?,14574->a.ncols)\r\n\r\nand then process aa. A more mathematically kosher way to do it is to randomize the columns so the data really are stationary. \r\n\r\nval aa = a(?,randperm(a.ncols));\r\n\r\nand then train on aa. Strictly speaking you should always randomize the columns of a dataset. In practice it often doesnt matter. Try both ways to be sure. "}}