{"action":"created","issue":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/780","labels_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/780/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/780/comments","events_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/780/events","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/780","id":111390927,"number":780,"title":"RBM/PretrainNetwork parameter flattening incorrect for backprop","user":{"login":"Shima32","id":10724756,"avatar_url":"https://avatars.githubusercontent.com/u/10724756?v=3","gravatar_id":"","url":"https://api.github.com/users/Shima32","html_url":"https://github.com/Shima32","followers_url":"https://api.github.com/users/Shima32/followers","following_url":"https://api.github.com/users/Shima32/following{/other_user}","gists_url":"https://api.github.com/users/Shima32/gists{/gist_id}","starred_url":"https://api.github.com/users/Shima32/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Shima32/subscriptions","organizations_url":"https://api.github.com/users/Shima32/orgs","repos_url":"https://api.github.com/users/Shima32/repos","events_url":"https://api.github.com/users/Shima32/events{/privacy}","received_events_url":"https://api.github.com/users/Shima32/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-10-14T12:48:32Z","updated_at":"2015-11-02T13:31:13Z","closed_at":"2015-11-02T13:31:13Z","body":"Alex Black comment on me problem : I think I know what's going on so RBMs have 3 types of parameters: weights, hidden bias, visible bias during pretraining we use all 3\r\nduring backprop we treat the RBM as a MLP and so just use two of the parameters (weights + hidden bias) however, based on the shapes of the input and the stack trace: it looks like network parameters (incorrectly) includes all 3 types of RBM parameters (instead of 2) whereas gradients (correctly) only include the 2 types\r\n\r\n\r\n\r\nStack Trace:\r\nOct 14, 2015 3:55:33 PM com.github.fommil.jni.JniLoader liberalLoad\r\nINFO: successfully loaded C:\\Users\\shimam\\AppData\\Local\\Temp\\jniloader4238490470366948938netlib-native_ref-win-i686.dll\r\nOct 14, 2015 3:55:33 PM com.github.fommil.jni.JniLoader liberalLoad\r\nINFO: successfully loaded C:\\Users\\shimam\\AppData\\Local\\Temp\\jniloader4751272666911484503netlib-native_system-win-i686.dll\r\no.d.n.m.MultiLayerNetwork - Training on layer 1 with 200 examples\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.l.ScoreIterationListener - Score at iteration 0 is 2.615318140341163\r\no.d.o.l.ScoreIterationListener - Score at iteration 20 is 2.680378213749647\r\no.d.o.l.ScoreIterationListener - Score at iteration 40 is 2.6104382911150457\r\no.d.o.l.ScoreIterationListener - Score at iteration 60 is 2.6004986231133937\r\no.d.o.l.ScoreIterationListener - Score at iteration 80 is 2.445558707113028\r\no.d.n.m.MultiLayerNetwork - Training on layer 2 with 200 examples\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.l.ScoreIterationListener - Score at iteration 0 is 5.407326913958072\r\no.d.o.l.ScoreIterationListener - Score at iteration 20 is 5.2723569441638\r\no.d.o.l.ScoreIterationListener - Score at iteration 40 is 5.372386981579781\r\no.d.o.l.ScoreIterationListener - Score at iteration 60 is 5.427417032880307\r\no.d.o.l.ScoreIterationListener - Score at iteration 80 is 5.1074470856671335\r\no.d.n.m.MultiLayerNetwork - Training on layer 3 with 200 examples\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.l.ScoreIterationListener - Score at iteration 0 is 4.832604580257892\r\no.d.o.l.ScoreIterationListener - Score at iteration 20 is 4.532549481212616\r\no.d.o.l.ScoreIterationListener - Score at iteration 40 is 4.522494414281368\r\no.d.o.l.ScoreIterationListener - Score at iteration 60 is 4.667439318916321\r\no.d.o.l.ScoreIterationListener - Score at iteration 80 is 4.642384226578474\r\no.d.n.m.MultiLayerNetwork - Training on layer 4 with 200 examples\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.l.ScoreIterationListener - Score at iteration 0 is 2.805648804940164\r\no.d.o.l.ScoreIterationListener - Score at iteration 20 is 2.77065380322659\r\no.d.o.l.ScoreIterationListener - Score at iteration 40 is 2.775658802371144\r\no.d.o.l.ScoreIterationListener - Score at iteration 60 is 2.8206674344427585\r\no.d.o.l.ScoreIterationListener - Score at iteration 80 is 2.8356824394673406\r\no.d.n.m.MultiLayerNetwork - Training on layer 5 with 200 examples\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.l.ScoreIterationListener - Score at iteration 0 is 0.46651955706471204\r\no.d.o.l.ScoreIterationListener - Score at iteration 20 is 0.5065245631858706\r\no.d.o.l.ScoreIterationListener - Score at iteration 40 is 0.561529569287777\r\no.d.o.l.ScoreIterationListener - Score at iteration 60 is 0.5615345752512813\r\no.d.o.l.ScoreIterationListener - Score at iteration 80 is 0.4915395813148022\r\no.d.n.m.MultiLayerNetwork - Training on layer 6 with 200 examples\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.l.ScoreIterationListener - Score at iteration 0 is 2.781080294088125\r\no.d.o.l.ScoreIterationListener - Score at iteration 20 is 2.7961003093643186\r\no.d.o.l.ScoreIterationListener - Score at iteration 40 is 2.671120326413393\r\no.d.o.l.ScoreIterationListener - Score at iteration 60 is 2.826148769022703\r\no.d.o.l.ScoreIterationListener - Score at iteration 80 is 2.636188815897465\r\no.d.n.m.MultiLayerNetwork - Training on layer 7 with 200 examples\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.l.ScoreIterationListener - Score at iteration 0 is 4.716687861729622\r\no.d.o.l.ScoreIterationListener - Score at iteration 20 is 4.531727921161175\r\no.d.o.l.ScoreIterationListener - Score at iteration 40 is 4.386767984737873\r\no.d.o.l.ScoreIterationListener - Score at iteration 60 is 4.666808057228089\r\no.d.o.l.ScoreIterationListener - Score at iteration 80 is 4.491848131956577\r\no.d.n.m.MultiLayerNetwork - Finetune phase\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\no.d.o.l.ScoreIterationListener - Score at iteration 0 is 4.012883911132812\r\no.d.o.l.ScoreIterationListener - Score at iteration 20 is 4.00514404296875\r\no.d.o.l.ScoreIterationListener - Score at iteration 40 is 3.997624816894531\r\no.d.o.l.ScoreIterationListener - Score at iteration 60 is 3.9909393310546877\r\no.d.o.l.ScoreIterationListener - Score at iteration 80 is 3.984224548339844\r\no.d.o.s.BaseOptimizer - Objective function automatically set to minimize. Set stepFunction in neural net configuration to change default settings.\r\nException in thread \"main\" java.lang.IllegalArgumentException: Shapes do not match: x.shape=[1, 422], y.shape=[1, 381]\r\n\tat org.nd4j.linalg.api.parallel.tasks.cpu.CPUTaskFactory.getTransformAction(CPUTaskFactory.java:89)\r\n\tat org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner.doTransformOp(DefaultOpExecutioner.java:409)\r\n\tat org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner.exec(DefaultOpExecutioner.java:62)\r\n\tat org.nd4j.linalg.api.ndarray.BaseNDArray.subi(BaseNDArray.java:2699)\r\n\tat org.nd4j.linalg.api.ndarray.BaseNDArray.subi(BaseNDArray.java:2680)\r\n\tat org.deeplearning4j.optimize.stepfunctions.NegativeGradientStepFunction.step(NegativeGradientStepFunction.java:37)\r\n\tat org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(StochasticGradientDescent.java:60)\r\n\tat org.deeplearning4j.optimize.Solver.optimize(Solver.java:52)\r\n\tat org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit(MultiLayerNetwork.java:1352)\r\n\tat org.deeplearning4j.nn.multilayer.MultiLayerNetwork.fit(MultiLayerNetwork.java:1382)\r\n\tat AutomatedTextSummarization.nuralNetwork.CreatingNN.main(CreatingNN.java:93)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)\r\n\r\n\r\n\r\n\r\nNetwork configuration:\r\npublic class CreatingNN {\r\n    private static Logger log = LoggerFactory.getLogger(CreatingNN.class);\r\n\r\n    public static void main(String[] args) throws  Exception {\r\n        RecordReader recordReader = new CSVRecordReader(0,\",\");\r\n        recordReader.initialize(new FileSplit(new ClassPathResource(\"features&Index.txt\").getFile()));\r\n        //reader,label index,number of possible labels\r\n        //with a DataSetIterator, fetching one or more new examples with each iteration,\r\n        // and loading those examples into a DataSet object that neural nets can work with.\r\n        // Calling next on the iterator advances it one step, and returns the new value it points to.\r\n        Nd4j.MAX_SLICES_TO_PRINT = 10;\r\n        Nd4j.MAX_ELEMENTS_PER_SLICE = 10;\r\n\r\n        final int numRows = 4;\r\n        final int numColumns = 1;\r\n        final int numInputs = 4;\r\n        int outputNum = 3;\r\n        int iterations = 100;\r\n        long seed = 123;\r\n        int listenerFreq = iterations/5;\r\n\r\n        log.info(\"Load data....\");\r\n        org.nd4j.linalg.dataset.api.iterator.DataSetIterator iterator = new RecordReaderDataSetIterator(recordReader,200,4,50);\r\n\r\n\r\n        log.info(\"Build model....\");\r\n        MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()\r\n                .seed(seed)\r\n                .iterations(iterations)\r\n                .learningRate(1e-6f)\r\n                .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)\r\n                .l1(1e-1).regularization(true).l2(2e-4)\r\n                .list(8)\r\n                .layer(0, new RBM.Builder().nIn(numRows * numColumns).nOut(10).lossFunction(LossFunctions.LossFunction.RMSE_XENT).build())\r\n                .layer(1, new RBM.Builder().nIn(10).nOut(8).lossFunction(LossFunctions.LossFunction.RMSE_XENT).build())\r\n                .layer(2, new RBM.Builder().nIn(8).nOut(5).lossFunction(LossFunctions.LossFunction.RMSE_XENT).build())\r\n                .layer(3, new RBM.Builder().nIn(5).nOut(1).lossFunction(LossFunctions.LossFunction.RMSE_XENT).build())//encoding stops\r\n                .layer(4, new RBM.Builder().nIn(1).nOut(5).lossFunction(LossFunctions.LossFunction.RMSE_XENT).build())//decoding starts\r\n                .layer(5, new RBM.Builder().nIn(5).nOut(8).lossFunction(LossFunctions.LossFunction.RMSE_XENT).build())\r\n                .layer(6, new RBM.Builder().nIn(8).nOut(10).lossFunction(LossFunctions.LossFunction.RMSE_XENT).build())\r\n                .layer(7, new OutputLayer.Builder(LossFunctions.LossFunction.RMSE_XENT).nIn(10).nOut(numRows * numColumns).build())\r\n                .pretrain(true).backprop(true)\r\n\r\n                .build();\r\n\r\n        MultiLayerNetwork model = new MultiLayerNetwork(conf);\r\n        model.init();\r\n\r\n        model.setListeners(Arrays.asList((IterationListener) new ScoreIterationListener(listenerFreq)));\r\n\r\n        log.info(\"Train model....\");\r\n      //  while(iterator.hasNext()) {\r\n            DataSet next = iterator.next();\r\n            next.normalizeZeroMeanZeroUnitVariance();\r\n            model.fit(new DataSet(next.getFeatureMatrix(), next.getFeatureMatrix()));\r\n      //  }\r\n        INDArray w;\r\n        log.info(\"Evaluate weights....\");\r\n      /*  for(org.deeplearning4j.nn.api.Layer layer : model.getLayers()) {\r\n             w = layer.getParam(DefaultParamInitializer.WEIGHT_KEY);\r\n            log.info(\"Weights: \" + w);\r\n            System.out.println(w);\r\n        }*/\r\n\r\n\r\n      //  INDArray output = model.output(test.getFeatureMatrix());\r\n       // String predicted = output.getRow(i).toString().trim();\r\n\r\n        INDArray layer4=model.getLayer(4).input();\r\n        System.out.println(layer4);\r\n\r\n\r\n    }\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/comments/153016811","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/780#issuecomment-153016811","issue_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/780","id":153016811,"user":{"login":"AlexDBlack","id":2360237,"avatar_url":"https://avatars.githubusercontent.com/u/2360237?v=3","gravatar_id":"","url":"https://api.github.com/users/AlexDBlack","html_url":"https://github.com/AlexDBlack","followers_url":"https://api.github.com/users/AlexDBlack/followers","following_url":"https://api.github.com/users/AlexDBlack/following{/other_user}","gists_url":"https://api.github.com/users/AlexDBlack/gists{/gist_id}","starred_url":"https://api.github.com/users/AlexDBlack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlexDBlack/subscriptions","organizations_url":"https://api.github.com/users/AlexDBlack/orgs","repos_url":"https://api.github.com/users/AlexDBlack/repos","events_url":"https://api.github.com/users/AlexDBlack/events{/privacy}","received_events_url":"https://api.github.com/users/AlexDBlack/received_events","type":"User","site_admin":false},"created_at":"2015-11-02T13:31:13Z","updated_at":"2015-11-02T13:31:13Z","body":"Fixed.\r\nhttps://github.com/deeplearning4j/deeplearning4j/pull/833"}}