{"action":"created","issue":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/318","labels_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/318/labels{/name}","comments_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/318/comments","events_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/318/events","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/318","id":85545628,"number":318,"title":"Large Word2Vec model deserialization fails with both jblas and jcublas","user":{"login":"danielchalef","id":131175,"avatar_url":"https://avatars.githubusercontent.com/u/131175?v=3","gravatar_id":"","url":"https://api.github.com/users/danielchalef","html_url":"https://github.com/danielchalef","followers_url":"https://api.github.com/users/danielchalef/followers","following_url":"https://api.github.com/users/danielchalef/following{/other_user}","gists_url":"https://api.github.com/users/danielchalef/gists{/gist_id}","starred_url":"https://api.github.com/users/danielchalef/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danielchalef/subscriptions","organizations_url":"https://api.github.com/users/danielchalef/orgs","repos_url":"https://api.github.com/users/danielchalef/repos","events_url":"https://api.github.com/users/danielchalef/events{/privacy}","received_events_url":"https://api.github.com/users/danielchalef/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-06-05T14:25:59Z","updated_at":"2015-06-14T21:41:27Z","closed_at":null,"body":"Loading a 22G Google word2vec text model (or the equivalent 9G binary model) fails when using both jblas and jcublas backends. The model can be successfully loaded by Google's `distance` and `word-analogy` tools.\r\n\r\nCode snippet:\r\n```\r\n    var file = new File(vecFile)\r\n    var vec = WordVectorSerializer.loadGoogleModel(file, false)\r\n```\r\nJcublas stacktrace:\r\n```\r\nException in thread \"main\" java.lang.IllegalArgumentException: Illegal indices [0, 0]\r\n\tat org.nd4j.linalg.api.ndarray.BaseNDArray.putScalar(BaseNDArray.java:896)\r\n\tat org.nd4j.linalg.api.ndarray.BaseNDArray.put(BaseNDArray.java:1360)\r\n\tat org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.readTextModel(WordVectorSerializer.java:112)\r\n\tat org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.loadGoogleModel(WordVectorSerializer.java:77)\r\n\tat com.kt.nlp.WikiWord2Vec$.queryVec(WikiWord2Vec.scala:36)\r\n\tat com.kt.nlp.WikiWord2Vec$.main(WikiWord2Vec.scala:31)\r\n\tat com.kt.nlp.WikiWord2Vec.main(WikiWord2Vec.scala)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)\r\n```\r\n\r\nJblas stacktrace:\r\n```\r\nException in thread \"main\" java.lang.NegativeArraySizeException\r\n\tat org.nd4j.linalg.api.buffer.BaseDataBuffer.<init>(BaseDataBuffer.java:189)\r\n\tat org.nd4j.linalg.api.buffer.FloatBuffer.<init>(FloatBuffer.java:39)\r\n\tat org.nd4j.linalg.api.buffer.factory.DefaultDataBufferFactory.createFloat(DefaultDataBufferFactory.java:42)\r\n\tat org.nd4j.linalg.factory.Nd4j.createBuffer(Nd4j.java:566)\r\n\tat org.nd4j.linalg.api.ndarray.BaseNDArray.<init>(BaseNDArray.java:183)\r\n\tat org.nd4j.linalg.jblas.NDArray.<init>(NDArray.java:103)\r\n\tat org.nd4j.linalg.jblas.JblasNDArrayFactory.create(JblasNDArrayFactory.java:201)\r\n\tat org.nd4j.linalg.factory.BaseNDArrayFactory.create(BaseNDArrayFactory.java:193)\r\n\tat org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:3109)\r\n\tat org.nd4j.linalg.factory.Nd4j.create(Nd4j.java:2500)\r\n\tat org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.readTextModel(WordVectorSerializer.java:98)\r\n\tat org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.loadGoogleModel(WordVectorSerializer.java:77)\r\n\tat com.kt.nlp.WikiWord2Vec$.queryVec(WikiWord2Vec.scala:36)\r\n\tat com.kt.nlp.WikiWord2Vec$.main(WikiWord2Vec.scala:31)\r\n\tat com.kt.nlp.WikiWord2Vec.main(WikiWord2Vec.scala)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:606)\r\n\tat com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)\r\n-- org.jblas INFO Deleting /tmp/jblas6657690143696141405/libjblas.so\r\n-- org.jblas INFO Deleting /tmp/jblas6657690143696141405/libjblas_arch_flavor.so\r\n-- org.jblas INFO Deleting /tmp/jblas6657690143696141405\r\n```"},"comment":{"url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/comments/111878051","html_url":"https://github.com/deeplearning4j/deeplearning4j/issues/318#issuecomment-111878051","issue_url":"https://api.github.com/repos/deeplearning4j/deeplearning4j/issues/318","id":111878051,"user":{"login":"agibsonccc","id":1144306,"avatar_url":"https://avatars.githubusercontent.com/u/1144306?v=3","gravatar_id":"","url":"https://api.github.com/users/agibsonccc","html_url":"https://github.com/agibsonccc","followers_url":"https://api.github.com/users/agibsonccc/followers","following_url":"https://api.github.com/users/agibsonccc/following{/other_user}","gists_url":"https://api.github.com/users/agibsonccc/gists{/gist_id}","starred_url":"https://api.github.com/users/agibsonccc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/agibsonccc/subscriptions","organizations_url":"https://api.github.com/users/agibsonccc/orgs","repos_url":"https://api.github.com/users/agibsonccc/repos","events_url":"https://api.github.com/users/agibsonccc/events{/privacy}","received_events_url":"https://api.github.com/users/agibsonccc/received_events","type":"User","site_admin":false},"created_at":"2015-06-14T21:41:27Z","updated_at":"2015-06-14T21:41:27Z","body":"Found out that the google model ingestion was also borked. (It was reading the floats wrong from the input stream) \r\n\r\nI'd be curious to see the results now."}}