{"push_id":752325491,"size":2,"distinct_size":2,"ref":"refs/heads/rnnlm","head":"591d56c65095469159f1b75c57809a6db432b9bb","before":"e5e733b427e0c043da0c38143b1b2ccdc18cc391","commits":[{"sha":"a92a1c7786c2f65344f8f3ff1cfc4aa545724b09","author":{"email":"ff0112eee266f577d811344bc77668fd63060693@comp.nus.edu.sg","name":"Wei Wang"},"message":"SINGA-42 Issue when loading checkpoints\n\nUpdate default value for ModelProto::reset_param_version to true.\nIt will reset all parameter version to ModelProto::step.\n\nFor resuming training from checkpoints, if users do not set it, the Trainer::Resume() function\nwill set it to false. Hence the parameter versions will continue from last checkpoint.\nIf users set it to true, then all parameters will be reset to the version as ModelProto::step.\n\nIf using the checkpoint as pre-training to initalize new model parameters,\nusers better use the default value (i.e., true),\notherwise some parameters' version would be much larger than others.\n\nfixbug from calling Worker::Put() in Worker::InitLocalParam().\n\nPreviously, the version/step passed to Put() is step_ which starts from ModelProto::step.\nHence, the params in the servers are put with version step_.\nWhen the params are load from checkpoint files and their param versions are not reset,\nthen the trainining may get stuck after one iteration.\nBecause if the start step is small (usually 0), parame version at the server\nside is small (i.e., 0), while the local_version() (assigned from version()) is the one from last checkpoint,\nwhich is large. Hence the Worker::Collect() function will get stuck.\nTo fix this bug, just pass the current param->version() to Worker::Put().\n\nremove hard code check for label value in layer.cc\n\nFixbug from setting checkpoint file path in Worker::Resume().\nNow the Worker::Resume() will clear checkpoint field in JobProto and add\nthe checkpoint files it finds under WORKSPACE/checkpoint/.","distinct":true,"url":"https://api.github.com/repos/kaiping/incubator-singa/commits/a92a1c7786c2f65344f8f3ff1cfc4aa545724b09"},{"sha":"591d56c65095469159f1b75c57809a6db432b9bb","author":{"email":"3799a27316448c5a714c6c137fddb7edca0f3845@users.noreply.github.com","name":"kaiping"},"message":"Merge pull request #7 from apache/master\n\nRebase from current incubator-singa master","distinct":true,"url":"https://api.github.com/repos/kaiping/incubator-singa/commits/591d56c65095469159f1b75c57809a6db432b9bb"}]}