{"action":"created","issue":{"url":"https://api.github.com/repos/moses-smt/mosesdecoder/issues/127","labels_url":"https://api.github.com/repos/moses-smt/mosesdecoder/issues/127/labels{/name}","comments_url":"https://api.github.com/repos/moses-smt/mosesdecoder/issues/127/comments","events_url":"https://api.github.com/repos/moses-smt/mosesdecoder/issues/127/events","html_url":"https://github.com/moses-smt/mosesdecoder/pull/127","id":107600382,"number":127,"title":"Make cube pruning deterministic","user":{"login":"mjdenkowski","id":636653,"avatar_url":"https://avatars.githubusercontent.com/u/636653?v=3","gravatar_id":"","url":"https://api.github.com/users/mjdenkowski","html_url":"https://github.com/mjdenkowski","followers_url":"https://api.github.com/users/mjdenkowski/followers","following_url":"https://api.github.com/users/mjdenkowski/following{/other_user}","gists_url":"https://api.github.com/users/mjdenkowski/gists{/gist_id}","starred_url":"https://api.github.com/users/mjdenkowski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mjdenkowski/subscriptions","organizations_url":"https://api.github.com/users/mjdenkowski/orgs","repos_url":"https://api.github.com/users/mjdenkowski/repos","events_url":"https://api.github.com/users/mjdenkowski/events{/privacy}","received_events_url":"https://api.github.com/users/mjdenkowski/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-09-21T21:17:59Z","updated_at":"2015-09-23T14:19:15Z","closed_at":"2015-09-23T09:50:53Z","pull_request":{"url":"https://api.github.com/repos/moses-smt/mosesdecoder/pulls/127","html_url":"https://github.com/moses-smt/mosesdecoder/pull/127","diff_url":"https://github.com/moses-smt/mosesdecoder/pull/127.diff","patch_url":"https://github.com/moses-smt/mosesdecoder/pull/127.patch"},"body":"This is again a pull request so other people can take a look.  There is an edge case with cube pruning where sort order is undefined for hypotheses with different translations but equal overall model scores.  This means that running the same test set multiple times could produce different outputs (more of a problem with smaller data where phrase pairs are more likely to have the same scores).\r\n\r\nThe simple fix for this was to break ties by comparing the current target phrases in each hypothesis, but this led to some concurrency problems.  Hypotheses keep pointers to target phrases, and these pointers kept getting deleted by other threads in the middle of comparisons, leading to segfaults.  I got around this by making a copy of the target phrase in each hypothesis, which incurs a minimal (1-2%) slowdown in decoding.  This is a good trade for my use case, but if other people feel strongly about it, deterministic tie breaking could be factored out as a config option.  I briefly looked at fixing the concurrency problem, but that turns into a pretty deep rabbit hole with lots of raw pointer management across many classes.  Refactoring that code would be a pretty significant undertaking.\r\n\r\nI can merge this in if there are no objections, but I wanted to make sure no one would be negatively affected."},"comment":{"url":"https://api.github.com/repos/moses-smt/mosesdecoder/issues/comments/142616084","html_url":"https://github.com/moses-smt/mosesdecoder/pull/127#issuecomment-142616084","issue_url":"https://api.github.com/repos/moses-smt/mosesdecoder/issues/127","id":142616084,"user":{"login":"hieuhoang","id":691732,"avatar_url":"https://avatars.githubusercontent.com/u/691732?v=3","gravatar_id":"","url":"https://api.github.com/users/hieuhoang","html_url":"https://github.com/hieuhoang","followers_url":"https://api.github.com/users/hieuhoang/followers","following_url":"https://api.github.com/users/hieuhoang/following{/other_user}","gists_url":"https://api.github.com/users/hieuhoang/gists{/gist_id}","starred_url":"https://api.github.com/users/hieuhoang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hieuhoang/subscriptions","organizations_url":"https://api.github.com/users/hieuhoang/orgs","repos_url":"https://api.github.com/users/hieuhoang/repos","events_url":"https://api.github.com/users/hieuhoang/events{/privacy}","received_events_url":"https://api.github.com/users/hieuhoang/received_events","type":"User","site_admin":false},"created_at":"2015-09-23T14:19:15Z","updated_at":"2015-09-23T14:19:15Z","body":"oh, internet might have crapped out. \r\n\r\nya, I have doubts. Perhaps make TargetPhrase m_target_phrase and pointer which only gets used when a command line arg has been set.\r\n\r\nWhat's the use case for this btw? In general the decoding isn't guaranteed to be deterministic\r\n   https://www.mail-archive.com/moses-support%40mit.edu/msg03726.html"}}