{"action":"created","issue":{"url":"https://api.github.com/repos/pluskid/Mocha.jl/issues/134","labels_url":"https://api.github.com/repos/pluskid/Mocha.jl/issues/134/labels{/name}","comments_url":"https://api.github.com/repos/pluskid/Mocha.jl/issues/134/comments","events_url":"https://api.github.com/repos/pluskid/Mocha.jl/issues/134/events","html_url":"https://github.com/pluskid/Mocha.jl/issues/134","id":108369413,"number":134,"title":"How to check accuracy on binary classification problems with {-1, +1} labels?","user":{"login":"nstiurca","id":1597203,"avatar_url":"https://avatars.githubusercontent.com/u/1597203?v=3","gravatar_id":"","url":"https://api.github.com/users/nstiurca","html_url":"https://github.com/nstiurca","followers_url":"https://api.github.com/users/nstiurca/followers","following_url":"https://api.github.com/users/nstiurca/following{/other_user}","gists_url":"https://api.github.com/users/nstiurca/gists{/gist_id}","starred_url":"https://api.github.com/users/nstiurca/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nstiurca/subscriptions","organizations_url":"https://api.github.com/users/nstiurca/orgs","repos_url":"https://api.github.com/users/nstiurca/repos","events_url":"https://api.github.com/users/nstiurca/events{/privacy}","received_events_url":"https://api.github.com/users/nstiurca/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-09-25T17:12:29Z","updated_at":"2015-09-25T18:20:27Z","closed_at":null,"body":"This \"issue\" isn't necessarily a bug or feature request, but rather I'm abusing the issue tracking in lieu of a proper discussion thread since Github doesn't seem to have anything of the sort.\r\n\r\nI submitted the HingeLossLayer earlier this week, and I need to check the accuracy as I train. The simplest thing would be if I could just use the existing AccuracyLayer, but I'm not sure what I'm supposed to give it as inputs. My labels are from the set {-1, +1}, and computing the hinge loss is a bit different from computing final prediction and correct/incorrect classification. For example, if my output layer outputs +0.25, and the label is +1, then the hinge loss is 0.75, but accuracy-wise this is a correct prediction because the sign is correct.\r\n\r\nTo get proper accuracy statistics for my use case, I can think of three possibilities, and I would like some feedback on how to proceed.\r\n\r\n 1. Write a new layer, say BinaryAccuracyLayer which expects labels from {-1, +1}, and checks only the signs of the prediction.\r\n 2. Modify the HingeLossLayer to accumulate accuracy statistics on top of the loss statistics it already accumulates. This might slow down the training a bit, but probably very little compared to how much work is done in ConvolutionLayers and InnerProductLayers.\r\n 3. Somehow leverage the existing AccuracyLayer, maybe by somehow transforming the values of the prediction layer and the labels. I'm not sure what this would look like, however."},"comment":{"url":"https://api.github.com/repos/pluskid/Mocha.jl/issues/comments/143313314","html_url":"https://github.com/pluskid/Mocha.jl/issues/134#issuecomment-143313314","issue_url":"https://api.github.com/repos/pluskid/Mocha.jl/issues/134","id":143313314,"user":{"login":"nstiurca","id":1597203,"avatar_url":"https://avatars.githubusercontent.com/u/1597203?v=3","gravatar_id":"","url":"https://api.github.com/users/nstiurca","html_url":"https://github.com/nstiurca","followers_url":"https://api.github.com/users/nstiurca/followers","following_url":"https://api.github.com/users/nstiurca/following{/other_user}","gists_url":"https://api.github.com/users/nstiurca/gists{/gist_id}","starred_url":"https://api.github.com/users/nstiurca/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nstiurca/subscriptions","organizations_url":"https://api.github.com/users/nstiurca/orgs","repos_url":"https://api.github.com/users/nstiurca/repos","events_url":"https://api.github.com/users/nstiurca/events{/privacy}","received_events_url":"https://api.github.com/users/nstiurca/received_events","type":"User","site_admin":false},"created_at":"2015-09-25T18:20:27Z","updated_at":"2015-09-25T18:20:27Z","body":"By the way, for multi-class labels, are the labels expected to be {0, 1, ..., N-1}, or {1, 2, ..., N}?"}}