{"action":"created","issue":{"url":"https://api.github.com/repos/drphilmarshall/SpaceWarps/issues/73","labels_url":"https://api.github.com/repos/drphilmarshall/SpaceWarps/issues/73/labels{/name}","comments_url":"https://api.github.com/repos/drphilmarshall/SpaceWarps/issues/73/comments","events_url":"https://api.github.com/repos/drphilmarshall/SpaceWarps/issues/73/events","html_url":"https://github.com/drphilmarshall/SpaceWarps/issues/73","id":40957359,"number":73,"title":"Emulate dynamic stage 2 by re-running stage 2 with just the high stage 1 skill users","user":{"login":"drphilmarshall","id":710903,"avatar_url":"https://avatars.githubusercontent.com/u/710903?v=3","gravatar_id":"","url":"https://api.github.com/users/drphilmarshall","html_url":"https://github.com/drphilmarshall","followers_url":"https://api.github.com/users/drphilmarshall/followers","following_url":"https://api.github.com/users/drphilmarshall/following{/other_user}","gists_url":"https://api.github.com/users/drphilmarshall/gists{/gist_id}","starred_url":"https://api.github.com/users/drphilmarshall/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drphilmarshall/subscriptions","organizations_url":"https://api.github.com/users/drphilmarshall/orgs","repos_url":"https://api.github.com/users/drphilmarshall/repos","events_url":"https://api.github.com/users/drphilmarshall/events{/privacy}","received_events_url":"https://api.github.com/users/drphilmarshall/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/drphilmarshall/SpaceWarps/labels/Analysis","name":"Analysis","color":"eb6420"},{"url":"https://api.github.com/repos/drphilmarshall/SpaceWarps/labels/Question","name":"Question","color":"5319e7"}],"state":"open","locked":false,"assignee":{"login":"cpadavis","id":1781988,"avatar_url":"https://avatars.githubusercontent.com/u/1781988?v=3","gravatar_id":"","url":"https://api.github.com/users/cpadavis","html_url":"https://github.com/cpadavis","followers_url":"https://api.github.com/users/cpadavis/followers","following_url":"https://api.github.com/users/cpadavis/following{/other_user}","gists_url":"https://api.github.com/users/cpadavis/gists{/gist_id}","starred_url":"https://api.github.com/users/cpadavis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cpadavis/subscriptions","organizations_url":"https://api.github.com/users/cpadavis/orgs","repos_url":"https://api.github.com/users/cpadavis/repos","events_url":"https://api.github.com/users/cpadavis/events{/privacy}","received_events_url":"https://api.github.com/users/cpadavis/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/drphilmarshall/SpaceWarps/milestones/4","html_url":"https://github.com/drphilmarshall/SpaceWarps/milestones/System%20and%20CFHTLS%20results%20paper%20drafts","labels_url":"https://api.github.com/repos/drphilmarshall/SpaceWarps/milestones/4/labels","id":565796,"number":4,"title":"System and CFHTLS results paper drafts","description":"Can we circulate to co-authors for comments on Tuesday Feb 10? AAAS starts the next day!","creator":{"login":"drphilmarshall","id":710903,"avatar_url":"https://avatars.githubusercontent.com/u/710903?v=3","gravatar_id":"","url":"https://api.github.com/users/drphilmarshall","html_url":"https://github.com/drphilmarshall","followers_url":"https://api.github.com/users/drphilmarshall/followers","following_url":"https://api.github.com/users/drphilmarshall/following{/other_user}","gists_url":"https://api.github.com/users/drphilmarshall/gists{/gist_id}","starred_url":"https://api.github.com/users/drphilmarshall/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drphilmarshall/subscriptions","organizations_url":"https://api.github.com/users/drphilmarshall/orgs","repos_url":"https://api.github.com/users/drphilmarshall/repos","events_url":"https://api.github.com/users/drphilmarshall/events{/privacy}","received_events_url":"https://api.github.com/users/drphilmarshall/received_events","type":"User","site_admin":false},"open_issues":17,"closed_issues":42,"state":"open","created_at":"2014-02-11T07:56:27Z","updated_at":"2015-01-28T23:31:46Z","due_on":"2015-02-10T00:00:00Z","closed_at":null},"comments":7,"created_at":"2014-08-22T22:42:54Z","updated_at":"2015-02-27T16:57:25Z","closed_at":null,"body":"If we had only asked the high skill users to take part in stage 2, we might have made a cut in stage 1 skill at say 0.2, and then only showed them the refinement app. We can emulate this by ignoring all agents with names not on this list. What happens to the completeness/purity? Do we have fewer false negatives? And do we see any lensed quasars appearing? This is worth a try I think: it's one way to address the false negatives problem (issue #68), which I think becomes more pressing since we discovered that the stage 2 agents don't know anything about lensed quasars... What do you think? We'd have to read the CFHTLS stage 1 agent names first, then propagate (watch out for the VICS82 stage 1 agents, thats a different bureau!)"},"comment":{"url":"https://api.github.com/repos/drphilmarshall/SpaceWarps/issues/comments/76428481","html_url":"https://github.com/drphilmarshall/SpaceWarps/issues/73#issuecomment-76428481","issue_url":"https://api.github.com/repos/drphilmarshall/SpaceWarps/issues/73","id":76428481,"user":{"login":"cpadavis","id":1781988,"avatar_url":"https://avatars.githubusercontent.com/u/1781988?v=3","gravatar_id":"","url":"https://api.github.com/users/cpadavis","html_url":"https://github.com/cpadavis","followers_url":"https://api.github.com/users/cpadavis/followers","following_url":"https://api.github.com/users/cpadavis/following{/other_user}","gists_url":"https://api.github.com/users/cpadavis/gists{/gist_id}","starred_url":"https://api.github.com/users/cpadavis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cpadavis/subscriptions","organizations_url":"https://api.github.com/users/cpadavis/orgs","repos_url":"https://api.github.com/users/cpadavis/repos","events_url":"https://api.github.com/users/cpadavis/events{/privacy}","received_events_url":"https://api.github.com/users/cpadavis/received_events","type":"User","site_admin":false},"created_at":"2015-02-27T16:57:25Z","updated_at":"2015-02-27T16:57:25Z","body":"I would say that the interpretation in the figure caption ('may reduce noise in the system') is outdated and we should be able to definitively say yes or no. (Though now I'm not sure how to precisely probe 'reduce noise'.)\r\n\r\nI was worried maybe I was cutting at too low a skill threshold, so I remade the roc curve cut at skill = 0.25 instead (that is where the vertical dashed line is in figure 13):\r\n\r\n![roc](https://cloud.githubusercontent.com/assets/1781988/6416853/1a89860a-be5e-11e4-8e41-af943e36dbe8.png)\r\n\r\nThis time we have 326 users and we lost something like 70 subjects. They still do marginally worse, but only marginally so.\r\n\r\nFigure 13 shows (just like the ROC curve does here) that you get nearly the same roc curve using just the stage 1 veterans (since high stage 1 skill AND high stage 2 skill together imply that you also made a large number of classifications). Users who had high stage 1 skill but low stage 2 skill generally made few classifications in stage 2 and so didn't contribute much to stage 2. The statement that you could reduce the noise in the system comes from the combination of these facts.\r\n\r\nI think my statement that high stage 1 skill does not necessarily imply high stage 2 skill is borne out in figure 13. Even though our eye is drawn to the bigger circles, there are _many_ small circles below the stage 2 skill = 0.1 dashed line. However our central intention with the figure (that cutting on high stage 1 skill does not throw out that much information) is emphasized by the circle size, and the roc curve supports it as well.\r\n\r\n(I think we should be _impressed_ and not _distraught_ that the top quarter of the spacewarps population (for some definition of 'top quarter') is able to do 99% of the work on their own.)"}}