{"action":"created","issue":{"url":"https://api.github.com/repos/ga4gh/server/issues/430","labels_url":"https://api.github.com/repos/ga4gh/server/issues/430/labels{/name}","comments_url":"https://api.github.com/repos/ga4gh/server/issues/430/comments","events_url":"https://api.github.com/repos/ga4gh/server/issues/430/events","html_url":"https://github.com/ga4gh/server/issues/430","id":83618989,"number":430,"title":"Need a checklist describing what is current working in the server","user":{"login":"diekhans","id":110053,"avatar_url":"https://avatars.githubusercontent.com/u/110053?v=3","gravatar_id":"","url":"https://api.github.com/users/diekhans","html_url":"https://github.com/diekhans","followers_url":"https://api.github.com/users/diekhans/followers","following_url":"https://api.github.com/users/diekhans/following{/other_user}","gists_url":"https://api.github.com/users/diekhans/gists{/gist_id}","starred_url":"https://api.github.com/users/diekhans/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/diekhans/subscriptions","organizations_url":"https://api.github.com/users/diekhans/orgs","repos_url":"https://api.github.com/users/diekhans/repos","events_url":"https://api.github.com/users/diekhans/events{/privacy}","received_events_url":"https://api.github.com/users/diekhans/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":20,"created_at":"2015-06-01T17:43:15Z","updated_at":"2015-06-01T20:22:48Z","closed_at":null,"body":"It's difficult to understand exactly what is implemented in the server.  What protocols, what schemas, fields in the schema, and methods.  A maintained checklist style document in the tree would be most helpful"},"comment":{"url":"https://api.github.com/repos/ga4gh/server/issues/comments/107695673","html_url":"https://github.com/ga4gh/server/issues/430#issuecomment-107695673","issue_url":"https://api.github.com/repos/ga4gh/server/issues/430","id":107695673,"user":{"login":"diekhans","id":110053,"avatar_url":"https://avatars.githubusercontent.com/u/110053?v=3","gravatar_id":"","url":"https://api.github.com/users/diekhans","html_url":"https://github.com/diekhans","followers_url":"https://api.github.com/users/diekhans/followers","following_url":"https://api.github.com/users/diekhans/following{/other_user}","gists_url":"https://api.github.com/users/diekhans/gists{/gist_id}","starred_url":"https://api.github.com/users/diekhans/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/diekhans/subscriptions","organizations_url":"https://api.github.com/users/diekhans/orgs","repos_url":"https://api.github.com/users/diekhans/repos","events_url":"https://api.github.com/users/diekhans/events{/privacy}","received_events_url":"https://api.github.com/users/diekhans/received_events","type":"User","site_admin":false},"created_at":"2015-06-01T20:22:48Z","updated_at":"2015-06-01T20:22:48Z","body":"\n501 == not implemented ;-)\n\nbut at this point, I would be happy to have\nsomething resembling a coherent API and servers\nthat produce gigabytes of failure message.\n\nSkipped tests should still be failures, but right now any kind\nof optimization is low priority.\n\n\n\nWayne Stidolph <notifications@github.com> writes:\n> As to how it's different than a 501 - an unexpected 5xx certainly says\n> \"server error\" but that doesn't tell the CTK what is wrong, what tests\n> might be skippable as a result. So absent some form of tagging and\n> advice-from-the-server, the CTK will go on to run the next test, and\n> generate the next failure, and the next block of log entries.\n> \n> But I agree there's a risk of inappropriate test-skipping , and that a\n> human can scrounge that information from whatever sources and build the\n> test-config file they want, and that \"YAGNI\" suggests to not worry about it\n> until people complain about the test failure volume.\n> \n> I look forward to the collected-capabilities checklist! (I need it for the\n> CTK already)\n> \n> On Mon, Jun 1, 2015 at 12:43 PM, Mark Diekhans <notifications@github.com>\n> wrote:\n> \n> >\n> > how is this different than a 501?\n> >\n> > I can see how it is useful, but I am not sure it's very high\n> > priority.\n> >\n> > Wayne Stidolph <notifications@github.com> writes:\n> > > The test suite's reason for having this sort of thing be machine-friendly\n> > > live-in-the-server is that since these are inter-system tests, the test\n> > > will fail *slowly* and an incomplete implementation in a function will\n> > > block all the corner-case tests which involve that function. Yes, those\n> > are\n> > > real failures and we will announce that loudly - but if we have a lot of\n> > > tests (and we should) then a known-incomplete condition of the server\n> > under\n> > > test will cause a bunch of essentially the same error reports, and people\n> > > become less willing to use the test suite (since it's full of repetition,\n> > > and it runs slow). The tests become *less* useful in driving development\n> > > when there's multiple reports driven by the same fault.\n> > >\n> > > If we can tag/Categorize the tests and skip-fail the ones that are\n> > disabled\n> > > using some notification from the server then we can announce the same\n> > test\n> > > results more quickly, and not have to evaluate every failure as \"oh,\n> > that's\n> > > another symptom of misordering of the X and Y fields\"\n> > >\n> > >\n> > > On Mon, Jun 1, 2015 at 12:22 PM, Danny Colligan <\n> > notifications@github.com>\n> > > wrote:\n> > >\n> > > > @diekhans <https://github.com/diekhans> how is the checklist going to\n> > be\n> > > > any different from a less information rich version of the issue\n> > tracker? If\n> > > > something is unimplemented that we want, an issue should be filed. If\n> > more\n> > > > information about server operation is needed, that can go into the\n> > > > documentation.\n> > > >\n> > > > —\n> > > > Reply to this email directly or view it on GitHub\n> > > > <https://github.com/ga4gh/server/issues/430#issuecomment-107677050>.\n> > > >\n> > >\n> > >\n> > >\n> > > --\n> > > Wayne Stidolph | (831) 419 7146 | LinkedIn\n> > > <http://www.linkedin.com/in/waynestidolph> | Google+\n> > > <https://google.com/+WayneStidolph> | twitter\n> > > <https://twitter.com/#!/wstidolph> | personal blog\n> > > <http://waynedge.blogspot.com/>\n> > >\n> > > —\n> > > Reply to this email directly or view it on GitHub.*\n> >\n> > >\n> >\n> > —\n> > Reply to this email directly or view it on GitHub\n> > <https://github.com/ga4gh/server/issues/430#issuecomment-107681485>.\n> >\n> \n> \n> \n> --\n> Wayne Stidolph | (831) 419 7146 | LinkedIn\n> <http://www.linkedin.com/in/waynestidolph> | Google+\n> <https://google.com/+WayneStidolph> | twitter\n> <https://twitter.com/#!/wstidolph> | personal blog\n> <http://waynedge.blogspot.com/>\n> \n> —\n> Reply to this email directly or view it on GitHub.*\n> \n"}}