{"action":"created","issue":{"url":"https://api.github.com/repos/TestAnything/Specification/issues/2","labels_url":"https://api.github.com/repos/TestAnything/Specification/issues/2/labels{/name}","comments_url":"https://api.github.com/repos/TestAnything/Specification/issues/2/comments","events_url":"https://api.github.com/repos/TestAnything/Specification/issues/2/events","html_url":"https://github.com/TestAnything/Specification/issues/2","id":53475039,"number":2,"title":"Standarize the notation of subtests in TAP","user":{"login":"kinow","id":304786,"avatar_url":"https://avatars.githubusercontent.com/u/304786?v=3","gravatar_id":"","url":"https://api.github.com/users/kinow","html_url":"https://github.com/kinow","followers_url":"https://api.github.com/users/kinow/followers","following_url":"https://api.github.com/users/kinow/following{/other_user}","gists_url":"https://api.github.com/users/kinow/gists{/gist_id}","starred_url":"https://api.github.com/users/kinow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kinow/subscriptions","organizations_url":"https://api.github.com/users/kinow/orgs","repos_url":"https://api.github.com/users/kinow/repos","events_url":"https://api.github.com/users/kinow/events{/privacy}","received_events_url":"https://api.github.com/users/kinow/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/TestAnything/Specification/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/TestAnything/Specification/labels/help+wanted","name":"help wanted","color":"159818"},{"url":"https://api.github.com/repos/TestAnything/Specification/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2015-01-06T02:49:09Z","updated_at":"2015-01-20T02:41:13Z","closed_at":null,"body":"This issue was originally reported in https://github.com/TestAnything/testanything.github.io/issues/20, and the conversation has been moved here since it is related to the TAP specification. \r\n\r\nIt concerns the subtests in TAP and a formal standard to be followed by TAP consumer and producers. In the latest TAP specification (TAP 13) there is no formal definition of subtests. There were proposals and draft documents on this, some tools like Test::More and tap4j support with limitations.\r\n\r\nPlease feel free to chime in and suggest enhancements for this standard. A message is being written to the mailing list pointing to this issue too. The mailing list has several users that participated in the IETF draft and have lots of experience with TAP and its implementations.\r\n\r\n/cc @Leont @xmojmr "},"comment":{"url":"https://api.github.com/repos/TestAnything/Specification/issues/comments/70595994","html_url":"https://github.com/TestAnything/Specification/issues/2#issuecomment-70595994","issue_url":"https://api.github.com/repos/TestAnything/Specification/issues/2","id":70595994,"user":{"login":"jonathanKingston","id":338988,"avatar_url":"https://avatars.githubusercontent.com/u/338988?v=3","gravatar_id":"","url":"https://api.github.com/users/jonathanKingston","html_url":"https://github.com/jonathanKingston","followers_url":"https://api.github.com/users/jonathanKingston/followers","following_url":"https://api.github.com/users/jonathanKingston/following{/other_user}","gists_url":"https://api.github.com/users/jonathanKingston/gists{/gist_id}","starred_url":"https://api.github.com/users/jonathanKingston/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonathanKingston/subscriptions","organizations_url":"https://api.github.com/users/jonathanKingston/orgs","repos_url":"https://api.github.com/users/jonathanKingston/repos","events_url":"https://api.github.com/users/jonathanKingston/events{/privacy}","received_events_url":"https://api.github.com/users/jonathanKingston/received_events","type":"User","site_admin":false},"created_at":"2015-01-20T02:41:13Z","updated_at":"2015-01-20T02:41:13Z","body":"I'm sorry but I really am not seeing the adverse reaction to a testing framework that requires you to upgrade your consumer when you decide to want subtests.\r\n\r\nAll ideas have their pitfalls so far. Its a testing framework not a GUI tool or phone system we should not be trying to remedy old frameworks. Testing frameworks need to report errors not be ignored accuracy should be the key.\r\n\r\nIt is not really greenfield at all as some have gone with the suggested wiki post / mailing list. Some have gone with the tap4j backwards format. Some have invented YAML subdocuments and etc. The best we can do is cause errors and explain why by incrementing the version number.\r\n\r\nThe assertions made are that the TAP output is read by one consumer for the one producer and that won't always be the case.\r\n\r\nI'm not precious to any of the suggested structures I have given at all however I am precious to allow the safest failures for older frameworks. You mention that test harness will fail and yes ideally we should look for a structure that has the least issues in the most popular consumers however as I keep saying we should not be worried about causing false positives, false negatives would be far worse.\r\n\r\nIdeally I would prefer an implementation that breaks all old implementations to force the consumer upgrade if subtests are used.\r\n\r\nIf you were upgrading a framework of any kind in a corporate setting, the best is to assume that everything interacting with the framework will break. If we make TAP 14 compelling enough then slowly people will upgrade. Upgrades in all software needs to be compelling enough as likely enough there will be issues.\r\n\r\nWhat I suggest is we keep looking, I really don't think it is ok that the best worst of the indentation is taken up. This causes YAML parsing issues for simplistic parsers.\r\n\r\nI'm back to lets throw as many examples at consumers until we find the most that either:\r\n- can cope with the suggested format\r\n- blow up and cause upgrades\r\n\r\nCan we at least keep looking/trying for a solution here, I think it is pretty obvious why no solution was ever chosen. Slumping for the first suggestion we know seems a little lacking."}}