{"action":"created","issue":{"url":"https://api.github.com/repos/dotnet/corefx/issues/1477","labels_url":"https://api.github.com/repos/dotnet/corefx/issues/1477/labels{/name}","comments_url":"https://api.github.com/repos/dotnet/corefx/issues/1477/comments","events_url":"https://api.github.com/repos/dotnet/corefx/issues/1477/events","html_url":"https://github.com/dotnet/corefx/issues/1477","id":70288005,"number":1477,"title":"Test infrastructure requirements and scenarios","user":{"login":"Priya91","id":7853870,"avatar_url":"https://avatars.githubusercontent.com/u/7853870?v=3","gravatar_id":"","url":"https://api.github.com/users/Priya91","html_url":"https://github.com/Priya91","followers_url":"https://api.github.com/users/Priya91/followers","following_url":"https://api.github.com/users/Priya91/following{/other_user}","gists_url":"https://api.github.com/users/Priya91/gists{/gist_id}","starred_url":"https://api.github.com/users/Priya91/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Priya91/subscriptions","organizations_url":"https://api.github.com/users/Priya91/orgs","repos_url":"https://api.github.com/users/Priya91/repos","events_url":"https://api.github.com/users/Priya91/events{/privacy}","received_events_url":"https://api.github.com/users/Priya91/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-04-23T02:44:30Z","updated_at":"2015-05-11T16:28:08Z","closed_at":null,"body":"cc @eatdrinksleepcode @weshaggard @krwq \r\n\r\n**Scenarios**\r\n\r\n1) As a developer I want to be able to run a set of tests that run quickly and and give a basic quality read of the library or libraries that I'm building from the command line.\r\n2) As a developer I want to be able to run the complete set of tests for a library or libraries from the command line.\r\n3) As a developer I want to be able to run a set of tests for a given library within Visual Studio and be able to debug them.\r\n4) As a developer I want to have control over how I filter out a set of tests I run based on different criteria, such as non-failing, platform specific, test category (i.e. innerloop, outerloop, perf),\r\n5) As a developer I want to be able to run my tests with code coverage enabled to help determine the amount of code my tests are exercising.\r\n\r\n**Requirements**\r\n\r\n1) Can run all passing InnerLoop tests\r\n2) Can run failing tests \r\n3) Can run all tests applicable to a particular platform\r\n4) Can run only the tests that are applicable to by filtering criteria \r\n5) Can enable code coverage\r\n\r\n**Definitions**\r\n\r\n1) InnerLoop: Set of tests to be run for basic quality check that are not long running.\r\n2) OuterLoop: Set of tests that are long running and do extensive integration and performance testing \r\n"},"comment":{"url":"https://api.github.com/repos/dotnet/corefx/issues/comments/100971676","html_url":"https://github.com/dotnet/corefx/issues/1477#issuecomment-100971676","issue_url":"https://api.github.com/repos/dotnet/corefx/issues/1477","id":100971676,"user":{"login":"iamjasonp","id":1616847,"avatar_url":"https://avatars.githubusercontent.com/u/1616847?v=3","gravatar_id":"","url":"https://api.github.com/users/iamjasonp","html_url":"https://github.com/iamjasonp","followers_url":"https://api.github.com/users/iamjasonp/followers","following_url":"https://api.github.com/users/iamjasonp/following{/other_user}","gists_url":"https://api.github.com/users/iamjasonp/gists{/gist_id}","starred_url":"https://api.github.com/users/iamjasonp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iamjasonp/subscriptions","organizations_url":"https://api.github.com/users/iamjasonp/orgs","repos_url":"https://api.github.com/users/iamjasonp/repos","events_url":"https://api.github.com/users/iamjasonp/events{/privacy}","received_events_url":"https://api.github.com/users/iamjasonp/received_events","type":"User","site_admin":false},"created_at":"2015-05-11T16:28:08Z","updated_at":"2015-05-11T16:28:08Z","body":"Is the intention to be have a master list of categories/additional requirements under which we should classify tests? I see the following categories of test having already been suggested/implemented: \r\n\r\n* InnerLoop\r\n* OuterLoop\r\n* Stress\r\n* Performance\r\n* OuterLoopWithAdditionalRequirementsAndMightModifyYourMachine \r\n\r\n(the last one's name is a little convoluted, obviously) \r\n\r\nIt might be worth considering an additional scenario, perhaps: \r\n\r\n* As a developer, I want to be able to specify any special dependencies and requirements for my test to run\r\n\r\nwhere filtering is covered by:\r\n> 4) As a developer I want to have control over how I filter out a set of tests I run based on different criteria, such as non-failing, platform specific, test category (i.e. innerloop, outerloop, perf),\r\n\r\nAs more tests are written, there may be more tests that will have some special requirement, e.g. as @roncain said, the need for a machine where the machine's config can be changed and later cleaned up. \r\n\r\nWithin the context of how tests are categorized today: perhaps something like `[OuterLoop(\"WcfServerRequired\")]` or `[Performance(\"CrossMachineOnly\")]`would be sufficient to specify any additional requirements on the base categories that allow them. \r\n\r\nWith that said though, it will be important that we don't have *too* many categories/custom requirements - I trust that the community will also keep an eye out on custom category explosion if we know it's something to look out for? "}}