{"action":"created","issue":{"url":"https://api.github.com/repos/dotnet/corefx/issues/1477","labels_url":"https://api.github.com/repos/dotnet/corefx/issues/1477/labels{/name}","comments_url":"https://api.github.com/repos/dotnet/corefx/issues/1477/comments","events_url":"https://api.github.com/repos/dotnet/corefx/issues/1477/events","html_url":"https://github.com/dotnet/corefx/issues/1477","id":70288005,"number":1477,"title":"Test infrastructure requirements and scenarios","user":{"login":"Priya91","id":7853870,"avatar_url":"https://avatars.githubusercontent.com/u/7853870?v=3","gravatar_id":"","url":"https://api.github.com/users/Priya91","html_url":"https://github.com/Priya91","followers_url":"https://api.github.com/users/Priya91/followers","following_url":"https://api.github.com/users/Priya91/following{/other_user}","gists_url":"https://api.github.com/users/Priya91/gists{/gist_id}","starred_url":"https://api.github.com/users/Priya91/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Priya91/subscriptions","organizations_url":"https://api.github.com/users/Priya91/orgs","repos_url":"https://api.github.com/users/Priya91/repos","events_url":"https://api.github.com/users/Priya91/events{/privacy}","received_events_url":"https://api.github.com/users/Priya91/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/dotnet/corefx/labels/1%20-%20Up%20Next","name":"1 - Up Next","color":"CCCCCC"},{"url":"https://api.github.com/repos/dotnet/corefx/labels/Infrastructure","name":"Infrastructure","color":"fbca04"}],"state":"open","locked":false,"assignee":{"login":"piotrpMSFT","id":2379589,"avatar_url":"https://avatars.githubusercontent.com/u/2379589?v=3","gravatar_id":"","url":"https://api.github.com/users/piotrpMSFT","html_url":"https://github.com/piotrpMSFT","followers_url":"https://api.github.com/users/piotrpMSFT/followers","following_url":"https://api.github.com/users/piotrpMSFT/following{/other_user}","gists_url":"https://api.github.com/users/piotrpMSFT/gists{/gist_id}","starred_url":"https://api.github.com/users/piotrpMSFT/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/piotrpMSFT/subscriptions","organizations_url":"https://api.github.com/users/piotrpMSFT/orgs","repos_url":"https://api.github.com/users/piotrpMSFT/repos","events_url":"https://api.github.com/users/piotrpMSFT/events{/privacy}","received_events_url":"https://api.github.com/users/piotrpMSFT/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/dotnet/corefx/milestones/1","html_url":"https://github.com/dotnet/corefx/milestones/1.0.0-beta7","labels_url":"https://api.github.com/repos/dotnet/corefx/milestones/1/labels","id":1118973,"number":1,"title":"1.0.0-beta7","description":"","creator":{"login":"joshfree","id":1302850,"avatar_url":"https://avatars.githubusercontent.com/u/1302850?v=3","gravatar_id":"","url":"https://api.github.com/users/joshfree","html_url":"https://github.com/joshfree","followers_url":"https://api.github.com/users/joshfree/followers","following_url":"https://api.github.com/users/joshfree/following{/other_user}","gists_url":"https://api.github.com/users/joshfree/gists{/gist_id}","starred_url":"https://api.github.com/users/joshfree/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joshfree/subscriptions","organizations_url":"https://api.github.com/users/joshfree/orgs","repos_url":"https://api.github.com/users/joshfree/repos","events_url":"https://api.github.com/users/joshfree/events{/privacy}","received_events_url":"https://api.github.com/users/joshfree/received_events","type":"User","site_admin":false},"open_issues":16,"closed_issues":15,"state":"open","created_at":"2015-05-19T18:01:35Z","updated_at":"2015-07-09T16:23:26Z","due_on":null,"closed_at":null},"comments":24,"created_at":"2015-04-23T02:44:30Z","updated_at":"2015-07-09T21:35:54Z","closed_at":null,"body":"Current updated proposal from @Chrisboh :\r\n\r\n**Requirements we need to satisfy**\r\n\t• Run a set of tests that all developers should run before checking in\r\n\t• Specify a set of tests to run from the command line\r\n\t• Be able to debug all test cases\r\n\t• Run with code coverage turned on\r\n\t• Only build what is required\r\n\r\n## Proposal\r\nThe key to this proposal starts at build time. We need to only build what we are going to run.  This is important both for overall runtime as well running tests using .net native.  In order to satisfy this we need to move our ability to filter up higher in the process.  \r\n\r\n\r\n## Building only What is Required\r\nThe following include is where we find all of our test projects and add them to the list of items we would like to build.  \r\n**File:** CoreFX\\src\\dirs.proj (line 8)\r\n`<Project Include=\"*\\tests\\**\\*.csproj\" Exclude=\"@(ExcludeProjects)\" />`\r\n\r\nWe need to add a condition to this to only include items that match a supplied filter.  This filter would contain the following items.\r\n\r\n**Filter List:**\r\n\t• InnerLoop\r\n\t• Functional\r\n\t• Partner\r\n\t• Customer\r\n\r\n#### InnerLoop\r\nThis will contain everything we consider an Unit test today and until we complete the work to run our tests based on selectivity should be the minimum required set of tests that get run before accepting any PR.\r\n\r\n#### Functional (OuterLoop Ring 1)\r\nThis category is made up of test cases that will test larger sections of the code and / or take longer to run.  The tests in this category should also give us a quick high level check that there are no major issues with regards to basic performance, stress, or security.\r\n\r\n#### Partner (OuterLoop Ring 2)\r\nThe goal of this category is to determine if we are ready for team Dogfooding.  All performance and stress tests should be in this category as well as any compliance / security tests.\r\n\r\n#### Customer (OuterLoop Ring 3)\r\nThe goal of this category is to determine if we are ready to release this for Partner Dogfooding.  This will contain all scenario test cases and will have a manual component added as well.\r\n\r\n\r\n## Defining Test Project\r\nIn order for our filtering to work correctly we need to ensure that test project is setup correctly.  The simple way to do this is to create a test project for each category in the Filter list.  This would give us at most 4 test projects but it will be more typical to have around 2 projects as I could see unifying scenario, compliance, and security tests in a single location.  In order to do this we will need to filter by test project name.  We will follow the following naming conventions.\r\n\r\n`*.<Category>.tests.dll`\r\n\r\nExample:\r\n`System.ObjectModel.InnerLoop.tests.dll`\r\n\r\n\r\n## Xunit Attributes\r\nNow it is time to address what we are going to be doing about the Xunit attributes.  First we will remove all attributes that can be replaced with the project filters.  This will include things like InnerLoop, OuterLoop, Performance, and Stress.  These attributes will not be used when deciding what binaries we should build and run.  What this doesn't mean is that we will be removing all attributes entirely.  We still need to keep and leverage certain attributes in order to give us the flexibility we need to further fine tune our runs.\r\n\r\nHere are some of the key attributes we will need to leave in place.\r\n\r\n#### Active Issue Attribute\r\nThis attribute will be used to filter out failing test cases when running locally.  When running our daily builds we should include these test cases and label them as failing with a known bug.\r\n\r\n#### Platform Specific Attribute\r\nThis attribute will be used to filter out test cases that do not apply to the platform you are currently running on.  There was some debate on whether we should promote this to the filter list or not.  At this time we don’t feel there are enough test cases to warrant promoting this to the filter list.  In the future if this becomes an issue we can revisit this.\r\n\r\n## Command Line\r\nWe will need to execute everything from the command line as well as be able to select everything we need.  Doing so will be done via two properties.  The first will be defining the test categories that you would like to include.  This will be done with the following property.\r\n\r\n`/p:TestTypes=\"InnerLoop;Functional\"`\r\n\r\nThe second property will be used to supply extra command line parameters to Xunit.  This will simply be a pass-through from the msbuild command line onto Xunit and we will use the following property for that.\r\n\r\n`/p:AdditionalXunitArgs=\"-notrait ActiveIssue\"`\r\n\r\nAdditionally if you do not specify the TestTypes property it should run all Developer test cases automatically. Also not specifying the AdditionalXunitArgs property should be default exclude any test with an Active Issue and regardless if the property is specified we should include the correct OS parameter.  \r\n\r\n## Debugging\r\nThis will be the same as what we have today.  In the future we should invest me making the Xunit.runner.VisualStudio work and this would simplify everything.\r\n"},"comment":{"url":"https://api.github.com/repos/dotnet/corefx/issues/comments/120150425","html_url":"https://github.com/dotnet/corefx/issues/1477#issuecomment-120150425","issue_url":"https://api.github.com/repos/dotnet/corefx/issues/1477","id":120150425,"user":{"login":"joshfree","id":1302850,"avatar_url":"https://avatars.githubusercontent.com/u/1302850?v=3","gravatar_id":"","url":"https://api.github.com/users/joshfree","html_url":"https://github.com/joshfree","followers_url":"https://api.github.com/users/joshfree/followers","following_url":"https://api.github.com/users/joshfree/following{/other_user}","gists_url":"https://api.github.com/users/joshfree/gists{/gist_id}","starred_url":"https://api.github.com/users/joshfree/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/joshfree/subscriptions","organizations_url":"https://api.github.com/users/joshfree/orgs","repos_url":"https://api.github.com/users/joshfree/repos","events_url":"https://api.github.com/users/joshfree/events{/privacy}","received_events_url":"https://api.github.com/users/joshfree/received_events","type":"User","site_admin":false},"created_at":"2015-07-09T21:35:54Z","updated_at":"2015-07-09T21:35:54Z","body":"/cc @piotrpMSFT "}}