{"action":"created","issue":{"url":"https://api.github.com/repos/basho/riak_test/issues/786","labels_url":"https://api.github.com/repos/basho/riak_test/issues/786/labels{/name}","comments_url":"https://api.github.com/repos/basho/riak_test/issues/786/comments","events_url":"https://api.github.com/repos/basho/riak_test/issues/786/events","html_url":"https://github.com/basho/riak_test/pull/786","id":68940701,"number":786,"title":"Riak Test v1.1.0","user":{"login":"jburwell","id":285646,"avatar_url":"https://avatars.githubusercontent.com/u/285646?v=3","gravatar_id":"","url":"https://api.github.com/users/jburwell","html_url":"https://github.com/jburwell","followers_url":"https://api.github.com/users/jburwell/followers","following_url":"https://api.github.com/users/jburwell/following{/other_user}","gists_url":"https://api.github.com/users/jburwell/gists{/gist_id}","starred_url":"https://api.github.com/users/jburwell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jburwell/subscriptions","organizations_url":"https://api.github.com/users/jburwell/orgs","repos_url":"https://api.github.com/users/jburwell/repos","events_url":"https://api.github.com/users/jburwell/events{/privacy}","received_events_url":"https://api.github.com/users/jburwell/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-04-16T14:50:18Z","updated_at":"2015-04-16T16:20:35Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/basho/riak_test/pulls/786","html_url":"https://github.com/basho/riak_test/pull/786","diff_url":"https://github.com/basho/riak_test/pull/786.diff","patch_url":"https://github.com/basho/riak_test/pull/786.patch"},"body":"**This PR is not intended to be merged at this time.  It is being used to review the work done thus far, and begin the process of cleanup for merge into master.**\r\n\r\nRiak Test v1.1.0 ...  more writeup to come.\r\n\r\n/cc @javajolt"},"comment":{"url":"https://api.github.com/repos/basho/riak_test/issues/comments/93776418","html_url":"https://github.com/basho/riak_test/pull/786#issuecomment-93776418","issue_url":"https://api.github.com/repos/basho/riak_test/issues/786","id":93776418,"user":{"login":"jburwell","id":285646,"avatar_url":"https://avatars.githubusercontent.com/u/285646?v=3","gravatar_id":"","url":"https://api.github.com/users/jburwell","html_url":"https://github.com/jburwell","followers_url":"https://api.github.com/users/jburwell/followers","following_url":"https://api.github.com/users/jburwell/following{/other_user}","gists_url":"https://api.github.com/users/jburwell/gists{/gist_id}","starred_url":"https://api.github.com/users/jburwell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jburwell/subscriptions","organizations_url":"https://api.github.com/users/jburwell/orgs","repos_url":"https://api.github.com/users/jburwell/repos","events_url":"https://api.github.com/users/jburwell/events{/privacy}","received_events_url":"https://api.github.com/users/jburwell/received_events","type":"User","site_admin":false},"created_at":"2015-04-16T16:20:35Z","updated_at":"2015-04-16T16:20:35Z","body":"# Notes on the planner design\r\n\r\n``rt_planner`` and ``rt_reporter`` are good starts towards the pipeline design I have in mind for r_t.  After reviewing the initial implementation, I have the following observations:\r\n\r\n* The GiddyUp dependency can be further isolated in the codebase\r\n* The riak_test_escript module contains more test planning work than necessary\r\n* Isolate the use of rt_config to the riak_test_escript module as much as possible to simplify component unit testing\r\n\r\nIdeally, we have a pipeline constructed with the following stages:\r\n```\r\nplanner->scheduler->runner\r\n```\r\nEach of these stages can emit a test_result event that is broadcast to a list of reporters (e.g. console, giddyup).  \r\n\r\nA planner stage accepts a list of tests, backends, and groups and creates a set of test plans to be scheduled for execution.  The following is (roughly) the test planner API I have in mind:\r\n```\r\n-spec start_link(pid(), [pid()], proplists:proplist() -> rt_util:result().\r\nstart_link(SchedulerPid, ReporterPids, Options) -> %% the options are implementation dependent\r\n\r\n-spec plan([atom()], [backend()], [atom()]) -> {ok, non_neg_integer()} | rt_util:error().\r\nplan(Tests, Backends, Groups) ->  %% When the test list is empty, run all available tests\r\n```\r\n\r\nA scheduler stage accepts a test plan, reserves the quantity of nodes requested by the plan, and submits the schedule to an executor stage for execution.  The following is (roughly) the test scheduler API I have in mind:\r\n\r\n```\r\n-spec start_link(pid(), [pid()], proplists:proplist()).\r\nstart_link(ExecutorPid, ReporterPids, Options) -> %% Options are implementation dependent\r\n\r\n-spec schedule(#rt_planner:test_plan()) -> rt_util:result().\r\nschedule(TestPlan) ->\r\n```\r\nFinally, a Runner stage provisions the nodes allocated in the test schedule, executes the test specified in the test schedule, and reports the results of the test run the to reporters.  The following is (roughly) the API I have in mind:\r\n\r\n```\r\n-spec start_link([pids()], proplists:proplist()) -> rt_util:result().\r\nstart_link(ReporterPids, Options) ->\r\n\r\n-spec run(rt_scheduler:test_schedule) -> rt_util:result().\r\nrun(TestSchedule) ->\r\n```\r\n\r\nBased on this (high-level) design, I propose the following refactorings:\r\n\r\n1. Convert the ``giddyup`` module to a ``gen_server`` that has the ``giddyup`` configuration information passed into its start_link function.  In addition to encapsulating the GiddyUp configuration, it will also allow asynchronous communication with GiddyUp to prevent blocking of the pipeline if/when we have a slow connection.\r\n1. Refine the ``rt_planner`` module to define the planner behavior and any common functions used by all planners\r\n1. Move all GiddyUp related planning functions into a new ``rt_giddyup_planner`` module -- removing the notion of wrapping test plans in the escript module.  If a list of tests, backends, and/or groups is provided to this planner, they will be used to filter the list of test plans retrieved from GiddyUp.\r\n1. Create an ``rt_simple_planner`` module which is used when a list of tests is specified on the command line.\r\n1. Add a build_test_pipeline function to the riak_test_escript that selects and starts the planner, scheduler, and reporter(s) and ties them together for execution.\r\n\r\n/cc @javajolt "}}