{"action":"created","issue":{"url":"https://api.github.com/repos/avocado-framework/avocado/issues/632","labels_url":"https://api.github.com/repos/avocado-framework/avocado/issues/632/labels{/name}","comments_url":"https://api.github.com/repos/avocado-framework/avocado/issues/632/comments","events_url":"https://api.github.com/repos/avocado-framework/avocado/issues/632/events","html_url":"https://github.com/avocado-framework/avocado/pull/632","id":84798863,"number":632,"title":"[V11] Load/List tests from multiple sources","user":{"login":"lmr","id":296807,"avatar_url":"https://avatars.githubusercontent.com/u/296807?v=3","gravatar_id":"","url":"https://api.github.com/users/lmr","html_url":"https://github.com/lmr","followers_url":"https://api.github.com/users/lmr/followers","following_url":"https://api.github.com/users/lmr/following{/other_user}","gists_url":"https://api.github.com/users/lmr/gists{/gist_id}","starred_url":"https://api.github.com/users/lmr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmr/subscriptions","organizations_url":"https://api.github.com/users/lmr/orgs","repos_url":"https://api.github.com/users/lmr/repos","events_url":"https://api.github.com/users/lmr/events{/privacy}","received_events_url":"https://api.github.com/users/lmr/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-06-03T22:30:41Z","updated_at":"2015-06-05T19:53:57Z","closed_at":"2015-06-05T19:53:57Z","pull_request":{"url":"https://api.github.com/repos/avocado-framework/avocado/pulls/632","html_url":"https://github.com/avocado-framework/avocado/pull/632","diff_url":"https://github.com/avocado-framework/avocado/pull/632.diff","patch_url":"https://github.com/avocado-framework/avocado/pull/632.patch"},"body":"This is a follow up to PR #630 \r\n\r\nThe PR has been renamed because it no longer contains the actual virt test compatibility plugins. However, you definitely want to test this together with the avocado-vt initial pull request (see below).\r\n\r\nThis is the most fleshed out virt test compat layer to date. It introduces loading of tests from multiple test sources, which allows the mix and match of regular avocado tests and avocado virt-tests:\r\n\r\n\t$ scripts/avocado run --vt-setup type_specific.io-github-autotest-qemu.migrate.default.tcp examples/tests/passtest.py --open-browser\r\n\tSETUP      : PASS (15.66 s)\r\n\tJOB ID     : 0a613e41201ee2fa726a6ef8617f6578aa901e3a\r\n\tJOB LOG    : /home/lmr/avocado/job-results/job-2015-04-30T01.09-0a613e4/job.log\r\n\tJOB HTML   : /home/lmr/avocado/job-results/job-2015-04-30T01.09-0a613e4/html/results.html\r\n\tTESTS      : 2\r\n\t(1/2) type_specific.io-github-autotest-qemu.migrate.default.tcp: PASS (89.62 s)\r\n\t(2/2) examples/tests/passtest.py: PASS (0.00 s)\r\n\tPASS       : 2\r\n\tERROR      : 0\r\n\tFAIL       : 0\r\n\tSKIP       : 0\r\n\tWARN       : 0\r\n\tINTERRUPT  : 0\r\n\tTIME       : 89.62 s\r\n\r\nYou can also do cool things like listing all virt test `migrate` subtests:\r\n\r\n        $ scripts/avocado list migrate --verbose\r\n        Type Test                                                                  \r\n        VT   type_specific.io-github-autotest-qemu.migrate.default.x-rdma                   \r\n        VT   type_specific.io-github-autotest-qemu.migrate.default.rdma                     \r\n        VT   type_specific.io-github-autotest-qemu.migrate.default.tcp                      \r\n        VT   type_specific.io-github-autotest-qemu.migrate.default.unix                     \r\n        VT   type_specific.io-github-autotest-qemu.migrate.default.exec.default_exec        \r\n        VT   type_specific.io-github-autotest-qemu.migrate.default.exec.gzip_exec           \r\n        VT   type_specific.io-github-autotest-qemu.migrate.default.fd                       \r\n        VT   type_specific.io-github-autotest-qemu.migrate.default.mig_cancel               \r\n        VT   type_specific.io-github-autotest-qemu.migrate.default.mig_cancel_x_rdma        \r\n        VT   type_specific.io-github-autotest-qemu.migrate.default.mig_cancel_rdma          \r\n        VT   type_specific.io-github-autotest-qemu.migrate.with_set_speed.x-rdma            \r\n        VT   type_specific.io-github-autotest-qemu.migrate.with_set_speed.rdma              \r\n        VT   type_specific.io-github-autotest-qemu.migrate.with_set_speed.tcp               \r\n        VT   type_specific.io-github-autotest-qemu.migrate.with_set_speed.unix              \r\n        VT   type_specific.io-github-autotest-qemu.migrate.with_set_speed.exec.default_exec \r\n        VT   type_specific.io-github-autotest-qemu.migrate.with_set_speed.exec.gzip_exec    \r\n        VT   type_specific.io-github-autotest-qemu.migrate.with_set_speed.fd                \r\n        VT   type_specific.io-github-autotest-qemu.migrate.with_reboot.x-rdma               \r\n        VT   type_specific.io-github-autotest-qemu.migrate.with_reboot.rdma                 \r\n        VT   type_specific.io-github-autotest-qemu.migrate.with_reboot.tcp                  \r\n        VT   type_specific.io-github-autotest-qemu.migrate.with_reboot.unix                 \r\n        VT   type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.default_exec    \r\n        VT   type_specific.io-github-autotest-qemu.migrate.with_reboot.exec.gzip_exec       \r\n        VT   type_specific.io-github-autotest-qemu.migrate.with_reboot.fd                   \r\n\r\n        ACCESS_DENIED: 0\r\n        BROKEN_SYMLINK: 0\r\n        BUGGY: 0\r\n        INSTRUMENTED: 0\r\n        MISSING: 0\r\n        NOT_A_TEST: 0\r\n        SIMPLE: 0\r\n        VT: 24\r\n\r\nYou can try out this work by following the steps:\r\n\r\n1) Check out this branch\r\n2) Set the environment variable 'VIRT_TEST_PATH' to a path containing a copy of virt test\r\n3) The copy of virt-test has to be in my new branch `avocado-integration-v4` [1]\r\n4) Check out the avocado-vt repo, then merge PR 1 -> https://github.com/avocado-framework/avocado-vt/pull/3. The code checked out has to be in the same base dir as your copy of avocado\r\n5) Type `make link` in the avocado dir, so that the plugins can be properly linked.\r\n6) Bootstrap your testing in the virt test dir by running `./run -t qemu --bootstrap`\r\n7) Run your tests with the command above\r\n8) Now you can use `scripts/avocado list` to see virt-tests listed there.\r\n\r\nThat's it. Virt Test at this point has way too many knobs, and it's challenging to look to each of them and make sure they are working well. I hope that at least you guys enjoy trying this out.\r\n \r\nChanges from v10:\r\n* First version where the actual plugins are in a separate repo. Please refer to https://github.com/avocado-framework/avocado-vt/pull/1 to review that code. This PR only has the avocado relevant parts. Now this is aiming for inclusion. Let's do this, guys.\r\n\r\nChanges from v9:\r\n* Now the plugin honors the old virt test setting of `$AUTOTEST_PATH`, in case you want to use autotest from a git repo instead of from an RPM. In order to get consistency with that option, we changed `$VIRT_TEST_DIR` -> `$VIRT_TEST_PATH`.\r\n* `avocado list --vt-guests`: Filter by guest list `i440fx`, which prevents issues with the `q35` variant not listing tests\r\n* Set priority of test lister to be 0, to prevent issue when the virt test lister is loaded before the test lister: `virt_test_compat_lister': 'Parser' object has no attribute 'lister'`\r\n\r\nChanges from v8:\r\n* Finally the `avocado list` functionality is complete. You can list tests by `scripts/avocado list`, use `--vt-type` (different test types generate different test lists) and even list guests com `scripts/avocado --vt-list-guests` (different guests generate different test lists).\r\n\r\nChanges from v7:\r\n* Allow avocado list to show tests from all test loader plugins. This way, when enabled, virt-tests are displayed. There's one last thing lacking here, how to add new arguments to the lister plugin. I know how to do that for the runner plugin, I'll need @ruda's help to understand how to do it.\r\n\r\nChanges from v6:\r\n* Fixed bug running virt tests with multiplex file specified in the command line\r\n* Added an avocado_params attribute to the virt test class of the plugin, allowing users to use 'frankenstein mode' - access avocado specific muliplexed params from inside virt-tests.\r\n* Removed the aggressive error message when the virt plugin is not configured.\r\n\r\nChanges from v5:\r\n* Cleaned up the plugin code and adapted it to the refactorings I made on `virt-test`'s side.\r\n\r\nChanges from v4:\r\n * Finally resolved the problem with loading multiple test sources, while keeping the base filesystem plugin working the way it usually does. With this, unittest hacks could be thrown away. Unfortunately, the overhead with the virt plugin enabled makes a couple of unittests to fail, but since the compat plugin is going to be on a separate project, that isn't a big problem per se. Now, I'm certain that there are ways to make the multiple test loading more efficient. Figuring it all out at once is not easy, though.\r\n\r\nChanges from v3:\r\n * Removed the list and bootstrap plugins. virt-test can already do that functionality for the user.\r\n * Went to a excruciatingly detailed review of command line options. Many options that don't make sense in avocado were removed, text reviewed, and options grouped logically. Next step is to move unneeded options to the config file.\r\n\r\n[1] See https://github.com/autotest/virt-test/pull/2208\r\n"},"comment":{"url":"https://api.github.com/repos/avocado-framework/avocado/issues/comments/109420576","html_url":"https://github.com/avocado-framework/avocado/pull/632#issuecomment-109420576","issue_url":"https://api.github.com/repos/avocado-framework/avocado/issues/632","id":109420576,"user":{"login":"lmr","id":296807,"avatar_url":"https://avatars.githubusercontent.com/u/296807?v=3","gravatar_id":"","url":"https://api.github.com/users/lmr","html_url":"https://github.com/lmr","followers_url":"https://api.github.com/users/lmr/followers","following_url":"https://api.github.com/users/lmr/following{/other_user}","gists_url":"https://api.github.com/users/lmr/gists{/gist_id}","starred_url":"https://api.github.com/users/lmr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lmr/subscriptions","organizations_url":"https://api.github.com/users/lmr/orgs","repos_url":"https://api.github.com/users/lmr/repos","events_url":"https://api.github.com/users/lmr/events{/privacy}","received_events_url":"https://api.github.com/users/lmr/received_events","type":"User","site_admin":false},"created_at":"2015-06-05T19:53:57Z","updated_at":"2015-06-05T19:53:57Z","body":"Closing in favor of #638"}}