{"action":"created","issue":{"url":"https://api.github.com/repos/ARMmbed/greentea/issues/30","labels_url":"https://api.github.com/repos/ARMmbed/greentea/issues/30/labels{/name}","comments_url":"https://api.github.com/repos/ARMmbed/greentea/issues/30/comments","events_url":"https://api.github.com/repos/ARMmbed/greentea/issues/30/events","html_url":"https://github.com/ARMmbed/greentea/pull/30","id":114812389,"number":30,"title":"Experimental, parallel test execution","user":{"login":"stegut01","id":14980131,"avatar_url":"https://avatars.githubusercontent.com/u/14980131?v=3","gravatar_id":"","url":"https://api.github.com/users/stegut01","html_url":"https://github.com/stegut01","followers_url":"https://api.github.com/users/stegut01/followers","following_url":"https://api.github.com/users/stegut01/following{/other_user}","gists_url":"https://api.github.com/users/stegut01/gists{/gist_id}","starred_url":"https://api.github.com/users/stegut01/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stegut01/subscriptions","organizations_url":"https://api.github.com/users/stegut01/orgs","repos_url":"https://api.github.com/users/stegut01/repos","events_url":"https://api.github.com/users/stegut01/events{/privacy}","received_events_url":"https://api.github.com/users/stegut01/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-11-03T13:11:10Z","updated_at":"2015-11-03T13:39:42Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ARMmbed/greentea/pulls/30","html_url":"https://github.com/ARMmbed/greentea/pull/30","diff_url":"https://github.com/ARMmbed/greentea/pull/30.diff","patch_url":"https://github.com/ARMmbed/greentea/pull/30.patch"},"body":"# Description\r\n\r\nAdded new experimental feature: thread model for running mulitple test jobs in parallel on multiple resources instances (boards) of the same type within greentea, where for each board we have a separate thread.\r\n\r\nCurrently each unique MUT (mbed under test) is flashed separately one by one, but with this new feature we are able to flash multiple boards at the same time, run test case on hardware and get results in parallel.\r\n\r\nTest execution time is reduced by n where n is number of MUTs (MUT: mbed under test).\r\n\r\n## Limitations\r\n\r\nAt the moment the parallelisation is only for resource instances of the **same type**.\r\nDon't use the ```--verbose-test-result``` mode with the new ```--parallel``` mode.\r\n\r\n# Usage\r\n\r\nTo test in parallel use:\r\n```\r\n$ mbedgt -t frdm-k64f-gcc --parallel 2\r\n```\r\nWhere ```-t``` specifies the target and ```--parallel``` enables parallel testing with an *int* argument, which specifies the number of boards used in parallel. Logically, it is not possible to test on more devices as connected.\r\n\r\n## Example\r\n\r\nWe have two K64F platforms connected:\r\n```\r\n$ mbedls\r\n+---------------------+----------------------------+-------------------+-------------------+--------------------------------+\r\n|platform_name        |platform_name_unique        |mount_point        |serial_port        |target_id                       |\r\n+---------------------+----------------------------+-------------------+-------------------+--------------------------------+\r\n|K64F                 |K64F[0]                     |E:                 |COM12              |02400201A0B31E095D4DE3B1        |\r\n|K64F                 |K64F[1]                     |F:                 |COM13              |02400201489A1E6CB564E3D4        |\r\n+---------------------+----------------------------+-------------------+-------------------+--------------------------------+\r\n```\r\nWe want to force Greentea to run 2 threads and divide the test cases, so that we can achieve a speedup of about 2x.\r\n\r\nWe will show example with tests from mbed-drivers repository. \r\n```\r\n$ cd mbed-drivers\r\n$ mbedgt -t frdm-k64f-gcc --parallel 2\r\n```\r\n```\r\nmbedgt: detecting connected mbed-enabled devices...\r\nmbedgt: detected 2 devices\r\n        detected 'K64F' -> 'K64F[0]', console at 'COM12', mounted at 'E:', target id '02400201A0B31E095D4DE3B1'\r\n        detected 'K64F' -> 'K64F[1]', console at 'COM13', mounted at 'F:', target id '02400201489A1E6CB564E3D4'\r\nmbedgt: yotta search for mbed-target 'k64f'\r\n        calling yotta: yotta --plain search -k mbed-target:k64f target\r\n        found target 'frdm-k64f-gcc'\r\n        found target 'frdm-k64f-armcc'\r\nmbedgt: processing 'frdm-k64f-gcc' yotta target compatible platforms...\r\nmbedgt: processing 'K64F' platform...\r\nmbedgt: using platform 'K64F' for test:\r\n        target_id_mbed_htm = '02400201A0B31E095D4DE3B1'\r\n        mount_point = 'E:'\r\n        target_id = '02400201A0B31E095D4DE3B1'\r\n        serial_port = 'COM12'\r\n        target_id_usb_id = '02400201A0B31E095D4DE3B1'\r\n        platform_name = 'K64F'\r\n        platform_name_unique = 'K64F[0]'\r\nmbedgt: using platform 'K64F' for test:\r\n        target_id_mbed_htm = '02400201489A1E6CB564E3D4'\r\n        mount_point = 'F:'\r\n        target_id = '02400201489A1E6CB564E3D4'\r\n        serial_port = 'COM13'\r\n        target_id_usb_id = '02400201489A1E6CB564E3D4'\r\n        platform_name = 'K64F'\r\n        platform_name_unique = 'K64F[1]'\r\nmbedgt: building your sources and tests with yotta...\r\n        calling yotta: yotta --target=frdm-k64f-gcc,* build\r\ninfo: generate for target: frdm-k64f-gcc 0.1.3 at C:\\Users\\stefan.gutmann@arm.com\\Documents\\GitHub\\mbed-drivers\\yotta_targets\\frdm-k64f-gcc\r\nGCC version is: 4.9.3\r\n-- Configuring done\r\n-- Generating done\r\n-- Build files have been written to: C:/Users/stefan.gutmann@arm.com/Documents/GitHub/mbed-drivers/build/frdm-k64f-gcc\r\nninja: no work to do.\r\nmbedgt: yotta build for target 'frdm-k64f-gcc' was successful\r\nmbedgt: running 23 tests for target 'frdm-k64f-gcc' and platform 'K64F'\r\nmbedgt: building your sources and tests with yotta...\r\n        calling yotta: yotta --target=frdm-k64f-gcc,* build\r\ninfo: generate for target: frdm-k64f-gcc 0.1.3 at C:\\Users\\stefan.gutmann@arm.com\\Documents\\GitHub\\mbed-drivers\\yotta_targets\\frdm-k64f-gcc\r\nGCC version is: 4.9.3\r\n-- Configuring done\r\n-- Generating done\r\n-- Build files have been written to: C:/Users/stefan.gutmann@arm.com/Documents/GitHub/mbed-drivers/build/frdm-k64f-gcc\r\nninja: no work to do.\r\nmbedgt: yotta build for target 'frdm-k64f-gcc' was successful\r\nmbedgt: running 23 tests for target 'frdm-k64f-gcc' and platform 'K64F'\r\n        use 4 instances for parallel testing\r\nmbedgt: Test on Hardware with Target ID: 02400201A0B31E095D4DE3B1\r\n        Test 'mbed-drivers-test-echo' .......................................................... OK in 6.33 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201489A1E6CB564E3D4\r\n        Test 'mbed-drivers-test-time_us' ....................................................... OK in 11.37 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201A0B31E095D4DE3B1\r\n        Test 'mbed-drivers-test-serial_interrupt' .............................................. OK in 6.18 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201A0B31E095D4DE3B1\r\n        Test 'mbed-drivers-test-functionpointer' ............................................... OK in 1.38 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201A0B31E095D4DE3B1\r\n        Test 'mbed-drivers-test-stdio' ......................................................... OK in 0.74 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201489A1E6CB564E3D4\r\n        Test 'mbed-drivers-test-blinky' ........................................................ TIMEOUT in 20.92 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201A0B31E095D4DE3B1\r\n        Test 'mbed-drivers-test-eventhandler' .................................................. ERROR in 0.51 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201489A1E6CB564E3D4\r\n        Test 'mbed-drivers-test-stl' ........................................................... OK in 1.38 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201A0B31E095D4DE3B1\r\n        Test 'mbed-drivers-test-div' ........................................................... OK in 1.38 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201A0B31E095D4DE3B1\r\n        Test 'mbed-drivers-test-cstring' ....................................................... FAIL in 1.36 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201489A1E6CB564E3D4\r\n        Test 'mbed-drivers-test-rtc' ........................................................... OK in 4.60 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201A0B31E095D4DE3B1\r\n        Test 'mbed-drivers-test-cpp' ........................................................... OK in 1.36 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201489A1E6CB564E3D4\r\n        Test 'mbed-drivers-test-timeout' ....................................................... OK in 11.54 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201A0B31E095D4DE3B1\r\n        Test 'mbed-drivers-test-ticker_3' ...................................................... OK in 11.36 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201A0B31E095D4DE3B1\r\n        Test 'mbed-drivers-test-heap_and_stack' ................................................ OK in 1.34 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201489A1E6CB564E3D4\r\n        Test 'mbed-drivers-test-ticker_2' ...................................................... OK in 11.38 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201A0B31E095D4DE3B1\r\n        Test 'mbed-drivers-test-hello' ......................................................... OK in 0.38 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201A0B31E095D4DE3B1\r\n        Test 'mbed-drivers-test-dev_null' ...................................................... OK in 3.50 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201489A1E6CB564E3D4\r\n        Test 'mbed-drivers-test-ticker' ........................................................ OK in 11.37 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201A0B31E095D4DE3B1\r\n        Test 'mbed-drivers-test-basic' ......................................................... OK in 1.38 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201489A1E6CB564E3D4\r\n        Test 'mbed-drivers-test-asynch_spi' .................................................... ERROR in 0.42 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201489A1E6CB564E3D4\r\n        Test 'mbed-drivers-test-detect' ........................................................ FAIL in 0.47 sec\r\nmbedgt: Test on Hardware with Target ID: 02400201A0B31E095D4DE3B1\r\n        Test 'mbed-drivers-test-sleep_timeout' ................................................. OK in 3.41 sec\r\nmbedgt: test report:\r\n+---------------+---------------+------------------------------------+---------+--------------------+-------------+\r\n| target        | platform_name | test                               | result  | elapsed_time (sec) | copy_method |\r\n+---------------+---------------+------------------------------------+---------+--------------------+-------------+\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-div              | OK      | 1.38               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-rtc              | OK      | 4.6                | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-cstring          | FAIL    | 1.36               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-cpp              | OK      | 1.36               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-timeout          | OK      | 11.54              | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-ticker_3         | OK      | 11.36              | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-ticker_2         | OK      | 11.38              | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-time_us          | OK      | 11.37              | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-hello            | OK      | 0.38               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-ticker           | OK      | 11.37              | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-echo             | OK      | 6.33               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-serial_interrupt | OK      | 6.18               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-blinky           | TIMEOUT | 20.92              | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-functionpointer  | OK      | 1.38               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-stdio            | OK      | 0.74               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-sleep_timeout    | OK      | 3.41               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-dev_null         | OK      | 3.5                | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-basic            | OK      | 1.38               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-eventhandler     | ERROR   | 0.51               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-heap_and_stack   | OK      | 1.34               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-asynch_spi       | ERROR   | 0.42               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-stl              | OK      | 1.38               | shell       |\r\n| frdm-k64f-gcc | K64F          | mbed-drivers-test-detect           | FAIL    | 0.47               | shell       |\r\n+---------------+---------------+------------------------------------+---------+--------------------+-------------+\r\n\r\nResult: 2 FAIL / 18 OK / 1 TIMEOUT / 2 ERROR\r\nCompleted in 141.05 sec\r\n```"},"comment":{"url":"https://api.github.com/repos/ARMmbed/greentea/issues/comments/153355366","html_url":"https://github.com/ARMmbed/greentea/pull/30#issuecomment-153355366","issue_url":"https://api.github.com/repos/ARMmbed/greentea/issues/30","id":153355366,"user":{"login":"PrzemekWirkus","id":6360720,"avatar_url":"https://avatars.githubusercontent.com/u/6360720?v=3","gravatar_id":"","url":"https://api.github.com/users/PrzemekWirkus","html_url":"https://github.com/PrzemekWirkus","followers_url":"https://api.github.com/users/PrzemekWirkus/followers","following_url":"https://api.github.com/users/PrzemekWirkus/following{/other_user}","gists_url":"https://api.github.com/users/PrzemekWirkus/gists{/gist_id}","starred_url":"https://api.github.com/users/PrzemekWirkus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PrzemekWirkus/subscriptions","organizations_url":"https://api.github.com/users/PrzemekWirkus/orgs","repos_url":"https://api.github.com/users/PrzemekWirkus/repos","events_url":"https://api.github.com/users/PrzemekWirkus/events{/privacy}","received_events_url":"https://api.github.com/users/PrzemekWirkus/received_events","type":"User","site_admin":false},"created_at":"2015-11-03T13:39:42Z","updated_at":"2015-11-03T13:39:42Z","body":"Comments:\r\n* Please remove CammelCase from prints (only use lowercase text) - we want to allign with how yotta prints stuff\r\n* Please make sure Ctrl+C stops parallel execution of Greentea\r\n"}}