{"action":"created","issue":{"url":"https://api.github.com/repos/sherpa/sherpa/issues/31","labels_url":"https://api.github.com/repos/sherpa/sherpa/issues/31/labels{/name}","comments_url":"https://api.github.com/repos/sherpa/sherpa/issues/31/comments","events_url":"https://api.github.com/repos/sherpa/sherpa/issues/31/events","html_url":"https://github.com/sherpa/sherpa/pull/31","id":82468248,"number":31,"title":"Feature - New build and test infrastructure","user":{"login":"olaurino","id":5692321,"avatar_url":"https://avatars.githubusercontent.com/u/5692321?v=3","gravatar_id":"","url":"https://api.github.com/users/olaurino","html_url":"https://github.com/olaurino","followers_url":"https://api.github.com/users/olaurino/followers","following_url":"https://api.github.com/users/olaurino/following{/other_user}","gists_url":"https://api.github.com/users/olaurino/gists{/gist_id}","starred_url":"https://api.github.com/users/olaurino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olaurino/subscriptions","organizations_url":"https://api.github.com/users/olaurino/orgs","repos_url":"https://api.github.com/users/olaurino/repos","events_url":"https://api.github.com/users/olaurino/events{/privacy}","received_events_url":"https://api.github.com/users/olaurino/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/sherpa/sherpa/labels/3%20%20-%20Working","name":"3  - Working","color":"ededed"},{"url":"https://api.github.com/repos/sherpa/sherpa/labels/area:build","name":"area:build","color":"d4c5f9"},{"url":"https://api.github.com/repos/sherpa/sherpa/labels/area:tests","name":"area:tests","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/sherpa/sherpa/milestones/1","html_url":"https://github.com/sherpa/sherpa/milestones/4.8rc1","labels_url":"https://api.github.com/repos/sherpa/sherpa/milestones/1/labels","id":1120770,"number":1,"title":"4.8rc1","description":"","creator":{"login":"olaurino","id":5692321,"avatar_url":"https://avatars.githubusercontent.com/u/5692321?v=3","gravatar_id":"","url":"https://api.github.com/users/olaurino","html_url":"https://github.com/olaurino","followers_url":"https://api.github.com/users/olaurino/followers","following_url":"https://api.github.com/users/olaurino/following{/other_user}","gists_url":"https://api.github.com/users/olaurino/gists{/gist_id}","starred_url":"https://api.github.com/users/olaurino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olaurino/subscriptions","organizations_url":"https://api.github.com/users/olaurino/orgs","repos_url":"https://api.github.com/users/olaurino/repos","events_url":"https://api.github.com/users/olaurino/events{/privacy}","received_events_url":"https://api.github.com/users/olaurino/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":8,"state":"open","created_at":"2015-05-20T11:33:14Z","updated_at":"2015-05-29T15:28:18Z","due_on":"2015-10-01T04:00:00Z","closed_at":null},"comments":8,"created_at":"2015-05-29T15:17:05Z","updated_at":"2015-05-29T18:21:52Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/sherpa/sherpa/pulls/31","html_url":"https://github.com/sherpa/sherpa/pull/31","diff_url":"https://github.com/sherpa/sherpa/pull/31.diff","patch_url":"https://github.com/sherpa/sherpa/pull/31.patch"},"body":"### Release Note\r\nThe GitHub repo is now integrated with [Travis CI](https://travis-ci.org), so that builds will be automatically started every time there a commit is pushed to GitHub, or a Pull Request is issued. The full regression tests suite will be run after the build.\r\nDevelopers using `git` will now be able to `init` and `update` a `git` submodule with external data and additional regression tests. A simple `python setup.py test` will run the tests suite. Package [`py.test`](http://pytest.org/latest/) is required for running the tests suite in this fashion. For compatibility with CIAO, `python -c \"import sherpa; sherpa.test()\"` is left untouched, and can be run without any additional dependencies. The `sherpa_test` console script has been updated to use the new framework and now requires `py.test`. The tests now try and find the external data directory at runtime, so no command line arguments are required and discovery of tests is automatic. However, for compatibility, `sherpa.test()` still needs the `datadir` argument to properly run: if `datadir` is not provided, only the embedded tests will be run (*smoke* test).\r\n\r\n### PR Description\r\nThis PR introduces a new testing infrastructure for running the whole Sherpa unit and regression tests suite from the `git` repo, a source distribution, or a binary package. This includes migrating the tests that would run only on the CXC systems (see [the `sherpa-test-data` repo](https://github.com/sherpa/sherpa-test-data)) as a `git` submodule. Moreover, it integrates the GitHub repository with [Travis CI](https://travis-ci.org), so that builds will be automatically started every time there a commit is pushed to GitHub, or a Pull Request is issued, and tests are run after the build.\r\n\r\nThis PR depends on PR #28. \r\n\r\n### Requirements\r\n  * we want to keep `python -c \"import sherpa; sherpa.test()\"` as is for compatibility with CIAO: the data directory path can be passed from the command line in this case.\r\n  * we want the tests to find out at runtime where the data directory is.\r\n  * the data directory needs to be external to the sherpa codebase, i.e. not in the same `git` repository.\r\n  * the data directory needs to be versioned.\r\n  * since the data directory and sherpa are versioned independently, we need to synchronize the Sherpa revision with the `sherpa-test-data` revision.\r\n  * ideally, `python setup.py test` should be able to discover the test data directory independently of whether `git` is being used or not, i.e. a regular user without `git` should be able to run the tests in a transparent way.\r\n\r\n### Design\r\nOne natural choice to meet most of the above requirements is to create a Python project containing the test data, and to include it as a `git` submodule of the main Sherpa repository. The project could be either a Python module (i.e. with a `__init__.py`), or its own installable Python package. \r\n\r\nIn the first case, we should rely on the relative position of the `sherpatest` module so that it can be imported. \r\n\r\nHowever, the first case would work fine assuming `git` is installed. Having the test data directory in its own Python project would allow it to be installed without `git`: it may be listed as a regular package in `test_requirements.txt` and installed with a `pip install -r test_requirements` command, or it might have its own `conda` package. In this case, however, it is not clear how one would synchronize the versions: this could be done for releases, but not during development.\r\n\r\nSo, making `git` unnecessary does not seem workable if not for releases, and having the `git` submodule as a standalone Python package seems to be more flexible.\r\n\r\nIn summary the design can be summarized as follows:\r\n  * create a standalone python package for holding the test data\r\n  * add it as a `git` submodule to the main Sherpa repo\r\n  * at release time, the test data package might be released alongside Sherpa with the same version, so that users may be able to run the full test suite simply by installing the test data package.\r\n  * in CIAO one may rely on the folder of the `git` submodule as the new `testdata` directory.\r\n  * `sherpa_test` will be used as a handle to run the tests in non-development mode. `sherpa.test()` can be used for testing in CIAO.\r\n  * create a `test_\r\n\r\n#### Summary of the UI\r\n  * During development, one runs `python setup.py develop` and tests with `python setup.py test`. The `git` submodule must be initialized and updated for the full regression tests to run, as described in the [`git-submodule` documentation](http://git-scm.com/docs/git-submodule).\r\n  * When installing from sources one runs `python setup.py install` and tests with `sherpa_test`.\r\n    * If using `git`, one can initialize and update the submodule for access to the full regression tests suite. \r\n    * Otherwise one can install the `sherpatest` package and the tests will figure out the location of the external data directory.\r\n  * When installing from binaries, after the installation one can run `sherpa_test`. For running the whole regression suite one can install `sherpatest` and run `sherpa_test`.\r\n\r\n### Implementation\r\nThe implementation relies on the cleanup of the `testdata` directory that @jbudynk performed. The cleaned-up folder was imported into [a new `git` repo](https://github.com/sherpa/sherpa-test-data) and transformed into an installable Python package. The repo was also included as a `git` submodule into the main Sherpa repo, so that changes in the revisions of the repos can be tracked, and one can always couple the Sherpa revision with the `sherpa-test-data` matching revision.\r\n\r\nSome of the old Sherpa tests required CIAO components and would fail when run in standalone mode. Such tests were modified accordingly. In particular:\r\n  * `ciao4.3/lev3fft/bar.py`: This script requires the CIAO Data Model support to read a region from a `FITS` region file. The test was changed to employ the region string equivalent to the one defined in the `FITS` file.\r\n  * `sherpa/astro/ui/tests/test_astro_ui.py:245`: This test failed because the `pyfits_backend` did not return a proper response matrix from a non-Chandra file representing a diagonal matrix (`threads/chi2/SWIFTJ0840.1+2946.pha.gz` in the old CXC `testdata` dir). The tests works fine when `PyCRATES` (the CIAO backend) is used. Since the test file did not have anything special to do with the code being tested, it was replaced by a Chandra file.\r\n\r\nOne more issue was triggered by using `py.test` rather than the plain `unittest`, as apparently they have different settings of the system path when tests are run: the failing test was `sherpa/astro/tests/test_astro.py:59`, which raised a `ImportError: No module named acis_bkg_model`. Making the test `eval` the model script is enough to make the test more portable and run both with and without `py.test`.\r\n\r\n### Tests\r\n[Build 10 on Travis CI](https://travis-ci.org/olaurino/sherpa/builds/64483010) shows several possible testing scenarios described above. In particular, builds are created with the following assumptions:\r\n  - `pyfits` is not installed, the user builds Sherpa with `python setup.py develop`, and tests it using the `git` submodule.\r\n  - same as above, but `pyfits` is installed.\r\n  - `pyfits` is not installed, the user builds Sherpa with `python setup.py install`, and tests it by installing the `sherpatest` package and by using `sherpa_test`.\r\n  - same as above, but `pyfits` is installed.\r\n  - `smoke test` mode: the user installs Sherpa with `python setup.py install`, `pyfits` is installed, `sherpatest` is not installed, and `sherpa_test` is used to test the installation.\r\n\r\nBuild 10 Travis configuration is too heavyweight for being the default configuration, so some of the environment combinations in the Travis configuration were commented out for the time being.\r\n\r\n[Build 13 on Travis](https://travis-ci.org/olaurino/sherpa/builds/64579829) shows the single scenario that has been left uncommented.\r\n\r\nI also tested on a CXC system that the full tests suite runs when XSPEC is installed.\r\n\r\n### Pass additional arguments to py.test\r\nIn order to run `py.test` with custom arguments, one can call `py.test` directly, or pass the arguments to `python setup.py test`.\r\n\r\nFor instance, the commands below show how to execute a single test case:\r\n```\r\npython setup.py test --pytest-args=python setup.py test \\\r\n  --pytest-args=\"sherpa/estmethods/tests/test_estmethods.py::test_estmethods\r\n```\r\n\r\nor a single test:\r\n```\r\npython setup.py test --pytest-args=python setup.py test \\\r\n  --pytest-args=\"sherpa/estmethods/tests/test_estmethods.py::test_estmethods::test_covar\r\n```\r\n\r\n### Future improvements\r\n  * we may want to add xspec, but this requires figuring out the most straightforward way of installing xspec and making it available to Travis workers.\r\n  * clean up `sherpa-test-data` some more: in particular, get rid of the `ciao-4.3` folder, and move files around to get a more rational folder structure. This requires changing the tests embedded in Sherpa, so it was left for a later stage.\r\n  * enable coverage reports (e.g. through `coveralls`). The configuration should already be there, but has been commented out for the time being.\r\n  * Improve test coverage by adding more tests.\r\n  * improve the `sherpa-test-data python package metadata.\r\n  * investigate issue with the pyfits backend and non-Chandra response matrices.\r\n  * test against different versions of Numpy in Travis.\r\n  * test conda binaries creation and upload development artifacts to binstar.\r\n"},"comment":{"url":"https://api.github.com/repos/sherpa/sherpa/issues/comments/106895552","html_url":"https://github.com/sherpa/sherpa/pull/31#issuecomment-106895552","issue_url":"https://api.github.com/repos/sherpa/sherpa/issues/31","id":106895552,"user":{"login":"olaurino","id":5692321,"avatar_url":"https://avatars.githubusercontent.com/u/5692321?v=3","gravatar_id":"","url":"https://api.github.com/users/olaurino","html_url":"https://github.com/olaurino","followers_url":"https://api.github.com/users/olaurino/followers","following_url":"https://api.github.com/users/olaurino/following{/other_user}","gists_url":"https://api.github.com/users/olaurino/gists{/gist_id}","starred_url":"https://api.github.com/users/olaurino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olaurino/subscriptions","organizations_url":"https://api.github.com/users/olaurino/orgs","repos_url":"https://api.github.com/users/olaurino/repos","events_url":"https://api.github.com/users/olaurino/events{/privacy}","received_events_url":"https://api.github.com/users/olaurino/received_events","type":"User","site_admin":false},"created_at":"2015-05-29T18:21:52Z","updated_at":"2015-05-29T18:21:52Z","body":"@cdeil \r\n\r\n> It would be nice to have some of that info how to run the tests in the https://github.com/sherpa/sherpa/blob/master/README.md#2d-testing-the-build or a separate DEVELOP.md. Or maybe you want to add that later to the Sphinx docs?\r\n\r\nAgreed. @DougBurke how do we coordinate documentation additions to PRs? We probably didn't discuss this, did we? I am happy to update the README.md, although I was not planning to do that just yet, as key aspects of this PR might still change.\r\n\r\n>> Build 10 Travis configuration is too heavyweight for being the default configuration, so some of the environment combinations in the Travis configuration were commented out for the time being.\r\n\r\n> Why? It's 5 builds that each ran for 5 minutes. Astropy has more builds and a much higher rate of triggered builds because of the many commits to many PRs.\r\n> I think ~ 10 or even 20 builds for Sherpa on travis-ci would be OK.\r\n\r\nI agree. However, I was testing different ways of installing and testing Sherpa in order to test the test infrastructure while I was tweaking it. At full speed, I don't think it's necessary to do this kind of meta-testing, while more builds may be necessary for \"better\" reasons. When there will be tests for different numpy versions, for pyfits vs astropy and maybe xspec vs no-xspec (not to mention python2 vs python3, linux vs mac), this kind of builds would just be noise. I can be convinced otherwise, though. So please let me know if you think that at least a subset of those original tests should be kept. Maybe we should keep the pyfits/no-pyfits tests and ditch the others?\r\n\r\n> One more thing ... if you do add some docs on the tests for future co-developers, please explain what smoke- and non-smoke tests are. It's not obvious to me (but I'll have a look tomorrow).\r\n\r\nGood point. A generic description of smoke testing is [on Wikipedia](http://en.wikipedia.org/wiki/Smoke_testing_(software)), but in CIAO we explicitly refer to smoke tests in the [user documentation](http://cxc.harvard.edu/ciao/threads/ciao_install/index.html#smoke): since the goal of this PR is to serve both the standalone development while keeping compatibility with the CIAO testing infrastructure, I referred to the *smoke test* in the CIAO lingo. I'll update the description of the PR to better explain the term. I am not sure this term will end up in the end-user documentation, but I'll keep your comment in mind!"}}