{"action":"created","issue":{"url":"https://api.github.com/repos/apache/trafficserver/issues/141","labels_url":"https://api.github.com/repos/apache/trafficserver/issues/141/labels{/name}","comments_url":"https://api.github.com/repos/apache/trafficserver/issues/141/comments","events_url":"https://api.github.com/repos/apache/trafficserver/issues/141/events","html_url":"https://github.com/apache/trafficserver/pull/141","id":48431618,"number":141,"title":"Add initial python-based functional test framework with example test","user":{"login":"duderino","id":1043139,"avatar_url":"https://avatars.githubusercontent.com/u/1043139?v=3","gravatar_id":"","url":"https://api.github.com/users/duderino","html_url":"https://github.com/duderino","followers_url":"https://api.github.com/users/duderino/followers","following_url":"https://api.github.com/users/duderino/following{/other_user}","gists_url":"https://api.github.com/users/duderino/gists{/gist_id}","starred_url":"https://api.github.com/users/duderino/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/duderino/subscriptions","organizations_url":"https://api.github.com/users/duderino/orgs","repos_url":"https://api.github.com/users/duderino/repos","events_url":"https://api.github.com/users/duderino/events{/privacy}","received_events_url":"https://api.github.com/users/duderino/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2014-11-11T20:34:54Z","updated_at":"2015-06-17T16:36:29Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/trafficserver/pulls/141","html_url":"https://github.com/apache/trafficserver/pull/141","diff_url":"https://github.com/apache/trafficserver/pull/141.diff","patch_url":"https://github.com/apache/trafficserver/pull/141.patch"},"body":"# Apache Traffic Server Functional Test Framework\r\n\r\nThe goal of the functional test framework is to allow traffic server contributors to develop and exercise a reasonable functional test framework from the comfort of their development environments.   This encourages traffic server developers to write their own functional test cases (it's not the province of a detached QA team).   This also gives traffic server developers critical test feedback as early in the development process as possible.\r\n\r\nThe framework consists of a 'process manager' responsible for configuring and spawning 1+ traffic server processes, a configuration-driven reusable origin process so you don't have to write your own for every test case (though sometimes you will), and a small number of utilities for the test cases themselves.   In this context the test cases are traffic server HTTP/SPDY clients.  That is:\r\n\r\n   * test cases/clients -->  traffic server  -->  (configurable/reusable origin | ad hoc origin)\r\n\r\n## Framework Prerequisites\r\n\r\n   * Python 2.6+ (the default python on RHEL, CentOS, etc. 6.5+)\r\n   * [Twisted](https://twistedmatrix.com/trac/) 14.0.2+\r\n      1. [Install pip](https://pip.pypa.io/en/latest/installing.html)\r\n      2. Install twisted and its dependencies using pip:\r\n```      \r\n$ sudo pip install --upgrade pyOpenSSL zope.interface twisted\r\n```\r\n\r\nIndividual test cases may add additional dependencies.  \r\n\r\n## The ProcessManager class:\r\n\r\nThe ProcessManager class can be found in tests/framework/atstf.py.  It is responsible for configuring, starting, and stopping the processes used by a test suite.\r\n\r\n### ProcessManager Configuration\r\n\r\nThe ProcessManager is driven by a JSON config file that describes all managed processes.   Individual test cases can  also read this config file to discover server ports, hostnames, etc.   As long as test cases do not hardcode these values they can later be turned against an external functional test environment.\r\n\r\n#### config.json path\r\n\r\nBy default the ProcessManager looks for a 'config.json' file in the test suite's current working directory.  An alternate location can be passed to its constructor:\r\n\r\n```python\r\ntm = atstf.ProcessManager(config_path=\"/tmp/config.json\")\r\n```\r\n\r\n#### trafficserver and other paths\r\n\r\nTest suites inside the trafficserver source repo only have to tell the ProcessManager the path to the source repo root and it will infer the rest:\r\n```python\r\ntm = atstf.ProcessManager(root_path=\"../..\")\r\n```\r\n\r\nTest suites outside the trafficserver source repo have to explicitly set each of the following paths:\r\n```python\r\ntm = atstf.ProcessManager(ats_path=\"/usr/local/bin/trafficserver\", \r\n                          default_config_path=\"~/trafficserver/proxy/config\",\r\n                          origin_path=\"~/trafficserver/tests/framework/origin.py\")\r\n```\r\n\r\nThe last two paths (default_config_path and origin_path) are paths into a local trafficserver source repo, but we can potentially incorporate those into a 'devel' package in the future.\r\n\r\n#### logging:\r\n\r\nYou can tell the ProcessManager to log stuff including the stdout and stderr of the processes it manages.  The ProcessManager uses python's [logging module](https://docs.python.org/2/library/logging.html).   If you configure python's logging module outside of ProcessManager, the ProcessManager will just use that config.   If you don't want to configure the logging module you can tell the ProcessManager to do it for you by passing it a log_level (e.g., logging.INFO):\r\n\r\n```python\r\nimport logging\r\n\r\ntm = atstf.ProcessManager(log_level=logging.DEBUG)\r\n```\r\n\r\n#### max start time\r\n\r\nBy default the ProcessManager will wait for up to 5 seconds for the processes it manages to start.   If any managed process does not start in time, the ProcessManager will throw an exception which should fail the test.\r\n\r\nThe ProcessManager considers a managed process to have started when it can successfully connect to all ports listed in the process's 'interfaces' section.  More on this later.\r\n\r\nTo change the max start time:\r\n```python\r\ntm = atstf.ProcessManager(max_start_sec=3)\r\n```\r\n\r\n#### The config.json\r\n\r\nThe JSON configuration file used by the ProcessManager has the following common format:\r\n\r\n```\r\n{\r\n    \"processes\": {\r\n        <process name>: {\r\n            \"type\": <\"ats\"|\"origin\"|\"python\"|\"other\">,\r\n            \"spawn\": <true|false>\r\n            \"interfaces\": {\r\n                <interface name>: {\r\n                    \"hostname\": <hostname|ipv4 addr|ipv6 addr>,\r\n                    \"port\": <port>\r\n                }\r\n            },\r\n            <other stuff specific to the process type>\r\n        }\r\n    },\r\n    <optional: other stuff specific to the test suite.  Ignored by ProcessManager>\r\n}\r\n```\r\n\r\nBecause this must be valid JSON, process names must be unique.   They will be incorporated into log messages and into exceptions thrown by the ProcessManager.\r\n\r\nThe process types supported by the ProcessManager are described below.   Different process types may support additional configuration options.\r\n\r\nThe 'spawn' attribute defaults to true.   If set to false the ProcessManager will not manage the process.  This is intended to support turning a functional test case against an externally managed set of processes.   The test case code can still discover hostnames and ports from the config.json without a code change.\r\n\r\nThe optional 'interfaces' attribute details which ports will be opened by a managed process.  If specified the ProcessManager will block caller until either all ports are up or the 'max start time' is exceeded.\r\n\r\n#### The config.json Process Types\r\n\r\nThe ProcessManager currently supports the following process types:\r\n\r\n##### Process Type 'ats'\r\n\r\n'ats' processes support additional config.json options.\r\n\r\nExample:\r\n```json\r\n{\r\n    \"servers\": {\r\n        \"ats1\": {\r\n            \"type\": \"ats\",\r\n            \"root\": \"ats1\",\r\n            \"interfaces\": {\r\n                \"http\": {\r\n                    \"type\": \"http\",\r\n                    \"hostname\": \"localhost\",\r\n                    \"port\": 8080\r\n                }\r\n            },\r\n            \"config\": {\r\n                \"records.config\": [\r\n                    {\r\n                        \"match\": \"^CONFIG proxy.config.diags.debug.enabled\",\r\n                        \"substitute\": \"CONFIG proxy.config.diags.debug.enabled INT 1\"\r\n                    }\r\n                ],\r\n                \"remap.config\": [\r\n                    {\r\n                        \"append\": \"map /foo/bar http://localhost:8082/foo/bar\"\r\n                    }\r\n                ]\r\n            }\r\n        }\r\n    }\r\n}\r\n```\r\n\r\nLIMITATION: Only the 'http' interface type is presently supported.   Need to add HTTPS and SPDY ASAP.\r\n\r\nThe ProcessManager will run the trafficserver in the 'root' dir.   This is the same as the trafficserver 'TS_ROOT' environment variable.   If you configure multiple 'ats' processes, be sure to assign each of them a distinct root dir.   The root dir should be relative to the test case dir and MUST NOT CONTAIN ANYTHING IMPORTANT!   If ats crashes and the system is configured to capture cores, they should appear in the root dir.\r\n\r\nBefore trafficserver is started:\r\n\r\n   * The ProcessManager will rm -rf the 'root' dir if it already exists, then recreate it with a minimal directory structure (a few var dirs and an etc dir)\r\n   * The ProcessManager will copy over all the default config files and body factory files from proxy/config\r\n   * If the config.json has a 'config' section, the ProcessManager will transform the copied default config files accordingly.   On a per config file basis:\r\n      * match/substitue directives will replace any line that matches a regex (match) with a substitution line\r\n      * append directives will simply add lines to the end of the config file.\r\n      \r\nAfter trafficserver is started:\r\n\r\n   * trafficserver will create transient files in the root/var dirs, including log files\r\n\r\n##### Process Type 'origin'\r\n\r\nA configurable origin server.   Multiple types of 'origins' can be specified on a per-path basis, but currently only a 'chunkedresponse' type has been developed.   The 'chunkedresponse' origin type supports the following configuration:\r\n\r\n   1. response status code\r\n   1. response headers\r\n   1. number of chunks to send in the response\r\n   1. size of each chunk\r\n   1. byte value of each byte in chunk\r\n   1. seconds to wait before sending the first chunk\r\n   1. seconds to wait before sending all subsequent chunks\r\n\r\nExample:\r\n```json\r\n{\r\n    \"servers\": {\r\n        \"origin1\": {\r\n            \"type\": \"origin\",\r\n            \"interfaces\": {\r\n                \"http\": {\r\n                    \"type\": \"http\",\r\n                    \"hostname\": \"localhost\",\r\n                    \"port\": 8082\r\n                }\r\n            },\r\n            \"actions\": {\r\n                \"GET\": {\r\n                    \"/foo/bar\": {\r\n                        \"type\": \"chunkedresponse\",\r\n                        \"status_code\": 200,\r\n                        \"headers\": {\r\n                            \"content-type\": \"text/plain\"\r\n                        },\r\n                        \"delay_first_chunk_sec\": 1,\r\n                        \"chunk_size_bytes\": 1024,\r\n                        \"num_chunks\": 10,\r\n                        \"delay_between_chunk_sec\": 0,\r\n                        \"chunk_byte_value\": 99\r\n                    }\r\n                }\r\n            }\r\n        }\r\n    }\r\n}\r\n```\r\n\r\n##### Process Type 'python'\r\n\r\nThe ProcessManager can manage ad hoc origins written in python providing they exit cleanly when sent a SIGTERM.   If they specify an 'interfaces' section then the ProcessManager will also wait for them to start.\r\n\r\nExample:\r\n```json\r\n{\r\n    \"servers\": {\r\n        \"python1\": {\r\n            \"type\": \"python\",\r\n            \"script\": \"ad_hoc_origin.py\",\r\n            \"args\": [\r\n                \"python1\",\r\n                \"config.json\"\r\n            ],\r\n            \"interfaces\": {\r\n                \"http\": {\r\n                    \"type\": \"http\",\r\n                    \"hostname\": \"localhost\",\r\n                    \"port\": 8082\r\n                }\r\n            }\r\n        }\r\n    }\r\n}\r\n```\r\n\r\n##### Process Type 'other'\r\n\r\nFinally the ProcessManager can manage any executable, again providing it exits cleanly when sent a SIGTERM.  If it specifies an 'interface' secion then the ProcessManager will also wait for it to start.\r\n\r\nExample:\r\n```json\r\n{\r\n    \"servers\": {\r\n        \"other1\": {\r\n            \"type\": \"other\",\r\n            \"spawn\": false,\r\n            \"executable\": \"/usr/local/apache2/bin/httpd\",\r\n            \"args\": [\r\n                \"-f\",\r\n                \"/usr/local/apache2/conf/httpd.conf\",\r\n                \"-k\",\r\n                \"start\",\r\n                \"-x\"\r\n            ],\r\n            \"interfaces\": {\r\n                \"http\": {\r\n                    \"type\": \"http\",\r\n                    \"hostname\": \"localhost\",\r\n                    \"port\": 8082\r\n                }\r\n            }\r\n        }\r\n    }\r\n}\r\n```\r\n\r\n## Writing Test Cases\r\n\r\n### Using the 'unittest' and 'twisted' modules\r\n\r\nTest cases should use [the Python Standard Library's unittest module](https://docs.python.org/2/library/unittest.html). Test cases also need to send traffic to trafficserver.   This particular example uses the [Python twisted framework](https://twistedmatrix.com/trac/), but other approaches are possible.  \r\n\r\nThe main advantage of twisted is that it can be used for lightweight load testing.   The main disadvantage is that Python's unittest module implements assertions on top of exceptions.   Failed assertions in twisted callbacks will be eaten by the twisted framework and will not be incorporated into the test results.\r\n\r\nThis example works around this issue by saving the state of its interaction with trafficserver, then performing all assertions in the main thread against the saved state at the end of the test case:\r\n\r\n```python\r\nimport sys\r\nimport unittest\r\n\r\nfrom twisted.internet import reactor\r\nfrom twisted.internet.defer import Deferred\r\nfrom twisted.internet.protocol import Protocol\r\nfrom twisted.web.client import Agent\r\nfrom twisted.web.http_headers import Headers\r\n\r\nsys.path = ['../framework'] + sys.path\r\nimport atstf\r\n\r\nclass ExampleTwistedTest(unittest.TestCase):\r\n    def test_ats1(self):\r\n        #\r\n        # Read test case configuration from the same config file used to start ATS and origin processes.\r\n        #\r\n\r\n        conf = atstf.parse_config()\r\n\r\n        ats1_conf = conf['processes']['ats1']\r\n\r\n        hostname = ats1_conf['interfaces']['http']['hostname']\r\n        port = ats1_conf['interfaces']['http']['port']\r\n\r\n        origin_conf = conf['processes']['origin1']\r\n\r\n        expected_status_code = origin_conf['actions']['GET']['/foo/bar']['status_code']\r\n        chunk_size_bytes = origin_conf['actions']['GET']['/foo/bar']['chunk_size_bytes']\r\n        num_chunks = origin_conf['actions']['GET']['/foo/bar']['num_chunks']\r\n        expected_bytes_received = chunk_size_bytes * num_chunks\r\n\r\n        #\r\n        # Create a twisted HTTP client that will interact with ATS and save all details of the interaction.   We'll\r\n        # later assert that the saved details are correct.   Ideally we'd just throw assertions into the client itself,\r\n        # but sadly assertions are implemented with exceptions and twisted eats any exception raised by our callbacks.\r\n        #\r\n        # See https://twistedmatrix.com/documents/current/web/howto/client.html for details on writing HTTP clients\r\n        #\r\n\r\n        agent = Agent(reactor)\r\n\r\n        request = agent.request('GET',  \"http://%s:%d/foo/bar\" % (hostname.encode(\"utf-8\"), port),\r\n                                Headers({'User-Agent': ['ExampleTwistedTest']}), None)\r\n\r\n        response_handler = ResponseHandler(reactor)\r\n\r\n        request.addCallback(response_handler.handle_response)\r\n        request.addCallback(response_handler.handle_completion)\r\n        request.addErrback(response_handler.handle_error)\r\n\r\n        reactor.run()\r\n\r\n        #\r\n        # Now assert\r\n        #\r\n\r\n        self.assertEqual(expected_status_code, response_handler.get_status_code())\r\n        self.assertEqual(expected_bytes_received, response_handler.get_body_handler().get_bytes_received())\r\n        self.assertEqual(\"Response body fully received\", \\\r\n                         response_handler.get_body_handler().get_reason().getErrorMessage())\r\n\r\nclass ResponseBodyHandler(Protocol):\r\n    def __init__(self, deferred):\r\n        self.__deferred = deferred\r\n        self.__bytes_received = 0\r\n        self.__reason = None\r\n\r\n    def dataReceived(self, data):\r\n        self.__bytes_received += len(data)\r\n\r\n    def connectionLost(self, reason):\r\n        #\r\n        # This function name is highly misleading.  According to the web client docs:\r\n        #\r\n        # When the body has been completely delivered, the protocol's connectionLost method is called. It is important\r\n        # to inspect the Failure passed to connectionLost . If the response body has been completely received, the\r\n        # failure will wrap a twisted.web.client.ResponseDone exception. This indicates that it is known that all data\r\n        # has been received. It is also possible for the failure to wrap a twisted.web.http.PotentialDataLoss exception:\r\n        # this indicates that the server framed the response such that there is no way to know when the entire response\r\n        # body has been received. Only HTTP/1.0 servers should behave this way. Finally, it is possible for the\r\n        # exception to be of another type, indicating guaranteed data loss for some reason (a lost connection, a memory\r\n        # error, etc).\r\n        #\r\n        self.__reason = reason\r\n        self.__deferred.callback(None)\r\n\r\n    def get_bytes_received(self):\r\n        return self.__bytes_received\r\n\r\n    def get_reason(self):\r\n        return self.__reason\r\n\r\nclass ResponseHandler:\r\n    def __init__(self, reactor):\r\n        self.__reactor = reactor\r\n        self.__deferred = Deferred()\r\n        self.__body_handler = ResponseBodyHandler(self.__deferred)\r\n        self.__error = None\r\n        self.__status_code = None\r\n\r\n    def handle_response(self, response):\r\n        self.__status_code = response.code\r\n        response.deliverBody(self.__body_handler)\r\n        return self.__deferred\r\n\r\n    def handle_completion(self):\r\n        self.__reactor.stop()\r\n\r\n    def handle_error(self, error):\r\n        self.__error = error\r\n        self.__reactor.stop()\r\n\r\n    def get_body_handler(self):\r\n        return self.__body_handler\r\n\r\n    def get_error(self):\r\n        return self.__error\r\n\r\n    def get_status_code(self):\r\n        return self.__status_code\r\n```\r\n\r\n\r\n### Pulling it all together with a Test Suite\r\n\r\nMultiple test cases should be run together in a test suite and the results of the entire suite should be summarized in a JUnit report suitable for Jenkins.  This is also a good place to instantiate and start the ProcessManager (start it before any test case is invoked, stop it after all test cases finish).   \r\n\r\nExample:\r\n```python\r\n#!/bin/env python\r\n\r\nimport sys\r\nimport logging\r\n\r\nsys.path = ['../framework'] + sys.path\r\nimport atstf\r\n\r\nfrom twisted_test import ExampleTwistedTest\r\n\r\ntest_cases = [ExampleTwistedTest]\r\n\r\nif __name__ == '__main__':\r\n    # Spawn ATS and origin processes\r\n\r\n    tm = atstf.ProcessManager(root_path=\"../..\", config_path=\"config.json\", log_level=logging.DEBUG, max_start_sec=5)\r\n\r\n    try:\r\n        tm.start()\r\n\r\n        # Run the test suite and generate a JUnit XML report\r\n\r\n        if atstf.run_tests(test_cases=test_cases, name='example', report='report.xml'):\r\n            sys.exit(0)\r\n        else:\r\n            sys.exit(1)\r\n    finally:\r\n        tm.stop()\r\n```\r\n\r\nHere the important bits are:\r\n\r\n   1. Initializing the ProcessManager and starting it before the running the test suite\r\n   1. Passing all the test case classes to run_tests().   This will auto discover all test cases in the test classes, run them, and spit the results out in a report.xml file in the cwd.\r\n   1. Exiting zero if the run_tests() method returns True, non-zero otherwise\r\n   \r\nNot so important is explicitly stopping the ProcessManager.   It's good practice, but the ProcessManager will stop automatically on process exit.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/trafficserver/issues/comments/112871149","html_url":"https://github.com/apache/trafficserver/pull/141#issuecomment-112871149","issue_url":"https://api.github.com/repos/apache/trafficserver/issues/141","id":112871149,"user":{"login":"zwoop","id":202131,"avatar_url":"https://avatars.githubusercontent.com/u/202131?v=3","gravatar_id":"","url":"https://api.github.com/users/zwoop","html_url":"https://github.com/zwoop","followers_url":"https://api.github.com/users/zwoop/followers","following_url":"https://api.github.com/users/zwoop/following{/other_user}","gists_url":"https://api.github.com/users/zwoop/gists{/gist_id}","starred_url":"https://api.github.com/users/zwoop/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zwoop/subscriptions","organizations_url":"https://api.github.com/users/zwoop/orgs","repos_url":"https://api.github.com/users/zwoop/repos","events_url":"https://api.github.com/users/zwoop/events{/privacy}","received_events_url":"https://api.github.com/users/zwoop/received_events","type":"User","site_admin":false},"created_at":"2015-06-17T16:36:29Z","updated_at":"2015-06-17T16:36:29Z","body":"We've landed LinkedIn's framework, can you please close this pull request?"}}