{"action":"created","issue":{"url":"https://api.github.com/repos/IBMStreams/administration/issues/69","labels_url":"https://api.github.com/repos/IBMStreams/administration/issues/69/labels{/name}","comments_url":"https://api.github.com/repos/IBMStreams/administration/issues/69/comments","events_url":"https://api.github.com/repos/IBMStreams/administration/issues/69/events","html_url":"https://github.com/IBMStreams/administration/issues/69","id":101751267,"number":69,"title":"Proposal for a new project: streamsx.rstm ","user":{"login":"rzbhatti","id":11372230,"avatar_url":"https://avatars.githubusercontent.com/u/11372230?v=3","gravatar_id":"","url":"https://api.github.com/users/rzbhatti","html_url":"https://github.com/rzbhatti","followers_url":"https://api.github.com/users/rzbhatti/followers","following_url":"https://api.github.com/users/rzbhatti/following{/other_user}","gists_url":"https://api.github.com/users/rzbhatti/gists{/gist_id}","starred_url":"https://api.github.com/users/rzbhatti/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rzbhatti/subscriptions","organizations_url":"https://api.github.com/users/rzbhatti/orgs","repos_url":"https://api.github.com/users/rzbhatti/repos","events_url":"https://api.github.com/users/rzbhatti/events{/privacy}","received_events_url":"https://api.github.com/users/rzbhatti/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-08-18T21:18:40Z","updated_at":"2015-08-19T22:39:07Z","closed_at":null,"body":"### Introduction\r\nA lightweight text search and match toolkit for high performance streaming applications is proposed here. The toolkit uses a **reduced state transition matrix (rstm)** technique (described below) for scalable and dynamic runtime operation. It is pertinent to most common enterprise scale real-time, Big Data streaming applications like cyber security, telco, digital content providers, and log analytics applications etc.      \r\n### Motivation\r\nIn a typical scenario a stream containing a text (rstring) field is filtered, searched or matched against a **set of target patterns**. Where the set of patterns may be one or mixture of the following:\r\n\r\n1. Partial keywords\r\n2. Begin patterns\r\n3. End patterns\r\n4. General regular expressions\r\n5. Binary data patterns\r\n\r\nIt may be very obvious that a simple loop implantation of partial string or regular expression matching, over a set of target patterns, is not a scalable solution. The performance of this sort of operation becomes highly dependent on the size of the target set of patters. A streams operator, implemented like this, working with a large set of target patterns could clearly be bottleneck. If the given set of target patterns is pre-compiled into a custom primitive streams operator, then removing a pattern from or adding to the pre-compiled operator may not be possible in run time. \r\n\r\n### Proposed Streams Toolkit based on “rstm”\r\nIn the proposed solution, the given set of target patterns is compiled into a reduced state transition matrix (rstm) binary. This rstm binary is dynamically loadable by the rstm runtime scanner. The performance of the run time remains independent of the size of the set of target patterns. The toolkit provides following:\r\n\r\n1. Compiler script to generate rstm binary file from a given set of target patterns\r\n  * uses  [ragel](https://www.colm.net/open-source/ragel) an open source finite state machine compiler to compile rstm tables  \r\n2. A C++ Primitive RSTM operator. Core rstm run time implementation  \r\n3. SPL composites. The following composites use the core rstm operator in different configurations  \r\n  * RstmKeywords\r\n  * RstmRegex\r\n  * RstmBeginPattern   \r\n  * RstmEndPattern   \r\n  * RstmBinaryPattern   \r\n\r\n### Understanding “rstm”\r\nA given set of target patterns is compiled into a deterministic finite automat (DFA) using a finite state machine (FSM) compiler (like [re2c](http://re2c.org/manual.html)  or  [ragel](https://www.colm.net/open-source/ragel)). The FSM is converted into an rstm binary for dynamic loading and high performance runtime data scanning.\r\n\r\n### Example\r\nWhitelisting DNS requests in a malicious url detection application. Top 500 domain names from [Alexa.com](http://www.alexa.com/topsites) are used to create a set of whitelisted domain URLs. The domain field of the incoming TDR tuples is matched. \r\n\r\n    namespace sample ;\r\n    use com.ibm.streamsx.rstm::RstmRegexMatch ;\r\n    \r\n    type testDataType = \r\n        rstring protocol,\r\n        rstring ip,\r\n        rstring dest_port,\r\n        rstring source_port,\r\n        rstring user_agent,\r\n        rstring host,\r\n        rstring uri;\r\n    \r\n    type testMatchOutputType = \r\n        rstring protocol,\r\n        rstring ip,\r\n        rstring dest_port,\r\n        rstring source_port,\r\n        rstring user_agent,\r\n        rstring host,\r\n        rstring uri,\r\n        boolean matchFound,\r\n        list<int32> matchIndices,\r\n        list<int32> matchOffsets,\r\n        list<rstring> matchTargets;\r\n    \r\n    composite regexAlexaWhiteList\r\n    {\r\n        graph\r\n            (stream<testDataType> httpMetaData) as FSource = FileSource()\r\n            {\r\n                param\r\n                format : csv ;\r\n              file : \"testData.csv\" ;\r\n            }\r\n    \r\n        (stream<testMatchOutputType> httpMetaDataOut) as RSTM = RstmRegexMatch(httpMetaData)\r\n            {\r\n                param\r\n                targetString : host ;\r\n              rstmFileName : \"alexaWhiteList.rstm.bin\" ;\r\n              regexFileName : \"alexaWhiteList.pcre\" ;\r\n              rstmUpdateCheckInterval : 5.0 ;\r\n                output\r\n                httpMetaDataOut :matchFound = matchFound(), matchIndices = matchIndices(),\r\n                matchTargets = matchTargets(), matchOffsets = matchOffsets() ;\r\n            }\r\n    \r\n        () as Sink = FileSink(httpMetaDataOut)\r\n            {\r\n                param\r\n                file : \"outPutFile.csv\" ;\r\n              format : csv ;\r\n            }\r\n    }\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/IBMStreams/administration/issues/comments/132812109","html_url":"https://github.com/IBMStreams/administration/issues/69#issuecomment-132812109","issue_url":"https://api.github.com/repos/IBMStreams/administration/issues/69","id":132812109,"user":{"login":"mikespicer","id":6910967,"avatar_url":"https://avatars.githubusercontent.com/u/6910967?v=3","gravatar_id":"","url":"https://api.github.com/users/mikespicer","html_url":"https://github.com/mikespicer","followers_url":"https://api.github.com/users/mikespicer/followers","following_url":"https://api.github.com/users/mikespicer/following{/other_user}","gists_url":"https://api.github.com/users/mikespicer/gists{/gist_id}","starred_url":"https://api.github.com/users/mikespicer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikespicer/subscriptions","organizations_url":"https://api.github.com/users/mikespicer/orgs","repos_url":"https://api.github.com/users/mikespicer/repos","events_url":"https://api.github.com/users/mikespicer/events{/privacy}","received_events_url":"https://api.github.com/users/mikespicer/received_events","type":"User","site_admin":false},"created_at":"2015-08-19T22:39:07Z","updated_at":"2015-08-19T22:39:07Z","body":"I certainly like the idea of the toolkit but want to confirm that you have appropriate approvals from IBM to release the code and are handling interactions with GPL code appropriately."}}