{"action":"created","issue":{"url":"https://api.github.com/repos/jasonish/py-idstools/issues/19","labels_url":"https://api.github.com/repos/jasonish/py-idstools/issues/19/labels{/name}","comments_url":"https://api.github.com/repos/jasonish/py-idstools/issues/19/comments","events_url":"https://api.github.com/repos/jasonish/py-idstools/issues/19/events","html_url":"https://github.com/jasonish/py-idstools/issues/19","id":92224610,"number":19,"title":"Encoding issue crash","user":{"login":"djtecha","id":3084627,"avatar_url":"https://avatars.githubusercontent.com/u/3084627?v=3","gravatar_id":"","url":"https://api.github.com/users/djtecha","html_url":"https://github.com/djtecha","followers_url":"https://api.github.com/users/djtecha/followers","following_url":"https://api.github.com/users/djtecha/following{/other_user}","gists_url":"https://api.github.com/users/djtecha/gists{/gist_id}","starred_url":"https://api.github.com/users/djtecha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/djtecha/subscriptions","organizations_url":"https://api.github.com/users/djtecha/orgs","repos_url":"https://api.github.com/users/djtecha/repos","events_url":"https://api.github.com/users/djtecha/events{/privacy}","received_events_url":"https://api.github.com/users/djtecha/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-06-30T23:31:25Z","updated_at":"2015-07-03T13:01:13Z","closed_at":null,"body":"Some rule in snort 2.9.7.2-1 is creating a breaking case in the python module:\r\n\r\nINFO: Loaded 25034 rule message map entries.\r\nINFO: Loaded 38 classifications.\r\nTraceback (most recent call last):\r\n  File \"py-idstools/bin/idstools-u2json\", line 12, in <module>\r\n    sys.exit(main())\r\n  File \"/root/py-idstools/idstools/scripts/u2json.py\", line 308, in main\r\n    as_json = json.dumps(formatter.format(record))\r\n  File \"/usr/lib64/python2.7/json/__init__.py\", line 243, in dumps\r\n    return _default_encoder.encode(obj)\r\n  File \"/usr/lib64/python2.7/json/encoder.py\", line 207, in encode\r\n    chunks = self.iterencode(o, _one_shot=True)\r\n  File \"/usr/lib64/python2.7/json/encoder.py\", line 270, in iterencode\r\n    return _iterencode(o, 0)\r\nUnicodeDecodeError: 'utf8' codec can't decode byte 0xec in position 633: invalid continuation byte\r\n\r\n\r\nUsing the latest bits. If I clear out the unified log and restart it it will work until this sequence is hit again."},"comment":{"url":"https://api.github.com/repos/jasonish/py-idstools/issues/comments/118342381","html_url":"https://github.com/jasonish/py-idstools/issues/19#issuecomment-118342381","issue_url":"https://api.github.com/repos/jasonish/py-idstools/issues/19","id":118342381,"user":{"login":"jasonish","id":644115,"avatar_url":"https://avatars.githubusercontent.com/u/644115?v=3","gravatar_id":"","url":"https://api.github.com/users/jasonish","html_url":"https://github.com/jasonish","followers_url":"https://api.github.com/users/jasonish/followers","following_url":"https://api.github.com/users/jasonish/following{/other_user}","gists_url":"https://api.github.com/users/jasonish/gists{/gist_id}","starred_url":"https://api.github.com/users/jasonish/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jasonish/subscriptions","organizations_url":"https://api.github.com/users/jasonish/orgs","repos_url":"https://api.github.com/users/jasonish/repos","events_url":"https://api.github.com/users/jasonish/events{/privacy}","received_events_url":"https://api.github.com/users/jasonish/received_events","type":"User","site_admin":false},"created_at":"2015-07-03T13:01:13Z","updated_at":"2015-07-03T13:01:13Z","body":"> I was also wondering if there is a way to merge the event with the packet\r\n> data into 1 JSON blob. I enjoy the modularity offered at the moment, but it\r\n> requires some interesting hacks to get them to show up together as an\r\n> elasticsearch event.\r\n\r\nYes, check out idstools-u2eve.  It outputs a JSON blob that combines the event record and the first associated packet which may be useful to you.\r\n\r\nBut otherwise, no I don't have a way to merge the event, all the packets and extra data that may be associated with an event.  This is because the order of the records in the file may not be correct, at least for packets.  For example you may have an event record, packet record, another event record, and packet record, then a packet record for the previous event record.\r\n\r\nSo to avoid the buffering, and protential delay of getting the event out, I didn't want to deal with it in a generic way.\r\n\r\nI'll look at the other issues in the next few days likely."}}