{"action":"created","issue":{"url":"https://api.github.com/repos/certtools/intelmq/issues/185","labels_url":"https://api.github.com/repos/certtools/intelmq/issues/185/labels{/name}","comments_url":"https://api.github.com/repos/certtools/intelmq/issues/185/comments","events_url":"https://api.github.com/repos/certtools/intelmq/issues/185/events","html_url":"https://github.com/certtools/intelmq/issues/185","id":81877174,"number":185,"title":"DataHarmonization Document improvements","user":{"login":"Phantasus","id":1078087,"avatar_url":"https://avatars.githubusercontent.com/u/1078087?v=3","gravatar_id":"","url":"https://api.github.com/users/Phantasus","html_url":"https://github.com/Phantasus","followers_url":"https://api.github.com/users/Phantasus/followers","following_url":"https://api.github.com/users/Phantasus/following{/other_user}","gists_url":"https://api.github.com/users/Phantasus/gists{/gist_id}","starred_url":"https://api.github.com/users/Phantasus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Phantasus/subscriptions","organizations_url":"https://api.github.com/users/Phantasus/orgs","repos_url":"https://api.github.com/users/Phantasus/repos","events_url":"https://api.github.com/users/Phantasus/events{/privacy}","received_events_url":"https://api.github.com/users/Phantasus/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/certtools/intelmq/labels/documentation","name":"documentation","color":"5319e7"},{"url":"https://api.github.com/repos/certtools/intelmq/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":16,"created_at":"2015-05-28T10:58:32Z","updated_at":"2015-08-24T15:23:55Z","closed_at":null,"body":"Hi,\r\n\r\nthe DataHarmonization document needs some improvements:\r\n\r\n1. When looking at it in a browser the table is too big to be actually displayed in a browser window.\r\n2. The document itself isn't so clearly understandable. e.g When to prefix a key with reported? When not? How is someone supposed to add a new field? Like http method? Should that then be reported_http_method?\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/certtools/intelmq/issues/comments/134247145","html_url":"https://github.com/certtools/intelmq/issues/185#issuecomment-134247145","issue_url":"https://api.github.com/repos/certtools/intelmq/issues/185","id":134247145,"user":{"login":"aaronkaplan","id":750019,"avatar_url":"https://avatars.githubusercontent.com/u/750019?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronkaplan","html_url":"https://github.com/aaronkaplan","followers_url":"https://api.github.com/users/aaronkaplan/followers","following_url":"https://api.github.com/users/aaronkaplan/following{/other_user}","gists_url":"https://api.github.com/users/aaronkaplan/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronkaplan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronkaplan/subscriptions","organizations_url":"https://api.github.com/users/aaronkaplan/orgs","repos_url":"https://api.github.com/users/aaronkaplan/repos","events_url":"https://api.github.com/users/aaronkaplan/events{/privacy}","received_events_url":"https://api.github.com/users/aaronkaplan/received_events","type":"User","site_admin":false},"created_at":"2015-08-24T15:23:55Z","updated_at":"2015-08-24T15:23:55Z","body":"Sebix and me did that on FR. Please contact him for details. \n\n---\nMobile\n\n\n> On 24.08.2015, at 13:48, Tomás Lima <notifications@github.com> wrote:\n> \n> On Mon, Aug 24, 2015 at 11:57 AM, Sebastian <notifications@github.com>\n> wrote:\n> \n> > On 08/24/2015 12:08 PM, Tomás Lima wrote:\n> > >\n> > > @sebix <https://github.com/sebix> , cymru_cc or geoip_cc doesnt exist.\n> > > All current fields are here:\n> > >\n> > https://github.com/certtools/intelmq/blob/master/intelmq/conf/harmonization.conf\n> > >\n> > These are the implemented fields, the theoretically standardized fields\n> > are here: https://github.com/certtools/intelmq/wiki/Data-Harmonization\n> > Yes, they are not congruent, therefore we have this discussion trying to\n> > unify them and re-standardize the fields again.\n> >\n> \n> Some fields are deprecated, other just were updated...well... like I\n> mentioned, I think we need to get the description of most of the fields and\n> put them in harmonization.conf. Then, we can create a quick script for\n> convert the harmonization.conf into a harmonization.md that we can put on\n> docs folder. Yes, discuss and propose something, then we will try to\n> improve together.\n> \n> \n> > >\n> > > I dont see the need to have multiple fields for that... in the\n> > > pipeline we can put the bots cymru and maxmind in the way we want...\n> > >\n> > AFAIK the idea was/is to use more than one data source for geolocation\n> > as they do not give identical results and geolocation is no exact\n> > science. Having two independent localizations probably results in more\n> > accurate location information.\n> >\n> \n> True, but have multiple fields related to localization seems to be\n> overwhelmed.\n> \n> \n> > >\n> > > cymru -> maxmind = means, if maxmind has a value, will overwrite cymru\n> > > maxmind -> cymru = means, if cymru has a value, will overwrite maxmind\n> > >\n> > What do you want to say? Is this a proposal for configuration?\n> >\n> \n> No, Im just presenting the possibilities depending of your confidence in\n> each source. For example, If you \"trust\" more in cymru, put cymru bot after\n> maxmind and Cymru bot will overwrite the geolocation information...\n> \n> \n> > >\n> > > In my perspective, I think we should minimize the number of fields\n> > > that does the same thing... but @aaronkaplan\n> > > <https://github.com/aaronkaplan> and @sebix <https://github.com/sebix>\n> > > , whats your feedback?\n> > >\n> > Right. More empty fields are overhead. But they could probably be handy\n> > to filter data more easily (instead of searching for them in\n> > `additional_data`. I'm not directly involved in the standardization\n> > process itself, just can give hints and raise helpful questions.\n> \n> \n> No, Im not suggesting put them in 'description.<something>' (old\n> 'additional_data'). Im just trying to keep the fields that are important\n> and remove the unnecessary.\n> —\n> Reply to this email directly or view it on GitHub.\n> \n"}}