{"action":"created","issue":{"url":"https://api.github.com/repos/lxa2015/lxa-py/issues/7","labels_url":"https://api.github.com/repos/lxa2015/lxa-py/issues/7/labels{/name}","comments_url":"https://api.github.com/repos/lxa2015/lxa-py/issues/7/comments","events_url":"https://api.github.com/repos/lxa2015/lxa-py/issues/7/events","html_url":"https://github.com/lxa2015/lxa-py/issues/7","id":107785027,"number":7,"title":"standardise JSON output","user":{"login":"notnami","id":1631585,"avatar_url":"https://avatars.githubusercontent.com/u/1631585?v=3","gravatar_id":"","url":"https://api.github.com/users/notnami","html_url":"https://github.com/notnami","followers_url":"https://api.github.com/users/notnami/followers","following_url":"https://api.github.com/users/notnami/following{/other_user}","gists_url":"https://api.github.com/users/notnami/gists{/gist_id}","starred_url":"https://api.github.com/users/notnami/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/notnami/subscriptions","organizations_url":"https://api.github.com/users/notnami/orgs","repos_url":"https://api.github.com/users/notnami/repos","events_url":"https://api.github.com/users/notnami/events{/privacy}","received_events_url":"https://api.github.com/users/notnami/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/lxa2015/lxa-py/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-09-22T19:35:42Z","updated_at":"2015-09-23T18:10:23Z","closed_at":null,"body":"As @JacksonLLee probably knows, the way linguistica currently serialises its results into JSON is somewhat ad-hoc. In most cases, we just create string representations of Python objects and write those into JSON as-is. Although this may be a convenient temporary option while we concentrate on other, more conceptually challenging parts of linguistica, this approach removes most benefits of using JSON: JSON works best as a portable, language-agnostic format, which can help us use linguistica's results across a variety of other programs (which may be written in Python, C++, or any other language). However, with the current approach we generate output that can only be deserialised with Python and, more importantly, may vary dramatically based on the kind of data that output represents given our undocumented internal conventions. \r\n\r\nI think that the sooner we begin to standardise our output, the less 'tech-debt' we will have later and the easier it will be to fix whatever issues may arise with the eventual standardisation. (It so happens that my part of the project depends on working with linguistica's output, so I have my own vested interest in making that output easier to process).\r\n\r\nI see the work as consisting roughly of 4 parts:\r\n\r\n1. Make a list of every kind of object that linguistica currently puts into JSON so that we may come up with the best way of representing it in the JSON standard. This brings me to the following question: If I run linguistica on, say, the Brown corpus, would the JSON files generated as a result of that contain every kind of object that we aim to cover in the foreseeable future? That is, do we expect to be serialising any other objects not included in the output of the current version?\r\n\r\n2. Create a sort of wrapper library which would be responsible for reading and writing our new standardised JSON output. We would be able to call this library from linguistica pretty much the same way we call the json_pload and json_pdump functions now — except on the inside this library will make sure that the JSON output is readable, portable, and consistent. I believe that we should write a function for serialising JSON first — that way, we reduce the possibility that something breaks in any of linguistica's critical parts. Also — please correct me if I'm wrong — other than for the config file, linguistica itself does not depend on reading JSON, so I don't see improving the reading of JSON as a current priority.\r\n\r\n3. Test that everything works, and that the main parts of linguistica continue to work as before. \r\n\r\n4. Describe and document the resulting output format as part of this project's ReadMe so that anyone using linguistica could work with its output without having to go deeper into how our code actually works.\r\n\r\nIf @JacksonLLee and everyone else considers this a worthwhile cause, I can start working on this. Despite the length of this issue post, I don't see this part of the project as particularly complicated, so I should be able to complete it by myself without taking too long. Of course, I would welcome suggestions and any potential help from other members of our group, so let me know what you think."},"comment":{"url":"https://api.github.com/repos/lxa2015/lxa-py/issues/comments/142684046","html_url":"https://github.com/lxa2015/lxa-py/issues/7#issuecomment-142684046","issue_url":"https://api.github.com/repos/lxa2015/lxa-py/issues/7","id":142684046,"user":{"login":"JacksonLLee","id":6993153,"avatar_url":"https://avatars.githubusercontent.com/u/6993153?v=3","gravatar_id":"","url":"https://api.github.com/users/JacksonLLee","html_url":"https://github.com/JacksonLLee","followers_url":"https://api.github.com/users/JacksonLLee/followers","following_url":"https://api.github.com/users/JacksonLLee/following{/other_user}","gists_url":"https://api.github.com/users/JacksonLLee/gists{/gist_id}","starred_url":"https://api.github.com/users/JacksonLLee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JacksonLLee/subscriptions","organizations_url":"https://api.github.com/users/JacksonLLee/orgs","repos_url":"https://api.github.com/users/JacksonLLee/repos","events_url":"https://api.github.com/users/JacksonLLee/events{/privacy}","received_events_url":"https://api.github.com/users/JacksonLLee/received_events","type":"User","site_admin":false},"created_at":"2015-09-23T18:10:23Z","updated_at":"2015-09-23T18:10:23Z","body":"1. All (or almost all?) JSON outputs have a corresponding customized text output for humans and for the most part we don't really have to view the JSON files ourselves. Perhaps that's why I picked the most compact JSON representation, though I imagine we might possibly want to inspect the JSON outputs for debugging or other purposes. Maybe in my next push update, the indent argument could be set to some positive integer.\r\n\r\n2. The affixes in any given signature are sorted (to ensure the uniqueness of the signatures), so any signature that contains an affix beginning with an apostrophe (= the single quote) will almost certainly have this affix as the *first* affix (e.g., 's-s, 's-NULL, etc). So far I haven't run into any parsing issues regarding this. Any comments from @JohnAGoldsmith and others?\r\n"}}