{"action":"created","issue":{"url":"https://api.github.com/repos/czlee/tabbycat/issues/157","labels_url":"https://api.github.com/repos/czlee/tabbycat/issues/157/labels{/name}","comments_url":"https://api.github.com/repos/czlee/tabbycat/issues/157/comments","events_url":"https://api.github.com/repos/czlee/tabbycat/issues/157/events","html_url":"https://github.com/czlee/tabbycat/issues/157","id":60614435,"number":157,"title":"Tournament data import module","user":{"login":"czlee","id":1725499,"avatar_url":"https://avatars.githubusercontent.com/u/1725499?v=3","gravatar_id":"","url":"https://api.github.com/users/czlee","html_url":"https://github.com/czlee","followers_url":"https://api.github.com/users/czlee/followers","following_url":"https://api.github.com/users/czlee/following{/other_user}","gists_url":"https://api.github.com/users/czlee/gists{/gist_id}","starred_url":"https://api.github.com/users/czlee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/czlee/subscriptions","organizations_url":"https://api.github.com/users/czlee/orgs","repos_url":"https://api.github.com/users/czlee/repos","events_url":"https://api.github.com/users/czlee/events{/privacy}","received_events_url":"https://api.github.com/users/czlee/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/czlee/tabbycat/labels/refactoring","name":"refactoring","color":"d4c5f9"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2015-03-11T05:56:44Z","updated_at":"2015-03-20T04:03:43Z","closed_at":null,"body":"This obsoletes #24.\r\n\r\nThis module will be responsible for doing the processing of CSV files passed to it. Its job is:\r\n* Read CSV files passed to it\r\n* Write the data in them to the database\r\n* Validate the data and decline to import faulty lines\r\n   * As a principle, it will not raise exceptions in all except the most dire circumstances, but instead will log messages using `logging.error()`.\r\n   * It will skip over faulty lines if it can.\r\n   * __Question__: Should it also return error messages for a client to write to the GUI?\r\n* Use `logging.info()` to print some comforting messages to the console.\r\n\r\nSpecifically, the following are not functions:\r\n* Any sort of user interface management, including\r\n   * \"Are you sure?\" messages\r\n   * Any sort of options handling\r\n* Finding or opening files in the file system (for flexibility, it expects an open file to be passed to it)\r\n* Setting up a new `Tournament` instance\r\n\r\nFeedback appreciated =)\r\n\r\nProposed API if implemented as class\r\n==============================\r\n\r\nAdvantages\r\n* Different importers can run without affecting each other's parameters (though why two would ever run simultaneously is unclear)\r\n* `tournament` argument can be passed as a parameter\r\n\r\nDisadvantages\r\n* More annoying for clients to set up.\r\n\r\n``` python\r\nclass TournamentDataImporter(object):\r\n    def __init__(self, tournament, **kwargs):\r\n        ...\r\n    def auto_make_rounds(self, num_rounds):\r\n        ...\r\n    def import_rounds(self, f):\r\n    def import_venues(self, f):\r\n    def import_teams(self, f):\r\n    def import_adjudicators(self, f):\r\n    [...]\r\n```\r\n\r\nClient:\r\n``` python\r\nimport debate.importer\r\nimport debate.models\r\ntournament = debate.models.Tournament.objects.get(slug='demo')\r\nimporter = debate.importer.TournamentDataImporter(tournament)\r\nimporter.import_rounds(open('rounds.csv'))\r\nimporter.import_venues(open('venues.csv'))\r\n```\r\n\r\nProposed API if implemented as functions\r\n==============================\r\nAdvantages\r\n* No set up required, can use default options\r\n\r\nDisadvantages\r\n* Parameters affect everyone\r\n* Some arguments might get repetitive\r\n\r\n``` python\r\ndef auto_make_rounds(tournament, num_rounds):\r\n    ...\r\ndef import_rounds(tournament, f):\r\n    ...\r\ndef import_venues(tournament, f):\r\ndef import_teams(tournament, f):\r\ndef import_adjudicators(tournament, f):\r\n```\r\n\r\nClient:\r\n``` python\r\nimport debate.importer as importer\r\nimport debate.models\r\ntournament = debate.models.Tournament.objects.get(slug='demo')\r\nimporter.import_rounds(tournament, open('rounds.csv'))\r\nimporter.import_venues(tournament, open('venues.csv'))\r\n```"},"comment":{"url":"https://api.github.com/repos/czlee/tabbycat/issues/comments/83887386","html_url":"https://github.com/czlee/tabbycat/issues/157#issuecomment-83887386","issue_url":"https://api.github.com/repos/czlee/tabbycat/issues/157","id":83887386,"user":{"login":"czlee","id":1725499,"avatar_url":"https://avatars.githubusercontent.com/u/1725499?v=3","gravatar_id":"","url":"https://api.github.com/users/czlee","html_url":"https://github.com/czlee","followers_url":"https://api.github.com/users/czlee/followers","following_url":"https://api.github.com/users/czlee/following{/other_user}","gists_url":"https://api.github.com/users/czlee/gists{/gist_id}","starred_url":"https://api.github.com/users/czlee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/czlee/subscriptions","organizations_url":"https://api.github.com/users/czlee/orgs","repos_url":"https://api.github.com/users/czlee/repos","events_url":"https://api.github.com/users/czlee/events{/privacy}","received_events_url":"https://api.github.com/users/czlee/received_events","type":"User","site_admin":false},"created_at":"2015-03-20T04:03:43Z","updated_at":"2015-03-20T04:03:43Z","body":"Some thoughts I encountered during implementation. The bottom line is that it's\r\nnot possible to write an import script that doesn't save to the database until\r\neverything is validated. But some compromises could be made.\r\n\r\n1. Should the importer save to the database after every file, or after all\r\n   files?\r\n\r\n2. If after _all_ files, I'm not sure that this is possible. It would require us\r\n   to create objects that reference other objects before they are saved. For\r\n   example, teams (often) require a reference to their institution. Now, if that\r\n   object was just created and we still have a handle to it, that's fine. But if\r\n   it requires a database lookup, _e.g._ to identify an institution from its\r\n   name/code, then we can't do that without first saving to the database. This\r\n   necessarily requires that saves be done, at least in groups, before the\r\n   entire import is completed.\r\n\r\n   Does this completely cripple our hopes of preventing primary key inflation?\r\n\r\n3. In particular, it's our current practice to `get_or_create` some items.\r\n   `get_or_create` saves model instances if they don't already exist. Should\r\n   we get rid of all of these?\r\n\r\n   If we do, then this would require us to insist (at a minimum) that all\r\n   institutions are completely imported before teams are created, all teams are\r\n   imported before speakers, rounds before motions, venue groups before venues,\r\n   _etc._\r\n\r\n   We can avoid asking the user to provide redundant information by just doing\r\n   multiple passes of files that involve this, _e.g._ in a teams list that also\r\n   provides information about institutions, do a first pass for institutions\r\n   (uniquifying them using Python processing rather than `get_or_create`), then\r\n   a second pass to create teams. But this still requires us to run multiple\r\n   database saves for that file.\r\n\r\n4. As a particularly fun case where this is distinctly unavoidable, consider\r\n   adjudicator-adjudicator conflicts (not currently supported). Either we\r\n   have to do two passes of the adjudicators file, or we have to ask that\r\n   adj-adj conflicts be provided in a separate list."}}