{"action":"created","issue":{"url":"https://api.github.com/repos/ARPA-SIMC/dballe/issues/13","labels_url":"https://api.github.com/repos/ARPA-SIMC/dballe/issues/13/labels{/name}","comments_url":"https://api.github.com/repos/ARPA-SIMC/dballe/issues/13/comments","events_url":"https://api.github.com/repos/ARPA-SIMC/dballe/issues/13/events","html_url":"https://github.com/ARPA-SIMC/dballe/issues/13","id":106924302,"number":13,"title":"Python db.load raise exception when loading from urlopen()","user":{"login":"edigiacomo","id":927277,"avatar_url":"https://avatars.githubusercontent.com/u/927277?v=3","gravatar_id":"","url":"https://api.github.com/users/edigiacomo","html_url":"https://github.com/edigiacomo","followers_url":"https://api.github.com/users/edigiacomo/followers","following_url":"https://api.github.com/users/edigiacomo/following{/other_user}","gists_url":"https://api.github.com/users/edigiacomo/gists{/gist_id}","starred_url":"https://api.github.com/users/edigiacomo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/edigiacomo/subscriptions","organizations_url":"https://api.github.com/users/edigiacomo/orgs","repos_url":"https://api.github.com/users/edigiacomo/repos","events_url":"https://api.github.com/users/edigiacomo/events{/privacy}","received_events_url":"https://api.github.com/users/edigiacomo/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ARPA-SIMC/dballe/labels/bug","name":"bug","color":"fc2929"}],"state":"closed","locked":false,"assignee":{"login":"spanezz","id":1324501,"avatar_url":"https://avatars.githubusercontent.com/u/1324501?v=3","gravatar_id":"","url":"https://api.github.com/users/spanezz","html_url":"https://github.com/spanezz","followers_url":"https://api.github.com/users/spanezz/followers","following_url":"https://api.github.com/users/spanezz/following{/other_user}","gists_url":"https://api.github.com/users/spanezz/gists{/gist_id}","starred_url":"https://api.github.com/users/spanezz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spanezz/subscriptions","organizations_url":"https://api.github.com/users/spanezz/orgs","repos_url":"https://api.github.com/users/spanezz/repos","events_url":"https://api.github.com/users/spanezz/events{/privacy}","received_events_url":"https://api.github.com/users/spanezz/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2015-09-17T07:06:38Z","updated_at":"2015-09-19T19:28:41Z","closed_at":"2015-09-19T19:28:41Z","body":"Starting a web server from top src dir:\r\n```\r\n$ python -m SimpleHTTPServer 8000\r\n```\r\n\r\nThe code sometimes raise `KeyError: 'could not detect the encoding of '` or `OSError: reading a 5505024-bytes record from : Illegal seek`.\r\n\r\n```\r\n# test.py\r\nimport dballe\r\nfrom urllib2 import urlopen\r\nfrom glob import glob\r\n\r\ndb = dballe.DB.connect_from_file(\"/tmp/buttami.db\")\r\ndb.reset()\r\nfor f in glob(\"extra/bufr/*.bufr\"):\r\n    r = urlopen(\"http://localhost:8000/{}\".format(f))\r\n    db.load(r)\r\n```\r\n\r\n\r\nInspecting with gdb, it seems that sometimes the encoding is not dected the line [int c = getc(stream);](https://github.com/ARPA-SIMC/dballe/blob/devel/dballe/file.cc#L90) return 255 (encoding not detected) or 0 (create a AOF file):\r\n\r\n```\r\n$ gdb python\r\n(gdb) l dballe/file.cc:98\r\n93\t    if (c == EOF)\r\n94\t        return create(BUFR, st.release(), close_on_exit, name);\r\n95\t\r\n96\t    if (ungetc(c, stream) == EOF)\r\n97\t        error_system::throwf(\"cannot put the first byte of %s back into the input stream\", name.c_str());\r\n98\t\r\n99\t    switch (c)\r\n100\t    {\r\n101\t        case 'B': return create(BUFR, st.release(), close_on_exit, name);\r\n102\t        case 'C': return create(CREX, st.release(), close_on_exit, name);\r\n(gdb) b dballe/file.cc:98\r\n(gdb) r test.py\r\nStarting program: /usr/bin/python test.py\r\n[Thread debugging using libthread_db enabled]\r\nUsing host libthread_db library \"/lib64/libthread_db.so.1\".\r\n\r\nBreakpoint 1, dballe::File::create (stream=0x7fbc60, close_on_exit=close_on_exit@entry=true, name=\"\") at file.cc:99\r\n99\t    switch (c)\r\n(gdb) p c\r\n$1 = 0\r\n```"},"comment":{"url":"https://api.github.com/repos/ARPA-SIMC/dballe/issues/comments/141700605","html_url":"https://github.com/ARPA-SIMC/dballe/issues/13#issuecomment-141700605","issue_url":"https://api.github.com/repos/ARPA-SIMC/dballe/issues/13","id":141700605,"user":{"login":"spanezz","id":1324501,"avatar_url":"https://avatars.githubusercontent.com/u/1324501?v=3","gravatar_id":"","url":"https://api.github.com/users/spanezz","html_url":"https://github.com/spanezz","followers_url":"https://api.github.com/users/spanezz/followers","following_url":"https://api.github.com/users/spanezz/following{/other_user}","gists_url":"https://api.github.com/users/spanezz/gists{/gist_id}","starred_url":"https://api.github.com/users/spanezz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spanezz/subscriptions","organizations_url":"https://api.github.com/users/spanezz/orgs","repos_url":"https://api.github.com/users/spanezz/repos","events_url":"https://api.github.com/users/spanezz/events{/privacy}","received_events_url":"https://api.github.com/users/spanezz/received_events","type":"User","site_admin":false},"created_at":"2015-09-19T19:28:41Z","updated_at":"2015-09-19T19:28:41Z","body":"Auto detection of encoding is done only on the first byte of the input, but there are examples in extra/bufr that have other leading data at the beginning of the file, which confuses autodetection.\r\n\r\nI have committed the option to specify the encoding from python explicitly:\r\n\r\n```\r\nIn [3]: db.load?\r\nType:        builtin_function_or_method\r\nString form: <built-in method load of dballe.DB object at 0x7f7aaa6d2290>\r\nDocstring:\r\nload(fp, encoding=None)\r\n\r\nLoad a file object in the database. An encoding can optionally be\r\nprovided as a string (\"BUFR\", \"CREX\", \"AOF\"). If encoding is None then\r\nload will try to autodetect based on the first byte of the file.\r\n```\r\n\r\nSo now you can do this and it should work:\r\n\r\n```python\r\n    db.load(r, \"BUFR\")\r\n```"}}