{"action":"closed","issue":{"url":"https://api.github.com/repos/poeml/mirrorbrain/issues/94","labels_url":"https://api.github.com/repos/poeml/mirrorbrain/issues/94/labels{/name}","comments_url":"https://api.github.com/repos/poeml/mirrorbrain/issues/94/comments","events_url":"https://api.github.com/repos/poeml/mirrorbrain/issues/94/events","html_url":"https://github.com/poeml/mirrorbrain/issues/94","id":85356152,"number":94,"title":"makehashes: Regular expression not quoted","user":{"login":"poeml","id":12400278,"avatar_url":"https://avatars.githubusercontent.com/u/12400278?v=3","gravatar_id":"","url":"https://api.github.com/users/poeml","html_url":"https://github.com/poeml","followers_url":"https://api.github.com/users/poeml/followers","following_url":"https://api.github.com/users/poeml/following{/other_user}","gists_url":"https://api.github.com/users/poeml/gists{/gist_id}","starred_url":"https://api.github.com/users/poeml/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/poeml/subscriptions","organizations_url":"https://api.github.com/users/poeml/orgs","repos_url":"https://api.github.com/users/poeml/repos","events_url":"https://api.github.com/users/poeml/events{/privacy}","received_events_url":"https://api.github.com/users/poeml/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/poeml/mirrorbrain/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/poeml/mirrorbrain/labels/resolved","name":"resolved","color":"ededed"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2015-06-05T01:01:34Z","updated_at":"2015-06-05T01:08:51Z","closed_at":"2015-06-05T01:08:51Z","body":"                                                                                                                                                                                       [          ]\n                                                                                                                                                                                       \n#### Issue migrated (2015-06-05) from old issue tracker http://mirrorbrain.org/issues/issue94\n\n```\nTitle    makehashes: Regular expression not quoted\n Priority   bug    Status   resolved\nSuperseder        Nosy List poeml, sascha_silbe, toma\nAssigned To poeml Keywords\n\n```\n\n\n##### msg345 (view)                                  Author: sascha_silbe                                                Date: 2012-01-09.22:36:12\nWhen running \"mb -d makehashes /srv/upload -t /srv/mirrorbrain/hashes/srv/upload\" on the server hosting http://download.sugarlabs.org/, MirrorBrain breaks with the following error:\n\n[...]\n 2/QueryAll:  SELECT filearr.path, hash.file_id\n                     FROM filearr\n                 LEFT JOIN hash\n                     ON hash.file_id = filearr.id\n                 WHERE filearr.path ~ '^services/gcc-c++/[^/]*$'\n 2/QueryR  :  SELECT filearr.path, hash.file_id\n                     FROM filearr\n                 LEFT JOIN hash\n                     ON hash.file_id = filearr.id\n                 WHERE filearr.path ~ '^services/gcc-c++/[^/]*$'\n 2/COMMIT  :  auto\nTraceback (most recent call last):\n    File \"/usr/bin/mb\", line 1638, in <module>\n      r = mirrordoctor.main()\n    File \"/usr/lib/pymodules/python2.6/cmdln.py\", line 257, in main\n      return self.cmd(args)\n    File \"/usr/lib/pymodules/python2.6/cmdln.py\", line 280, in cmd\n      retval = self.onecmd(argv)\n    File \"/usr/lib/pymodules/python2.6/cmdln.py\", line 412, in onecmd\n      return self._dispatch_cmd(handler, argv)\n    File \"/usr/lib/pymodules/python2.6/cmdln.py\", line 1100, in _dispatch_cmd\n      return handler(argv[0], opts, *args)\n    File \"/usr/bin/mb\", line 1024, in do_makehashes\n      for i, j in mb.files.dir_filelist(self.conn, dst_dir_db)]\n    File \"/usr/lib/pymodules/python2.6/mb/files.py\", line 160, in dir_filelist\n      result = conn.Server._connection.queryAll(query)\n    File \"/usr/lib/python2.6/dist-packages/sqlobject/dbconnection.py\", line 356, in queryAll\n      return self._runWithConnection(self._queryAll, s)\n    File \"/usr/lib/python2.6/dist-packages/sqlobject/dbconnection.py\", line 256, in _runWithConnection\n      val = meth(conn, *args)\n    File \"/usr/lib/python2.6/dist-packages/sqlobject/dbconnection.py\", line 349, in _queryAll\n      self._executeRetry(conn, c, s)\n    File \"/usr/lib/python2.6/dist-packages/sqlobject/dbconnection.py\", line 335, in _executeRetry\n      return cursor.execute(query)\npsycopg2.DataError: invalid regular expression: quantifier operand invalid\n\n\nservices/gcc-c++ is the name of a directory below /srv/upload:\n\nsilbe@sunjammer:~$ ls -d /srv/upload/services/gcc-c++\n/srv/upload/services/gcc-c++\n\nMirrorBrain should escape special (regular expression) characters in paths before using them as part of a regular expression.\n\nAdditional info:\nThe host is running MirrorBrain 2.15.0-1 on Ubuntu 10.04:\n\nsilbe@sunjammer:~$ lsb_release -ir\nDistributor ID: Ubuntu\nRelease:        10.04\nsilbe@sunjammer:~$ dpkg -l mirrorbrain|grep ^ii\nii  mirrorbrain                       2.15.0-1                                        MirrorBrain is a scalable download redirector and Metalink generator.\n\n##### msg358 (view)                                      Author: toma                                                    Date: 2012-03-25.10:50:45\nBug confirmed. KDE runs into this one as well.\n\n##### msg364 (view)                                      Author: poeml                                                   Date: 2012-03-26.22:31:22\nIt seems to me that the only way to deal with this is to manually escape regexp\nspecial characters in the path names.\n\n select 'services/gcc-c++/a' ~ '***:^services/gcc-c\\+\\+/[^/]*$' as result;\n result\n--------\n t\n\n\nThere is no PostgreSQL function to do this, and there doesn't seem to be a way to\nembed a literal string inside a regular expression.\n\n##### msg377 (view)                                      Author: poeml                                                   Date: 2012-04-11.21:22:20\nSo the task is to pass a regexp from Python to PostgreSQL, through SQLobject and psycopg2, that\ncontains some characters that need to be treated as literals.\n\nSo, let's pass all literal characters as literal characters! I.e., using octal \\000 syntax.\n\nFixed in r8271.\n\n\n\n\nIndex: ../mb/mb/files.py\n===================================================================\n--- ../mb/mb/files.py   (revision 8270)\n+++ ../mb/mb/files.py   (revision 8271)\n@@ -1,5 +1,6 @@\n from sqlobject.sqlbuilder import AND\n\n+from mb import util\n\n def has_file(conn, path, mirror_id):\n       \"\"\"check if file 'path' exists on mirror 'mirror_id'\n@@ -156,7 +157,7 @@\n                      FROM filearr\n                  LEFT JOIN hash\n                      ON hash.file_id = filearr.id\n-               WHERE filearr.path ~ '^%s/[^/]*$'\"\"\" % path\n+               WHERE filearr.path ~ '^\"\"\" + util.pgsql_regexp_esc(path) +\"\"\"/[^/]*$'\"\"\"\n\n       result = conn.Server._connection.queryAll(query)\n       return result\nIndex: ../mb/mb/util.py\n===================================================================\n--- ../mb/mb/util.py    (revision 8270)\n+++ ../mb/mb/util.py    (revision 8271)\n@@ -210,3 +210,9 @@\n           netloc = netloc.split('@')[1]\n       return urlparse.urlunsplit((u[0], netloc, u[2], u[3], u[4]))\n\n+def pgsql_regexp_esc(s):\n+    if s:\n+        return '\\\\\\\\' + '\\\\\\\\'.join(['%03o' % ord(c) for c in s])\n+    else:\n+        return s\n+\n\n```\nHistory\n         Date             User     Action             Args\n2012-04-11 21:22:20 poeml        set    status: chatting -> resolved\n                                          messages: + msg377\n2012-03-26 22:31:22 poeml        set    messages: + msg364\n2012-03-26 21:47:00 poeml        set    assignedto: poeml\n                                          nosy: + poeml\n                                          status: unread -> chatting\n2012-03-25 10:50:45 toma         set    nosy: + toma\n                                          messages: + msg358\n2012-01-09 22:36:14 sascha_silbe create\n\n```\n##### (end of migrated issue)\n\n"}}