{"action":"created","issue":{"url":"https://api.github.com/repos/lefcha/imapfilter/issues/93","labels_url":"https://api.github.com/repos/lefcha/imapfilter/issues/93/labels{/name}","comments_url":"https://api.github.com/repos/lefcha/imapfilter/issues/93/comments","events_url":"https://api.github.com/repos/lefcha/imapfilter/issues/93/events","html_url":"https://github.com/lefcha/imapfilter/issues/93","id":81460081,"number":93,"title":"Meta-Searching always searches ALL messages on server","user":{"login":"shartge","id":684712,"avatar_url":"https://avatars.githubusercontent.com/u/684712?v=3","gravatar_id":"","url":"https://api.github.com/users/shartge","html_url":"https://github.com/shartge","followers_url":"https://api.github.com/users/shartge/followers","following_url":"https://api.github.com/users/shartge/following{/other_user}","gists_url":"https://api.github.com/users/shartge/gists{/gist_id}","starred_url":"https://api.github.com/users/shartge/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shartge/subscriptions","organizations_url":"https://api.github.com/users/shartge/orgs","repos_url":"https://api.github.com/users/shartge/repos","events_url":"https://api.github.com/users/shartge/events{/privacy}","received_events_url":"https://api.github.com/users/shartge/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":25,"created_at":"2015-05-27T14:02:12Z","updated_at":"2015-06-14T19:40:48Z","closed_at":null,"body":"Hi!\r\n\r\nI am unsure if I am using imapfilter correctly but it looks like that meta-searching, while selecting the correct subset of messages, always searches all messages on the server and not the range of UID returned by the previous operation.\r\n\r\nFor example, this simple configuration, just selecting the messages since yesterday and then further selecting some mails with a specific header value:\r\n\r\n```lua\r\noptions.timeout = 120\r\noptions.subscribe = true\r\noptions.keepalive = 5\r\noptions.starttls = true\r\n\r\nfunction sort_mail()\r\n        \r\n        myaccount = IMAP {\r\n                server = 'myserver',\r\n                username = 'myuser',\r\n                password = 'mypasswd',\r\n        }\r\n\r\n        -- Get the status of a mailbox\r\n        myaccount.INBOX:check_status()\r\n\r\n        mails = myaccount.INBOX:arrived_since(form_date(1))\r\n\r\n        internallist = mails:contain_field('List-Id', 'some-internal-list')\r\n\r\nend\r\n\r\nsort_mail()\r\n```\r\n\r\nAnd this is the debuglog (I trimmed the STARTTLS and LOGIN phase) generated:\r\n\r\n```\r\n* CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL C\r\nATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SPECIAL-\r\nUSE COMPRESS=DEFLATE\r\n1005 OK Capability completed.\r\n\r\nsending command (4):\r\n\r\n1006 NAMESPACE\r\n\r\ngetting response (4):\r\n\r\n* NAMESPACE ((\"\" \".\")) NIL NIL\r\n1006 OK Namespace completed.\r\n\r\nnamespace (4): '' '.'\r\nsending command (4):\r\n\r\n1007 STATUS \"INBOX\" (MESSAGES RECENT UNSEEN UIDNEXT)\r\n\r\ngetting response (4):\r\n\r\n* STATUS INBOX (MESSAGES 1894 RECENT 0 UIDNEXT 99370 UNSEEN 0)\r\n1007 OK Status completed (0.000 secs).\r\n\r\nsending command (4):\r\n\r\n1008 SELECT \"INBOX\"\r\n\r\ngetting response (4):\r\n\r\n* FLAGS (\\Answered \\Flagged \\Deleted \\Seen \\Draft $Label3 $Label1 $Label2 $Label4 $Label5 $MDNSent Junk NonJunk $Forwarded redirected $Labelwork)\r\n* OK [PERMANENTFLAGS (\\Answered \\Flagged \\Deleted \\Seen \\Draft $Label3 $Label1 $Label2 $Label4 $Label5 $MDNSent Junk NonJunk $Forwarded redirected $Labelwork \\*)] Flags permitted.\r\n* 1894 EXISTS\r\n* 0 RECENT\r\n* OK [UIDVALIDITY 1051467451] UIDs valid\r\n* OK [UIDNEXT 99370] Predicted next UID\r\n* OK [HIGHESTMODSEQ 135505] Highest\r\n1008 OK [READ-WRITE] Select completed (0.000 secs).\r\nsending command (4):\r\n\r\n1009 UID SEARCH ALL SINCE 26-May-2015\r\n\r\ngetting response (4):\r\n\r\n* SEARCH 99325 99326 99329 99332 99334 99343 99345 99348 99349 99351 99352 99353 99354 99361 99363 99364 99368\r\n1009 OK Search completed (0.011 secs).\r\n\r\nsending command (4):\r\n\r\n100A UID SEARCH ALL HEADER List-Id \"some-internal-list\"\r\n\r\ngetting response (4):\r\n\r\n* SEARCH\r\n100A OK Search completed (0.013 secs).\r\n\r\nsending command (4):\r\n\r\n100B LOGOUT\r\n\r\ngetting response (4):\r\n\r\n* BYE Logging out\r\n100B OK Logout completed.\r\n```\r\n\r\nAs you can see, it first gets a list of UIDs from the \"SEARCH ALL SINCE 26-May-2015\" and then, when searching for the \"List-Id\" just uses \"ALL\" again instead of search only in the UIDs gathered via the first query.\r\n\r\nFrom looking at the code in mailbox.lua and common.lua it seems that meta-searching only happens inside of imapfilter and is never used to optimize the queries send to the server. Which would explain why my simple example takes 15s to execute on a big INBOX.\r\n\r\nOr I am doing something stupid and I am using imapfilter in the wrong way.\r\n"},"comment":{"url":"https://api.github.com/repos/lefcha/imapfilter/issues/comments/111869759","html_url":"https://github.com/lefcha/imapfilter/issues/93#issuecomment-111869759","issue_url":"https://api.github.com/repos/lefcha/imapfilter/issues/93","id":111869759,"user":{"login":"lefcha","id":640819,"avatar_url":"https://avatars.githubusercontent.com/u/640819?v=3","gravatar_id":"","url":"https://api.github.com/users/lefcha","html_url":"https://github.com/lefcha","followers_url":"https://api.github.com/users/lefcha/followers","following_url":"https://api.github.com/users/lefcha/following{/other_user}","gists_url":"https://api.github.com/users/lefcha/gists{/gist_id}","starred_url":"https://api.github.com/users/lefcha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lefcha/subscriptions","organizations_url":"https://api.github.com/users/lefcha/orgs","repos_url":"https://api.github.com/users/lefcha/repos","events_url":"https://api.github.com/users/lefcha/events{/privacy}","received_events_url":"https://api.github.com/users/lefcha/received_events","type":"User","site_admin":false},"created_at":"2015-06-14T19:40:48Z","updated_at":"2015-06-14T19:40:48Z","body":"Hm, that response looks fine...\r\n\r\nCan you copy/paste the line(s) from `mailbox.lua` that you changed to remove the 50 items limitation?"}}