{"action":"created","issue":{"url":"https://api.github.com/repos/jondot/sneakers/issues/111","labels_url":"https://api.github.com/repos/jondot/sneakers/issues/111/labels{/name}","comments_url":"https://api.github.com/repos/jondot/sneakers/issues/111/comments","events_url":"https://api.github.com/repos/jondot/sneakers/issues/111/events","html_url":"https://github.com/jondot/sneakers/issues/111","id":66402728,"number":111,"title":"How to handle duplicated messages?","user":{"login":"sharshenov","id":1262244,"avatar_url":"https://avatars.githubusercontent.com/u/1262244?v=3","gravatar_id":"","url":"https://api.github.com/users/sharshenov","html_url":"https://github.com/sharshenov","followers_url":"https://api.github.com/users/sharshenov/followers","following_url":"https://api.github.com/users/sharshenov/following{/other_user}","gists_url":"https://api.github.com/users/sharshenov/gists{/gist_id}","starred_url":"https://api.github.com/users/sharshenov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sharshenov/subscriptions","organizations_url":"https://api.github.com/users/sharshenov/orgs","repos_url":"https://api.github.com/users/sharshenov/repos","events_url":"https://api.github.com/users/sharshenov/events{/privacy}","received_events_url":"https://api.github.com/users/sharshenov/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-04-05T05:06:23Z","updated_at":"2015-04-09T18:54:32Z","closed_at":null,"body":"As said in [RMQ documentation](https://www.rabbitmq.com/reliability.html#consumer):\r\n> In the event of network failure (or a node crashing), messages can be duplicated, and consumers must be prepared to handle them. If possible, the simplest way to handle this is to ensure that your consumers handle messages in an idempotent way rather than explicitly deal with deduplication.\r\n> If a message is delivered to a consumer and then requeued (because it was not acknowledged before the consumer connection dropped, for example) then RabbitMQ will set the redelivered flag on it when it is delivered again (whether to the same consumer or a different one). This is a hint that a consumer may have seen this message before (although that's not guaranteed, the message may have made it out of the broker but not into a consumer before the connection dropped). Conversely if the redelivered flag is not set then it is guaranteed that the message has not been seen before. Therefore if a consumer finds it more expensive to deduplicate messages or process them in an idempotent manner, it can do this only for messages with the redelivered flag set. \r\n\r\nThere is some scenarios where consumers can not handle messages in a idempotent way (email delivery, file operations, etc). `redelivered` flag is not a silver bullet, I think. In order to perform deduplication the consumer should have a list of recent messages and reject a duplicated messages. All workers of the consumer should have an access to this list. I think, the best way is to use Redis for this purpose.\r\n\r\nIdeally, there should be some option for Sneakers to enable messages check for uniqueness before processing them.\r\n```ruby\r\nclass MyWorker\r\n  include Sneakers::Worker\r\n\r\n  from_queue :email, unique: { timeout: 30.minutes }\r\n\r\n  def work msg\r\n    MyJobProcessor.new(msg).perform\r\n\r\n    ack!\r\n  end\r\n\r\nend\r\n```\r\nWhat do you think?"},"comment":{"url":"https://api.github.com/repos/jondot/sneakers/issues/comments/91326990","html_url":"https://github.com/jondot/sneakers/issues/111#issuecomment-91326990","issue_url":"https://api.github.com/repos/jondot/sneakers/issues/111","id":91326990,"user":{"login":"jondot","id":83390,"avatar_url":"https://avatars.githubusercontent.com/u/83390?v=3","gravatar_id":"","url":"https://api.github.com/users/jondot","html_url":"https://github.com/jondot","followers_url":"https://api.github.com/users/jondot/followers","following_url":"https://api.github.com/users/jondot/following{/other_user}","gists_url":"https://api.github.com/users/jondot/gists{/gist_id}","starred_url":"https://api.github.com/users/jondot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jondot/subscriptions","organizations_url":"https://api.github.com/users/jondot/orgs","repos_url":"https://api.github.com/users/jondot/repos","events_url":"https://api.github.com/users/jondot/events{/privacy}","received_events_url":"https://api.github.com/users/jondot/received_events","type":"User","site_admin":false},"created_at":"2015-04-09T18:54:32Z","updated_at":"2015-04-09T18:54:32Z","body":"Yup, @michaelklishin is correct - HLL is an approximation. It will suit cases where you don't want to perform a redundant job - because you don't want to \"pay\" for it, and if you did perform a redudant job approx %0.18 of the times - it can be tolerated.\r\nOtherwise exact IDs should be traced, always.\r\n\r\nCallbacks and lonley_job are interesting - for a long while now I was wanting to also provide a generic base worker types such as:\r\n\r\n`RedisWorker`\r\n`CountingWorker`\r\netc.\r\na `CallbackWorker` may be due in this model. What do you think?\r\n"}}