{"action":"created","issue":{"url":"https://api.github.com/repos/Particular/NServiceBus/issues/3032","labels_url":"https://api.github.com/repos/Particular/NServiceBus/issues/3032/labels{/name}","comments_url":"https://api.github.com/repos/Particular/NServiceBus/issues/3032/comments","events_url":"https://api.github.com/repos/Particular/NServiceBus/issues/3032/events","html_url":"https://github.com/Particular/NServiceBus/issues/3032","id":113345998,"number":3032,"title":"SLR does not handle outgoing messages properly in v6","user":{"login":"tmasternak","id":1092707,"avatar_url":"https://avatars.githubusercontent.com/u/1092707?v=3","gravatar_id":"","url":"https://api.github.com/users/tmasternak","html_url":"https://github.com/tmasternak","followers_url":"https://api.github.com/users/tmasternak/followers","following_url":"https://api.github.com/users/tmasternak/following{/other_user}","gists_url":"https://api.github.com/users/tmasternak/gists{/gist_id}","starred_url":"https://api.github.com/users/tmasternak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tmasternak/subscriptions","organizations_url":"https://api.github.com/users/tmasternak/orgs","repos_url":"https://api.github.com/users/tmasternak/repos","events_url":"https://api.github.com/users/tmasternak/events{/privacy}","received_events_url":"https://api.github.com/users/tmasternak/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Particular/NServiceBus/labels/Impact:%20L","name":"Impact: L","color":"e11d21"},{"url":"https://api.github.com/repos/Particular/NServiceBus/labels/Size:%20M","name":"Size: M","color":"B6FF00"},{"url":"https://api.github.com/repos/Particular/NServiceBus/labels/State:%20Getting%20Ready","name":"State: Getting Ready","color":"ededed"},{"url":"https://api.github.com/repos/Particular/NServiceBus/labels/Type:%20Bug","name":"Type: Bug","color":"e10c02"}],"state":"open","locked":false,"assignee":{"login":"weralabaj","id":7335079,"avatar_url":"https://avatars.githubusercontent.com/u/7335079?v=3","gravatar_id":"","url":"https://api.github.com/users/weralabaj","html_url":"https://github.com/weralabaj","followers_url":"https://api.github.com/users/weralabaj/followers","following_url":"https://api.github.com/users/weralabaj/following{/other_user}","gists_url":"https://api.github.com/users/weralabaj/gists{/gist_id}","starred_url":"https://api.github.com/users/weralabaj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/weralabaj/subscriptions","organizations_url":"https://api.github.com/users/weralabaj/orgs","repos_url":"https://api.github.com/users/weralabaj/repos","events_url":"https://api.github.com/users/weralabaj/events{/privacy}","received_events_url":"https://api.github.com/users/weralabaj/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/Particular/NServiceBus/milestones/62","html_url":"https://github.com/Particular/NServiceBus/milestones/6.0.0","labels_url":"https://api.github.com/repos/Particular/NServiceBus/milestones/62/labels","id":704266,"number":62,"title":"6.0.0","description":"Setting up deployment process","creator":{"login":"SimonCropp","id":122666,"avatar_url":"https://avatars.githubusercontent.com/u/122666?v=3","gravatar_id":"","url":"https://api.github.com/users/SimonCropp","html_url":"https://github.com/SimonCropp","followers_url":"https://api.github.com/users/SimonCropp/followers","following_url":"https://api.github.com/users/SimonCropp/following{/other_user}","gists_url":"https://api.github.com/users/SimonCropp/gists{/gist_id}","starred_url":"https://api.github.com/users/SimonCropp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SimonCropp/subscriptions","organizations_url":"https://api.github.com/users/SimonCropp/orgs","repos_url":"https://api.github.com/users/SimonCropp/repos","events_url":"https://api.github.com/users/SimonCropp/events{/privacy}","received_events_url":"https://api.github.com/users/SimonCropp/received_events","type":"User","site_admin":false},"open_issues":20,"closed_issues":113,"state":"open","created_at":"2014-06-28T13:41:09Z","updated_at":"2015-11-09T12:58:53Z","due_on":null,"closed_at":null},"comments":5,"created_at":"2015-10-26T12:36:08Z","updated_at":"2015-11-10T08:58:50Z","closed_at":null,"body":"taskforce: @weralabaj @tmasternak\r\n\r\n# Problem\r\n\r\nThe goal of this task is to fix two issues identified in Recoverability features in current development version of v6.\r\n* Incorrect logic for moving message to timeouts or error queues. Currently SLR and Faults reuse the receive transaction on the transport level to move messages to timeouts and errors. This transaction gets committed which may (and usually will) result in side-effects of message processing being committed as well e.g. any messages which where produced by handler.\r\n* Current implementation of FLR errors (stored in memory structure which is cleaned as a whole each 5 min) may result in consistently failing message being processed infinitely many times. For example in side-by-side deployment when two instances of endpoint consume message from the same queue. In such case it is possible that instance A will do message processing until it fails and store FLR counter for it in memory. The message will be picked up by instance B which will behave in the same way but before it returns message to queue instance A FLR error data will already be cleared. In fact that will happen also for single instance if rolling back the transaction will take more than 5 min.\r\n\r\n# Redesign approach \r\nThe first problem can be solved by the approach taken in v5 i.e. SLR and Faults handling will happen as soon as message reaches SLR or Faults behaviors and will be executed if FLR error counter has reached maximum for SLR to be executed or SLR counter (stored in message header) has reached maximum for Faults logic to be executed.\r\nThe second problem will be solved by implementing LRU cache for FLR errors with relatively big constraint on elements count (e.g. 1000). In such case we will prevent any memory leaks from happening and we will minimize the possibility of infinite message processing (for FLR message counter to be cleared 1000 other unique failing messages would have to be observed before reaching maximum FLR retires for the original one). The size of LRU cache should take into consideration the size of the exception object which is stored for each failing message.\r\nRecoverability implementation should not be reused for satellites as their requirements are usually different. In addition for satellites some recoverability features should not be enabled and design should prevent from accidentally turning them on e.g. SLR for .Timeouts. \r\n\r\n# Impact\r\nTaking into consideration first issues described in the *Problem* section the impact is large.\r\n\r\n# Plan of Attack\r\n - [ ] Change SLR and Faults behavior to match v5\r\n - [ ] Merge FLR, SLR and Faults into single behavior\r\n - [ ] Implement LRU cache for FLR counters\r\n - [ ] Adjust recoverability implementations in satellites\r\n"},"comment":{"url":"https://api.github.com/repos/Particular/NServiceBus/issues/comments/155361871","html_url":"https://github.com/Particular/NServiceBus/issues/3032#issuecomment-155361871","issue_url":"https://api.github.com/repos/Particular/NServiceBus/issues/3032","id":155361871,"user":{"login":"tmasternak","id":1092707,"avatar_url":"https://avatars.githubusercontent.com/u/1092707?v=3","gravatar_id":"","url":"https://api.github.com/users/tmasternak","html_url":"https://github.com/tmasternak","followers_url":"https://api.github.com/users/tmasternak/followers","following_url":"https://api.github.com/users/tmasternak/following{/other_user}","gists_url":"https://api.github.com/users/tmasternak/gists{/gist_id}","starred_url":"https://api.github.com/users/tmasternak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tmasternak/subscriptions","organizations_url":"https://api.github.com/users/tmasternak/orgs","repos_url":"https://api.github.com/users/tmasternak/repos","events_url":"https://api.github.com/users/tmasternak/events{/privacy}","received_events_url":"https://api.github.com/users/tmasternak/received_events","type":"User","site_admin":false},"created_at":"2015-11-10T08:58:50Z","updated_at":"2015-11-10T08:58:50Z","body":"@andreasohlund I think that we need to agree on the cache size and the memory constraints for it we should be should be pretty much ready with the scope."}}