{"action":"created","issue":{"url":"https://api.github.com/repos/Particular/ServiceControl/issues/440","labels_url":"https://api.github.com/repos/Particular/ServiceControl/issues/440/labels{/name}","comments_url":"https://api.github.com/repos/Particular/ServiceControl/issues/440/comments","events_url":"https://api.github.com/repos/Particular/ServiceControl/issues/440/events","html_url":"https://github.com/Particular/ServiceControl/issues/440","id":55005511,"number":440,"title":"ServiceControl creates duplicate messages in the DB if the log queue is not writable","user":{"login":"mackie1001","id":2169646,"avatar_url":"https://avatars.githubusercontent.com/u/2169646?v=3","gravatar_id":"","url":"https://api.github.com/users/mackie1001","html_url":"https://github.com/mackie1001","followers_url":"https://api.github.com/users/mackie1001/followers","following_url":"https://api.github.com/users/mackie1001/following{/other_user}","gists_url":"https://api.github.com/users/mackie1001/gists{/gist_id}","starred_url":"https://api.github.com/users/mackie1001/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mackie1001/subscriptions","organizations_url":"https://api.github.com/users/mackie1001/orgs","repos_url":"https://api.github.com/users/mackie1001/repos","events_url":"https://api.github.com/users/mackie1001/events{/privacy}","received_events_url":"https://api.github.com/users/mackie1001/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Particular/ServiceControl/labels/Bug","name":"Bug","color":"eb6420"},{"url":"https://api.github.com/repos/Particular/ServiceControl/labels/Needs:%20Scheduling","name":"Needs: Scheduling","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-01-21T11:10:42Z","updated_at":"2015-01-27T23:48:31Z","closed_at":null,"body":"Please refer to this discussion topic for more details: https://groups.google.com/forum/#!topic/particularsoftware/dlorCAKqHcY\r\n\r\nSummary of my findings:\r\n\r\nI think if you have the option to copy messages to a further audit queue turned on but for whatever reason it fails (in my case it not having permission) it still writes the message to the FailedMessages/ProcessedMessages as well as the FailedAuditImports/FailedErrorImports collections. This means than on restart it will process FailedErrorImports and FailedAuditImports again and create a duplicate in FailedMessages/ProcessedMessages. If the problem persists then it's possible to create multiple duplicates like this.\r\n\r\nI propose that if the import process fails to write to the .log queue then it should roll back the write to FailedMessages/ProcessedMessages and only write to FailedAudtiImports/FailedErrorImports. Some form of deduplication check on import would be a good catch-all too but might affect performance significantly (I'm no RavenDB expert!)\r\n![servicecontrol duplicates](https://cloud.githubusercontent.com/assets/2169646/5835235/07dce7f8-a15e-11e4-9b5d-acf1caf65f96.png)\r\n\r\nIt might be an idea to check that the .log queues can be written to before allowing the service to start too. Better that than potentially failing import of thousands of messages all for the same reason. "},"comment":{"url":"https://api.github.com/repos/Particular/ServiceControl/issues/comments/71753746","html_url":"https://github.com/Particular/ServiceControl/issues/440#issuecomment-71753746","issue_url":"https://api.github.com/repos/Particular/ServiceControl/issues/440","id":71753746,"user":{"login":"johnsimons","id":122651,"avatar_url":"https://avatars.githubusercontent.com/u/122651?v=3","gravatar_id":"","url":"https://api.github.com/users/johnsimons","html_url":"https://github.com/johnsimons","followers_url":"https://api.github.com/users/johnsimons/followers","following_url":"https://api.github.com/users/johnsimons/following{/other_user}","gists_url":"https://api.github.com/users/johnsimons/gists{/gist_id}","starred_url":"https://api.github.com/users/johnsimons/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johnsimons/subscriptions","organizations_url":"https://api.github.com/users/johnsimons/orgs","repos_url":"https://api.github.com/users/johnsimons/repos","events_url":"https://api.github.com/users/johnsimons/events{/privacy}","received_events_url":"https://api.github.com/users/johnsimons/received_events","type":"User","site_admin":false},"created_at":"2015-01-27T23:48:31Z","updated_at":"2015-01-27T23:48:31Z","body":"Because we do not use distributed transactions this is impossible to achieve.\r\nEven if we write first to the .log queue then we could end up failing the write to db and end with duplicates in the forwarding queue.\r\nWhich one is worse, not sure."}}