{"action":"created","issue":{"url":"https://api.github.com/repos/Particular/NServiceBus.AzureStorageQueues/issues/6","labels_url":"https://api.github.com/repos/Particular/NServiceBus.AzureStorageQueues/issues/6/labels{/name}","comments_url":"https://api.github.com/repos/Particular/NServiceBus.AzureStorageQueues/issues/6/comments","events_url":"https://api.github.com/repos/Particular/NServiceBus.AzureStorageQueues/issues/6/events","html_url":"https://github.com/Particular/NServiceBus.AzureStorageQueues/issues/6","id":60611152,"number":6,"title":"Failing timeout message doesn't show in error queue","user":{"login":"SeanFeldman","id":1309622,"avatar_url":"https://avatars.githubusercontent.com/u/1309622?v=3","gravatar_id":"","url":"https://api.github.com/users/SeanFeldman","html_url":"https://github.com/SeanFeldman","followers_url":"https://api.github.com/users/SeanFeldman/followers","following_url":"https://api.github.com/users/SeanFeldman/following{/other_user}","gists_url":"https://api.github.com/users/SeanFeldman/gists{/gist_id}","starred_url":"https://api.github.com/users/SeanFeldman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SeanFeldman/subscriptions","organizations_url":"https://api.github.com/users/SeanFeldman/orgs","repos_url":"https://api.github.com/users/SeanFeldman/repos","events_url":"https://api.github.com/users/SeanFeldman/events{/privacy}","received_events_url":"https://api.github.com/users/SeanFeldman/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Particular/NServiceBus.AzureStorageQueues/labels/Bug","name":"Bug","color":"eb6420"},{"url":"https://api.github.com/repos/Particular/NServiceBus.AzureStorageQueues/labels/Needs:%20Reproduction","name":"Needs: Reproduction","color":"84b6eb"}],"state":"open","locked":false,"assignee":{"login":"SeanFeldman","id":1309622,"avatar_url":"https://avatars.githubusercontent.com/u/1309622?v=3","gravatar_id":"","url":"https://api.github.com/users/SeanFeldman","html_url":"https://github.com/SeanFeldman","followers_url":"https://api.github.com/users/SeanFeldman/followers","following_url":"https://api.github.com/users/SeanFeldman/following{/other_user}","gists_url":"https://api.github.com/users/SeanFeldman/gists{/gist_id}","starred_url":"https://api.github.com/users/SeanFeldman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SeanFeldman/subscriptions","organizations_url":"https://api.github.com/users/SeanFeldman/orgs","repos_url":"https://api.github.com/users/SeanFeldman/repos","events_url":"https://api.github.com/users/SeanFeldman/events{/privacy}","received_events_url":"https://api.github.com/users/SeanFeldman/received_events","type":"User","site_admin":false},"milestone":null,"comments":7,"created_at":"2015-03-11T05:06:55Z","updated_at":"2015-03-13T10:42:20Z","closed_at":null,"body":"## Who's Affected\r\n\r\nAnyone who is using Timeouts with Azure Storage persistence\r\n\r\n## Symptoms\r\n\r\nTimeouts are executed regulary during long period of time and stop after the following exception\r\n\r\n_EventName=\"MessageEvent\" Message=\"Failed to process messageSystem.Data.Services.Client.DataServiceRequestException: An error occurred while processing this request. ---> System.Data.Services.Client.DataServiceClientException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)   --- End of inner exception stack trace ---   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)   at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)   at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)   at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)   at System.Net.TlsStream.Read(Byte[] buffer, Int32 offset, Int32 size)   at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)   at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)   --- End of inner exception stack trace ---   at System.Net.HttpWebRequest.GetResponse()   at System.Data.Services.Client.HttpWebRequestMessage.GetResponse()   --- End of inner exception stack trace ---   --- End of inner exception stack trace ---   at System.Data.Services.Client.SaveResult.HandleResponse()   at System.Data.Services.Client.BaseSaveResult.EndRequest()   at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options)   at NServiceBus.Azure.TimeoutPersister.TryRemove(String timeoutId, TimeoutData& timeoutData) in y:\\BuildAgent\\work\\ba77a0c29cee2af1\\src\\NServiceBus.Azure\\Timeout\\TimeoutLogic\\TimeoutPersister.cs:line 179   at NServiceBus.Timeout.Hosting.Windows.TimeoutDispatcherProcessor.Handle(TransportMessage message) in c:\\BuildAgent\\work\\31f8c64a6e8a2d7c\\src\\NServiceBus.Core\\Timeout\\Hosting\\Windows\\TimeoutDispatcherProcessor.cs:line 33   at NServiceBus.Satellites.SatelliteLauncher.HandleMessageReceived(Object sender, TransportMessageReceivedEventArgs e, ISatellite satellite) in c:\\BuildAgent\\work\\31f8c64a6e8a2d7c\\src\\NServiceBus.Core\\Satellites\\SatelliteLauncher.cs:line 92   at NServiceBus.Unicast.Transport.TransportReceiver.OnTransportMessageReceived(TransportMessage msg) in c:\\BuildAgent\\work\\31f8c64a6e8a2d7c\\src\\NServiceBus.Core\\Unicast\\Transport\\TransportReceiver.cs:line 453   at NServiceBus.Unicast.Transport.TransportReceiver.ProcessMessage(TransportMessage message) in c:\\BuildAgent\\work\\31f8c64a6e8a2d7c\\src\\NServiceBus.Core\\Unicast\\Transport\\TransportReceiver.cs:line 390   at NServiceBus.Unicast.Transport.TransportReceiver.TryProcess(TransportMessage message) in c:\\BuildAgent\\work\\31f8c64a6e8a2d7c\\src\\NServiceBus.Core\\Unicast\\Transport\\TransportReceiver.cs:line 260   at NServiceBus.Azure.Transports.WindowsAzureStorageQueues.PollingDequeueStrategy.Action(Object obj) in y:\\BuildAgent\\work\\ba77a0c29cee2af1\\src\\NServiceBus.Azure.Transports.WindowsAzureStorageQueues\\PollingDequeueStrategy.cs:line 101\" TraceSource=\"WaWorkerHost.exe\"_\r\n\r\n## Issue\r\n\r\nWhen `TimeoutPersister` is attempting to remove a successfully processed timeout data, it also supposed to return the next timeout data with `timeoutData` out arguent. `ServiceContext` is [saving changes](https://github.com/Particular/NServiceBus.Azure/blob/support-5.3/src/NServiceBus.Azure/Timeout/TimeoutLogic/TimeoutPersister.cs#L183) without retries using `context.SaveChanges()` which doesn't retry and if throws exceptions, causes that exception to bubble up and next timeout data not to be persisted. This in turn causes timeout not to be executed later at all.\r\n\r\n## Possible solution\r\n\r\n**Based on MSDN documentation, this is what we'd need to do**\r\n[`TableServiceContext` is obsoleted](https://msdn.microsoft.com/en-us/library/microsoft.windowsazure.storage.table.dataservices.tableservicecontext.aspx). Instead `CloudTable` should be used and when `CloudTable.Execute()` is used, [`TableRequestOptions.RetryPolicy`](https://msdn.microsoft.com/en-us/library/microsoft.windowsazure.storage.table.tablerequestoptions_members.aspx) one of the [retry policies](https://msdn.microsoft.com/en-us/library/microsoft.windowsazure.storage.retrypolicies.aspx) could be applied (ex: [ExponentialRetry](https://msdn.microsoft.com/en-us/library/microsoft.windowsazure.storage.retrypolicies.exponentialretry.exponentialretry.aspx).\r\n\r\n**Original issue** reported on user group: [Random scheduler task randomly stops executing](https://groups.google.com/forum/#!topic/particularsoftware/MOv1g3wrwys)"},"comment":{"url":"https://api.github.com/repos/Particular/NServiceBus.AzureStorageQueues/issues/comments/78911289","html_url":"https://github.com/Particular/NServiceBus.AzureStorageQueues/issues/6#issuecomment-78911289","issue_url":"https://api.github.com/repos/Particular/NServiceBus.AzureStorageQueues/issues/6","id":78911289,"user":{"login":"Morlakh","id":11457730,"avatar_url":"https://avatars.githubusercontent.com/u/11457730?v=3","gravatar_id":"","url":"https://api.github.com/users/Morlakh","html_url":"https://github.com/Morlakh","followers_url":"https://api.github.com/users/Morlakh/followers","following_url":"https://api.github.com/users/Morlakh/following{/other_user}","gists_url":"https://api.github.com/users/Morlakh/gists{/gist_id}","starred_url":"https://api.github.com/users/Morlakh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Morlakh/subscriptions","organizations_url":"https://api.github.com/users/Morlakh/orgs","repos_url":"https://api.github.com/users/Morlakh/repos","events_url":"https://api.github.com/users/Morlakh/events{/privacy}","received_events_url":"https://api.github.com/users/Morlakh/received_events","type":"User","site_admin":false},"created_at":"2015-03-13T10:42:20Z","updated_at":"2015-03-13T10:42:20Z","body":"Hello,\r\n\r\nThe endpoint continues to run and other tasks continue to execute just fine. If you check the code excerpt I posted on user group (https://groups.google.com/d/msg/particularsoftware/MOv1g3wrwys/3ytzC_rqeUsJ) you'll see what I'm talking about. \r\n\r\nAs for transport and persistence - yes, we use the same storage account for both. I assume that that's the culprit and we should split those to use separate storage accounts, shouldn't we?"}}