{"action":"created","issue":{"url":"https://api.github.com/repos/dotnet/orleans/issues/100","labels_url":"https://api.github.com/repos/dotnet/orleans/issues/100/labels{/name}","comments_url":"https://api.github.com/repos/dotnet/orleans/issues/100/comments","events_url":"https://api.github.com/repos/dotnet/orleans/issues/100/events","html_url":"https://github.com/dotnet/orleans/pull/100","id":56969103,"number":100,"title":"Updated Microsoft.WindowsAzure.Storage and Microsoft.WindowsAzure.ConfigurationManager to the latest Nuget packages and code converted accordingly","user":{"login":"veikkoeeva","id":5158694,"avatar_url":"https://avatars.githubusercontent.com/u/5158694?v=3","gravatar_id":"","url":"https://api.github.com/users/veikkoeeva","html_url":"https://github.com/veikkoeeva","followers_url":"https://api.github.com/users/veikkoeeva/followers","following_url":"https://api.github.com/users/veikkoeeva/following{/other_user}","gists_url":"https://api.github.com/users/veikkoeeva/gists{/gist_id}","starred_url":"https://api.github.com/users/veikkoeeva/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/veikkoeeva/subscriptions","organizations_url":"https://api.github.com/users/veikkoeeva/orgs","repos_url":"https://api.github.com/users/veikkoeeva/repos","events_url":"https://api.github.com/users/veikkoeeva/events{/privacy}","received_events_url":"https://api.github.com/users/veikkoeeva/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/dotnet/orleans/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":{"login":"gabikliot","id":10171207,"avatar_url":"https://avatars.githubusercontent.com/u/10171207?v=3","gravatar_id":"","url":"https://api.github.com/users/gabikliot","html_url":"https://github.com/gabikliot","followers_url":"https://api.github.com/users/gabikliot/followers","following_url":"https://api.github.com/users/gabikliot/following{/other_user}","gists_url":"https://api.github.com/users/gabikliot/gists{/gist_id}","starred_url":"https://api.github.com/users/gabikliot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gabikliot/subscriptions","organizations_url":"https://api.github.com/users/gabikliot/orgs","repos_url":"https://api.github.com/users/gabikliot/repos","events_url":"https://api.github.com/users/gabikliot/events{/privacy}","received_events_url":"https://api.github.com/users/gabikliot/received_events","type":"User","site_admin":false},"milestone":null,"comments":39,"created_at":"2015-02-08T21:42:14Z","updated_at":"2015-02-20T04:12:49Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/dotnet/orleans/pulls/100","html_url":"https://github.com/dotnet/orleans/pull/100","diff_url":"https://github.com/dotnet/orleans/pull/100.diff","patch_url":"https://github.com/dotnet/orleans/pull/100.patch"},"body":"This resolves #79 *when* the converted code works as intended.\r\n\r\nWith this I sought to see approximately how much effort it would take to stab the storage system and what issues there would be. There aren’t many storage related tests and I thought that creating them would take too much effort for what I aimed at. I am not sure if this would be the best approach forward. One could reason testing (and fixing) these new binaries could be done observing by experience if they behave well enough. That is, there are other means besides a test harness to be assured of functionality, e.g. logs, alerts, experience of running a sufficiently busy system. If nothing else, this refactoring attempt presents some way to transform the code to quite close the needed functionality.\r\n\r\nThe resulting code is not good nor is it even correct in all parts. This uses the new binaries, but there needs to be cleanup especially in [AzureTableDataManager](https://github.com/veikkoeeva/orleans/blob/master/src/Orleans/AzureUtils/AzureTableDataManager.cs) storage methods. Even after that, it feels it’s somewhat a throw-away phase. Albeit this might be the case, I feel it would be an important step when assessing the conformance of this new version with the previous one. Especially this version would allow using newer libraries with the Orleans Nuget packages. In the meantime there could be incremental performance updates and even a new storage system implementation. There’s a blog post [Windows Azure Storage Client Library 2.0 Tables Deep Dive]( http://blogs.msdn.com/b/windowsazurestorage/archive/2012/11/06/windows-azure-storage-client-library-2-0-tables-deep-dive.aspx) which explains the newer constructs quite well.\r\n\r\n###### Some spots that look potential bugs and cases for improvement\r\n* [DeleteTableEntriesAsync(IReadOnlyCollection<Tuple<T, string>> list)](https://github.com/veikkoeeva/orleans/blob/master/src/Orleans/AzureUtils/AzureTableDataManager.cs#L517)\r\ndoesn’t check the list length. The batches are limited to 100 elements by Azure runtime. Even better would be take an arbitrary length list and chunk it to appropriately sized batches.\r\n* It’s counterpart [BulkInsertTableEntries(IReadOnlyCollection<T> data)](https://github.com/veikkoeeva/orleans/blob/master/src/Orleans/AzureUtils/AzureTableDataManager.cs#L634)\r\ndoes have a check, but it could also do chunking.\r\n* [ReadTableEntriesAndEtagsAsync(Expression<Func<T, bool>> predicate)](https://github.com/veikkoeeva/orleans/blob/master/src/Orleans/AzureUtils/AzureTableDataManager.cs#L574)\r\nlook like do not need complicated custom error handling anymore.\r\n* Currently ``ClouldTableClient`` is passed as a parameter to a lot of function. Perhaps better would be to pass ``CloudTable``, as usually the immediate call is ``var tableReference = tableClient.GetTableReference(TableName);`` to obtain a reference and then use it.\r\n* Perhaps functionality to explicitly resolve entity type and do reflection isn’t needed anymore with refactoring the storage functionality.\r\n* ~~In [ReminderTableEntry](https://github.com/veikkoeeva/orleans/blob/master/src/OrleansRuntime/ReminderService/RemindersTableManager.cs#L47) the ETag is shadowed.~~\r\n* ~~This refactoring isn't ready. There are todos, which should be turned into issues, commented code that ought to be removed and, as mentioned, issues with how to update, insert, upsert, merge entities. This could be minor issues if someone points out the desired behaviour in terms of the new library (or straight HTTP verbs).~~\r\n\r\n###### How to proceed\r\n* I don't mind other people taking this code to make progress or adding to this here (I'm not sure how to do that, but probably doable).\r\n* Can someone provide tests to provide the correct functinality or run the refactored code to attain a reasonable idea if it works correctly? If not, then perhaps one path forwards would be to capture data from a present system, create test material and check the results of the new system against this material. This probably needs to be done soon after this update anyway (but perhaps this update may alter the way entities are stored, which is desirably differently than currently).\r\n\r\n###### A bold aim\r\nI think the bolder aim could be to provide a stream to storage, or two streams. One stream could save straight to storage, the other could batch messages and save (adaptively on load?) based either on some time interval or elements in the queue. This stream perhaps could have a configurable failover so that when, say, Azure Table Storage (ATS) goes offline, a secondary system would be attempted (a SQL Server, or another ATS storage (via connectionstring)). This system could be perhaps provide library components to a common pubsub functionality. This kind of batching doesn’t look like too complicated from an Rx perspective. As an example: http://blog.petegoo.com/2013/01/20/reactive-extensions-in-message-processing/ (though delay sensitive stream would be more complicated).\r\n\r\nThese streams could, naturally, provide a place to intercept the messages."},"comment":{"url":"https://api.github.com/repos/dotnet/orleans/issues/comments/75187003","html_url":"https://github.com/dotnet/orleans/pull/100#issuecomment-75187003","issue_url":"https://api.github.com/repos/dotnet/orleans/issues/100","id":75187003,"user":{"login":"veikkoeeva","id":5158694,"avatar_url":"https://avatars.githubusercontent.com/u/5158694?v=3","gravatar_id":"","url":"https://api.github.com/users/veikkoeeva","html_url":"https://github.com/veikkoeeva","followers_url":"https://api.github.com/users/veikkoeeva/followers","following_url":"https://api.github.com/users/veikkoeeva/following{/other_user}","gists_url":"https://api.github.com/users/veikkoeeva/gists{/gist_id}","starred_url":"https://api.github.com/users/veikkoeeva/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/veikkoeeva/subscriptions","organizations_url":"https://api.github.com/users/veikkoeeva/orgs","repos_url":"https://api.github.com/users/veikkoeeva/repos","events_url":"https://api.github.com/users/veikkoeeva/events{/privacy}","received_events_url":"https://api.github.com/users/veikkoeeva/received_events","type":"User","site_admin":false},"created_at":"2015-02-20T04:12:49Z","updated_at":"2015-02-20T04:12:49Z","body":"It's about six o'clock in the morning here, I'll try try in about fifteen hours if you haven't merge this by then. I'm excited!"}}