{"action":"created","issue":{"url":"https://api.github.com/repos/ayende/ravendb/issues/2497","labels_url":"https://api.github.com/repos/ayende/ravendb/issues/2497/labels{/name}","comments_url":"https://api.github.com/repos/ayende/ravendb/issues/2497/comments","events_url":"https://api.github.com/repos/ayende/ravendb/issues/2497/events","html_url":"https://github.com/ayende/ravendb/pull/2497","id":97793503,"number":2497,"title":"Some optimizations RavenDB-3724","user":{"login":"redknightlois","id":1045411,"avatar_url":"https://avatars.githubusercontent.com/u/1045411?v=3","gravatar_id":"","url":"https://api.github.com/users/redknightlois","html_url":"https://github.com/redknightlois","followers_url":"https://api.github.com/users/redknightlois/followers","following_url":"https://api.github.com/users/redknightlois/following{/other_user}","gists_url":"https://api.github.com/users/redknightlois/gists{/gist_id}","starred_url":"https://api.github.com/users/redknightlois/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redknightlois/subscriptions","organizations_url":"https://api.github.com/users/redknightlois/orgs","repos_url":"https://api.github.com/users/redknightlois/repos","events_url":"https://api.github.com/users/redknightlois/events{/privacy}","received_events_url":"https://api.github.com/users/redknightlois/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-07-28T20:20:44Z","updated_at":"2015-07-29T06:15:34Z","closed_at":"2015-07-29T05:17:19Z","pull_request":{"url":"https://api.github.com/repos/ayende/ravendb/pulls/2497","html_url":"https://github.com/ayende/ravendb/pull/2497","diff_url":"https://github.com/ayende/ravendb/pull/2497.diff","patch_url":"https://github.com/ayende/ravendb/pull/2497.patch"},"body":"This is the best I could do without seriously rethinking how map-reduce works. I must say that the constraints of the data model does not help the case. RavenDB-3724 is a must to be done for RavenDB rethinking the storage model. \r\n\r\nAnother thing that will improve this is changing the storage to a binary representation as Json Serialization/Deserialization is insanely costly in this scenario.\r\n\r\nAfter I finish the prefix trees POC I will look into what we can do for multi-step map-reduce and if they could not be for 4.0 at least know what we need in order to not closing the door to it when we rethink the indexing storage model (which is a must for 4.0). "},"comment":{"url":"https://api.github.com/repos/ayende/ravendb/issues/comments/125853456","html_url":"https://github.com/ayende/ravendb/pull/2497#issuecomment-125853456","issue_url":"https://api.github.com/repos/ayende/ravendb/issues/2497","id":125853456,"user":{"login":"ayende","id":116915,"avatar_url":"https://avatars.githubusercontent.com/u/116915?v=3","gravatar_id":"","url":"https://api.github.com/users/ayende","html_url":"https://github.com/ayende","followers_url":"https://api.github.com/users/ayende/followers","following_url":"https://api.github.com/users/ayende/following{/other_user}","gists_url":"https://api.github.com/users/ayende/gists{/gist_id}","starred_url":"https://api.github.com/users/ayende/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ayende/subscriptions","organizations_url":"https://api.github.com/users/ayende/orgs","repos_url":"https://api.github.com/users/ayende/repos","events_url":"https://api.github.com/users/ayende/events{/privacy}","received_events_url":"https://api.github.com/users/ayende/received_events","type":"User","site_admin":false},"created_at":"2015-07-29T06:15:34Z","updated_at":"2015-07-29T06:15:34Z","body":"Opps, that isn't the reason they are failing.\n\n*Hibernating Rhinos Ltd  *\n\nOren Eini* l CEO l *Mobile: + 972-52-548-6969\n\nOffice: +972-4-622-7811 *l *Fax: +972-153-4-622-7811\n\n\n\nOn Wed, Jul 29, 2015 at 8:58 AM, Oren Eini (Ayende Rahien) <\nayende@ayende.com> wrote:\n\n> Raven.Tests.Storage.Voron.MappedResultsStorageActionsTests.DeleteScheduledReduction1(requestedStorage:\n> \"voron\") [FAIL]\n>    System.ArgumentOutOfRangeException : Specified argument was out of the\n> range of valid values.\n>    Parameter name: Unexpected reduce level 3\n>    Stack Trace:\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\RemainingReductionPerLevel.cs(39,0):\n> at Raven.Database.Storage.RemainingRed\n> uctionPerLevel.IncrementPerLevelCounters(Int32 level)\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\Voron\\StorageActions\\MappedResultsStorageActions.cs(497,0):\n> at Raven.Databa\n> se.Storage.Voron.StorageActions.MappedResultsStorageActions.ScheduleReductions(Int32\n> view, Int32 level, ReduceKeyAndBuck\n> et reduceKeysAndBuckets)\n>\n> c:\\Work\\ravendb\\Raven.Tests\\Storage\\Voron\\MappedResultsStorageActionsTests.cs(1355,0):\n> at Raven.Tests.Storage.Voro\n> n.MappedResultsStorageActionsTests.<DeleteScheduledReduction1>b__11d(IStorageActionsAccessor\n> accessor)\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\Voron\\TransactionalStorage.cs(247,0):\n> at Raven.Storage.Voron.TransactionalS\n> torage.ExecuteBatch(Action`1 action)\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\Voron\\TransactionalStorage.cs(229,0):\n> at Raven.Storage.Voron.TransactionalS\n> torage.Batch(Action`1 action)\n>\n> c:\\Work\\ravendb\\Raven.Tests\\Storage\\Voron\\MappedResultsStorageActionsTests.cs(1358,0):\n> at Raven.Tests.Storage.Voro\n> n.MappedResultsStorageActionsTests.DeleteScheduledReduction1(String\n> requestedStorage)\n>\n> Raven.Tests.Storage.Voron.MappedResultsStorageActionsTests.DeleteScheduledReduction1(requestedStorage:\n> \"esent\") [FAIL]\n>    System.ArgumentOutOfRangeException : Specified argument was out of the\n> range of valid values.\n>    Parameter name: Unexpected reduce level 3\n>    Stack Trace:\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\RemainingReductionPerLevel.cs(39,0):\n> at Raven.Database.Storage.RemainingRed\n> uctionPerLevel.IncrementPerLevelCounters(Int32 level)\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\Esent\\StorageActions\\MappedResults.cs(158,0):\n> at Raven.Database.Storage.Ese\n> nt.StorageActions.DocumentStorageActions.ScheduleReductions(Int32 view,\n> Int32 level, ReduceKeyAndBucket reduceKeysAndBuc\n> ket)\n>\n> c:\\Work\\ravendb\\Raven.Tests\\Storage\\Voron\\MappedResultsStorageActionsTests.cs(1355,0):\n> at Raven.Tests.Storage.Voro\n> n.MappedResultsStorageActionsTests.<DeleteScheduledReduction1>b__11d(IStorageActionsAccessor\n> accessor)\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\Esent\\TransactionalStorage.cs(838,0):\n> at Raven.Storage.Esent.TransactionalS\n> torage.ExecuteBatch(Action`1 action, EsentTransactionContext\n> transactionContext)\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\Esent\\TransactionalStorage.cs(817,0):\n> at Raven.Storage.Esent.TransactionalS\n> torage.Batch(Action`1 action)\n>\n> c:\\Work\\ravendb\\Raven.Tests\\Storage\\Voron\\MappedResultsStorageActionsTests.cs(1358,0):\n> at Raven.Tests.Storage.Voro\n> n.MappedResultsStorageActionsTests.DeleteScheduledReduction1(String\n> requestedStorage)\n>\n> Raven.Tests.Storage.Voron.MappedResultsStorageActionsTests.GetScheduledReductionForDebug(requestedStorage:\n> \"voron\") [FAI\n> L]\n>    System.ArgumentOutOfRangeException : Specified argument was out of the\n> range of valid values.\n>    Parameter name: Unexpected reduce level 3\n>    Stack Trace:\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\RemainingReductionPerLevel.cs(39,0):\n> at Raven.Database.Storage.RemainingRed\n> uctionPerLevel.IncrementPerLevelCounters(Int32 level)\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\Voron\\StorageActions\\MappedResultsStorageActions.cs(497,0):\n> at Raven.Databa\n> se.Storage.Voron.StorageActions.MappedResultsStorageActions.ScheduleReductions(Int32\n> view, Int32 level, ReduceKeyAndBuck\n> et reduceKeysAndBuckets)\n>\n> c:\\Work\\ravendb\\Raven.Tests\\Storage\\Voron\\MappedResultsStorageActionsTests.cs(1106,0):\n> at Raven.Tests.Storage.Voro\n> n.MappedResultsStorageActionsTests.<GetScheduledReductionForDebug>b__ea(IStorageActionsAccessor\n> accessor)\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\Voron\\TransactionalStorage.cs(247,0):\n> at Raven.Storage.Voron.TransactionalS\n> torage.ExecuteBatch(Action`1 action)\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\Voron\\TransactionalStorage.cs(229,0):\n> at Raven.Storage.Voron.TransactionalS\n> torage.Batch(Action`1 action)\n>\n> c:\\Work\\ravendb\\Raven.Tests\\Storage\\Voron\\MappedResultsStorageActionsTests.cs(1109,0):\n> at Raven.Tests.Storage.Voro\n> n.MappedResultsStorageActionsTests.GetScheduledReductionForDebug(String\n> requestedStorage)\n>\n> Raven.Tests.Storage.Voron.MappedResultsStorageActionsTests.GetScheduledReductionForDebug(requestedStorage:\n> \"esent\") [FAI\n> L]\n>    System.ArgumentOutOfRangeException : Specified argument was out of the\n> range of valid values.\n>    Parameter name: Unexpected reduce level 3\n>    Stack Trace:\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\RemainingReductionPerLevel.cs(39,0):\n> at Raven.Database.Storage.RemainingRed\n> uctionPerLevel.IncrementPerLevelCounters(Int32 level)\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\Esent\\StorageActions\\MappedResults.cs(158,0):\n> at Raven.Database.Storage.Ese\n> nt.StorageActions.DocumentStorageActions.ScheduleReductions(Int32 view,\n> Int32 level, ReduceKeyAndBucket reduceKeysAndBuc\n> ket)\n>\n> c:\\Work\\ravendb\\Raven.Tests\\Storage\\Voron\\MappedResultsStorageActionsTests.cs(1106,0):\n> at Raven.Tests.Storage.Voro\n> n.MappedResultsStorageActionsTests.<GetScheduledReductionForDebug>b__ea(IStorageActionsAccessor\n> accessor)\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\Esent\\TransactionalStorage.cs(838,0):\n> at Raven.Storage.Esent.TransactionalS\n> torage.ExecuteBatch(Action`1 action, EsentTransactionContext\n> transactionContext)\n>\n> c:\\Work\\ravendb\\Raven.Database\\Storage\\Esent\\TransactionalStorage.cs(817,0):\n> at Raven.Storage.Esent.TransactionalS\n> torage.Batch(Action`1 action)\n>\n> c:\\Work\\ravendb\\Raven.Tests\\Storage\\Voron\\MappedResultsStorageActionsTests.cs(1109,0):\n> at Raven.Tests.Storage.Voro\n> n.MappedResultsStorageActionsTests.GetScheduledReductionForDebug(String\n> requestedStorage)\n>\n>\n> *Hibernating Rhinos Ltd  *\n>\n> Oren Eini* l CEO l *Mobile: + 972-52-548-6969\n>\n> Office: +972-4-622-7811 *l *Fax: +972-153-4-622-7811\n>\n>\n>\n> On Tue, Jul 28, 2015 at 11:20 PM, Federico Andres Lois <\n> notifications@github.com> wrote:\n>\n>> This is the best I could do without seriously rethinking how map-reduce\n>> works. I must say that the constraints of the data model does not help the\n>> case. RavenDB-3724 is a must to be done for RavenDB rethinking the storage\n>> model.\n>>\n>> Another thing that will improve this is changing the storage to a binary\n>> representation as Json Serialization/Deserialization is insanely costly in\n>> this scenario.\n>>\n>> After I finish the prefix trees POC I will look into what we can do for\n>> multi-step map-reduce and if they could not be for 4.0 at least know what\n>> we need in order to not closing the door to it when we rethink the indexing\n>> storage model (which is a must for 4.0).\n>> ------------------------------\n>> You can view, comment on, or merge this pull request online at:\n>>\n>>   https://github.com/ayende/ravendb/pull/2497\n>> Commit Summary\n>>\n>>    - Indentation and typos.\n>>    - Performance improvements to the single and multi-level reduce.\n>>    - Improve performance using a explicit IEqualityComparer.\n>>    - Merge remote-tracking branch 'raven-authoritative/master'\n>>    - Debug breakdown will also give some addition estimated size in\n>>    megabytes of the trees.\n>>\n>> File Changes\n>>\n>>    - *M* Raven.Database/Indexing/BaseBatchSizeAutoTuner.cs\n>>    <https://github.com/ayende/ravendb/pull/2497/files#diff-0> (46)\n>>    - *M* Raven.Database/Indexing/MapReduceIndex.cs\n>>    <https://github.com/ayende/ravendb/pull/2497/files#diff-1> (12)\n>>    - *M* Raven.Database/Indexing/ReducingExecuter.cs\n>>    <https://github.com/ayende/ravendb/pull/2497/files#diff-2> (232)\n>>    - *M* Raven.Database/Storage/Esent/StorageActions/MappedResults.cs\n>>    <https://github.com/ayende/ravendb/pull/2497/files#diff-3> (24)\n>>    - *M* Raven.Database/Storage/IMappedResultsStorageAction.cs\n>>    <https://github.com/ayende/ravendb/pull/2497/files#diff-4> (51)\n>>    - *M*\n>>    Raven.Database/Storage/Voron/StorageActions/MappedResultsStorageActions.cs\n>>    <https://github.com/ayende/ravendb/pull/2497/files#diff-5> (23)\n>>    - *M* Raven.Tests.Issues/RavenDB_766.cs\n>>    <https://github.com/ayende/ravendb/pull/2497/files#diff-6> (4)\n>>    - *M* Raven.Tests.Issues/RavenDB_863.cs\n>>    <https://github.com/ayende/ravendb/pull/2497/files#diff-7> (4)\n>>    - *M* Raven.Tests/Storage/Voron/MappedResultsStorageActionsTests.cs\n>>    <https://github.com/ayende/ravendb/pull/2497/files#diff-8> (22)\n>>    - *M* Raven.Voron/Voron/Trees/TreeMutableState.cs\n>>    <https://github.com/ayende/ravendb/pull/2497/files#diff-9> (4)\n>>\n>> Patch Links:\n>>\n>>    - https://github.com/ayende/ravendb/pull/2497.patch\n>>    - https://github.com/ayende/ravendb/pull/2497.diff\n>>\n>> —\n>> Reply to this email directly or view it on GitHub\n>> <https://github.com/ayende/ravendb/pull/2497>.\n>>\n>\n>\n"}}