{"action":"created","issue":{"url":"https://api.github.com/repos/strongloop/loopback/issues/986","labels_url":"https://api.github.com/repos/strongloop/loopback/issues/986/labels{/name}","comments_url":"https://api.github.com/repos/strongloop/loopback/issues/986/comments","events_url":"https://api.github.com/repos/strongloop/loopback/issues/986/events","html_url":"https://github.com/strongloop/loopback/issues/986","id":53859446,"number":986,"title":"Race condition in the replication algorithm","user":{"login":"bajtos","id":1140553,"avatar_url":"https://avatars.githubusercontent.com/u/1140553?v=3","gravatar_id":"","url":"https://api.github.com/users/bajtos","html_url":"https://github.com/bajtos","followers_url":"https://api.github.com/users/bajtos/followers","following_url":"https://api.github.com/users/bajtos/following{/other_user}","gists_url":"https://api.github.com/users/bajtos/gists{/gist_id}","starred_url":"https://api.github.com/users/bajtos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bajtos/subscriptions","organizations_url":"https://api.github.com/users/bajtos/orgs","repos_url":"https://api.github.com/users/bajtos/repos","events_url":"https://api.github.com/users/bajtos/events{/privacy}","received_events_url":"https://api.github.com/users/bajtos/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/strongloop/loopback/labels/%23fib-13","name":"#fib-13","color":"006b75"},{"url":"https://api.github.com/repos/strongloop/loopback/labels/%23plan","name":"#plan","color":"ededed"},{"url":"https://api.github.com/repos/strongloop/loopback/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/strongloop/loopback/labels/replication","name":"replication","color":"5319e7"}],"state":"open","locked":false,"assignee":{"login":"bajtos","id":1140553,"avatar_url":"https://avatars.githubusercontent.com/u/1140553?v=3","gravatar_id":"","url":"https://api.github.com/users/bajtos","html_url":"https://github.com/bajtos","followers_url":"https://api.github.com/users/bajtos/followers","following_url":"https://api.github.com/users/bajtos/following{/other_user}","gists_url":"https://api.github.com/users/bajtos/gists{/gist_id}","starred_url":"https://api.github.com/users/bajtos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bajtos/subscriptions","organizations_url":"https://api.github.com/users/bajtos/orgs","repos_url":"https://api.github.com/users/bajtos/repos","events_url":"https://api.github.com/users/bajtos/events{/privacy}","received_events_url":"https://api.github.com/users/bajtos/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/strongloop/loopback/milestones/17","html_url":"https://github.com/strongloop/loopback/milestones/%23Epic:%20Offline%20Sync%20V1","labels_url":"https://api.github.com/repos/strongloop/loopback/milestones/17/labels","id":928316,"number":17,"title":"#Epic: Offline Sync V1","description":null,"creator":{"login":"bajtos","id":1140553,"avatar_url":"https://avatars.githubusercontent.com/u/1140553?v=3","gravatar_id":"","url":"https://api.github.com/users/bajtos","html_url":"https://github.com/bajtos","followers_url":"https://api.github.com/users/bajtos/followers","following_url":"https://api.github.com/users/bajtos/following{/other_user}","gists_url":"https://api.github.com/users/bajtos/gists{/gist_id}","starred_url":"https://api.github.com/users/bajtos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bajtos/subscriptions","organizations_url":"https://api.github.com/users/bajtos/orgs","repos_url":"https://api.github.com/users/bajtos/repos","events_url":"https://api.github.com/users/bajtos/events{/privacy}","received_events_url":"https://api.github.com/users/bajtos/received_events","type":"User","site_admin":false},"open_issues":8,"closed_issues":5,"state":"open","created_at":"2015-01-09T09:26:13Z","updated_at":"2015-03-04T17:45:47Z","due_on":null,"closed_at":null},"comments":5,"created_at":"2015-01-09T11:38:27Z","updated_at":"2015-03-04T17:48:16Z","closed_at":null,"body":"When two clients are preforming replication at the same time and the same object was changed in both clients, some of the changes may get lost. The replication algorithm should check the data before the update to verify it was not changed by another party during the replication process. This will most likely require LoopBack to support transactions for SQL datasources and [Update if Current](http://docs.mongodb.org/manual/tutorial/isolate-sequence-of-operations/#update-if-current] for MongoDB.\r\n\r\nAnother example are external changes. At the moment, there is a timer-based hook to detect any changes made by non-loopback applications and update change tracking records. If the sync algorithm is started before this timed check was run, changes made by the external writer may get discarded. "},"comment":{"url":"https://api.github.com/repos/strongloop/loopback/issues/comments/77208027","html_url":"https://github.com/strongloop/loopback/issues/986#issuecomment-77208027","issue_url":"https://api.github.com/repos/strongloop/loopback/issues/986","id":77208027,"user":{"login":"bajtos","id":1140553,"avatar_url":"https://avatars.githubusercontent.com/u/1140553?v=3","gravatar_id":"","url":"https://api.github.com/users/bajtos","html_url":"https://github.com/bajtos","followers_url":"https://api.github.com/users/bajtos/followers","following_url":"https://api.github.com/users/bajtos/following{/other_user}","gists_url":"https://api.github.com/users/bajtos/gists{/gist_id}","starred_url":"https://api.github.com/users/bajtos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bajtos/subscriptions","organizations_url":"https://api.github.com/users/bajtos/orgs","repos_url":"https://api.github.com/users/bajtos/repos","events_url":"https://api.github.com/users/bajtos/events{/privacy}","received_events_url":"https://api.github.com/users/bajtos/received_events","type":"User","site_admin":false},"created_at":"2015-03-04T17:48:16Z","updated_at":"2015-03-04T17:48:16Z","body":"For v1.0, I would like to implement the following solution, which should be bullet-proof and support all existing replication use-cases, although at the cost of lower performance. Later in the future, we can implement optimised variants.\r\n\r\n - The `diff` step will return the current revisions of the data in the database (what we think we have) for each delta.\r\n - The instructions for \"bulkUpdate\" will include this \"baseline\" revision.\r\n - The implementation of \"bulkUpdate\" will update a record only if it has the same revision as sent in the instructions.\r\n\r\nThe initial implementation of the last step (unoptimised but generic) for a single model instance:\r\n\r\n 1. Query the database to get the current instance \"data\" via Model.findById(id)\r\n 2. Compute the current revision of the model instance from \"data\"\r\n 3. If the computed revision does not match the revision from bulkUpdate, then fail with a conflict.\r\n 4. Run \"updateAll\", using \"data\" as the where-filter and the values from bulkUpdate instructions as the values to set.\r\n 5. If updateAll modified one row, then we are done. Otherwise fail with a conflict, because somebody has updated the instance under our hands.\r\n\r\nWhen a conflict is detected, we should probably run \"rectify change\" for the conflicted model.\r\n\r\nIn many cases, it may be possible to automatically repeat the bulkUpdate() and/or replicate() until it passes. However, this will be probably omitted from the initial implementation for v1.0.\r\n\r\n@ritch @raymondfeng Thoughts?\r\n\r\nI have also one meta question for you. Should we run Operation hooks for the DAO calls made from `bulkUpdate`? I am inclining to disable these hooks, or at least the \"before save\" hook, because if they modify the data being saved (typically change the \"modified\" property to the current date time), then the replicated data will be different from the original.\r\n\r\nAnother question is how to deal with ACLs, I have opened a new issue for that: #1166"}}