{"action":"created","issue":{"url":"https://api.github.com/repos/meteor/meteor/issues/4915","labels_url":"https://api.github.com/repos/meteor/meteor/issues/4915/labels{/name}","comments_url":"https://api.github.com/repos/meteor/meteor/issues/4915/comments","events_url":"https://api.github.com/repos/meteor/meteor/issues/4915/events","html_url":"https://github.com/meteor/meteor/issues/4915","id":99864496,"number":4915,"title":"Separating mongo, ddp, and accounts-base","user":{"login":"ccorcos","id":1794527,"avatar_url":"https://avatars.githubusercontent.com/u/1794527?v=3","gravatar_id":"","url":"https://api.github.com/users/ccorcos","html_url":"https://github.com/ccorcos","followers_url":"https://api.github.com/users/ccorcos/followers","following_url":"https://api.github.com/users/ccorcos/following{/other_user}","gists_url":"https://api.github.com/users/ccorcos/gists{/gist_id}","starred_url":"https://api.github.com/users/ccorcos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ccorcos/subscriptions","organizations_url":"https://api.github.com/users/ccorcos/orgs","repos_url":"https://api.github.com/users/ccorcos/repos","events_url":"https://api.github.com/users/ccorcos/events{/privacy}","received_events_url":"https://api.github.com/users/ccorcos/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-08-09T05:01:08Z","updated_at":"2015-08-11T17:01:48Z","closed_at":null,"body":"These three packages are surprisingly intertwined:\r\n\r\n`ddp` handles `Meteor.methods`, `Meteor.publish` and `Meteor.subscribe`. `Meteor.publish` and `Meteor.subscribe` assume the existence of `mongo` and `minimongo` to move data from server to client. `accounts-base` assumes the existence of `mongo` and `minimongo` to store user records on the client and the server. And lastly, `ddp` gives you `this.userId` within `Meteor.publish` and `Meteor.methods` which involves everything above.\r\n\r\nAfter building `ccorcos:any-db`, I realized that for Meteor to really support other databases well, there needs to be some more fundamental changes to the underlying core packages. The package names are entirely up for debate, but here's what I want to propose:\r\n\r\n- ddp-methods\r\n- ddp-pub\r\n- ddp-sub\r\n- accounts-generic\r\n- ddp-sub-mongo\r\n- accounts-mongo\r\n\r\n## `ddp-methods`\r\n\r\nIt seems to me like `Meteor.methods` is a completely separate functionality from `Meteor.publish` and `Meteor.subscribe` so I think it would be reasonable to pull this out into its own package. Also, some people like @msavin have been blogging about how to make Meteor less reactive so it would be cool to have a lean Meteor app without pub-sub if you wanted.\r\n\r\n## `ddp-pub`\r\n\r\nFor databases with reactive drivers like `mongo` that support `Cursor.observeChanges`, then we can publish the the same way we're used to. However, for databases that aren't reactive, we can use the `diff-sequence` package to poll and diff the query results. And if we cache these subscriptions based on their query, we can proactively trigger a refresh whenever the data is changed. This is what I did in `ccorcos:any-db` and its actually really simple. The only thing missing here is [DDP doesn't currently support ordered publications](https://github.com/meteor/meteor/blob/devel/packages/ddp/DDP.md#procedure-2). I built [a hack for `ccorcos:any-db` to send ordered collections](https://github.com/ccorcos/meteor-any-db/blob/master/src/db.coffee#L149), but this is not ideal.\r\n\r\nI'm imagining something that looks more like this:\r\n\r\n```js\r\nMeteor.publish('messages', {ordered:true, cursor:false}, function(roomId) {\r\n return Messages.find({roomId:roomId}).fetch()\r\n})\r\n```\r\n\r\nAnd in a `Meteor.method`, you can trigger the subscription to be refreshed on a write.\r\n\r\n```js\r\nMeteor.methods({ \r\n  addMsg:function(roomId, text){ \r\n    if (Meteor.isServer) {\r\n      Messages.insert({roomId:roomId, text:text})\r\n      Meteor.update('messages', roomId)\r\n    }\r\n  } \r\n});\r\n```\r\n\r\n## `ddp-sub-mongo`\r\n\r\n`ddp-sub-mongo` is exactly the same functionality we have right now with minimongo. All data is thrown into the appropriate `minimongo` collection and we can query the collection to get results out. This works great most of the time, but there a two very related pain points that I want to address:\r\n\r\n1) It can be impossible to separate which data belongs to which subscription on the client. For example:\r\n\r\n```js\r\nProducts = new Mongo.Collection('products')\r\nMeteor.subscribe('bestProducts')\r\nMeteor.subscribe('worstProducts')\r\nProducts.findOne() // => {_id:'1234', name: 'Crayola Colored Bubbles'}\r\n```\r\n\r\n2) Some complicated database queries can sort data based on information that isn't available on the client, thus it can be impossible to sort the data appropriately.\r\n\r\nThese two issues can be overcome by being able to publish ordered collections and keeping track of which documents belong to which (possibly multiple) subscriptions -- which leads me to...\r\n\r\n## `ddp-sub`\r\n\r\nThis package allows you to subscribe to ordered collections and keeps your subscriptions separate from each other.\r\n\r\nHere's a `ddp-sub` example with Blaze:\r\n\r\n```jsx\r\nTemplate.chatroom.onRendered(function() {\r\n this.autorun(() => {\r\n   this.messages = Meteor.subscribe('messages', this.currentData().roomId)\r\n })\r\n})\r\nTemplate.chatroom.helpers({\r\n messages: function() {\r\n   return Template.instance().messages.fetch()\r\n }\r\n})\r\n```\r\n\r\nHere's a `ddp-sub` example with React:\r\n\r\n```jsx\r\nReact.createClass({\r\n mixins: [MeteorDataMixin],\r\n getMeteorData: function() {\r\n   const messages = Meteor.subscribe('messages', this.props.roomId)\r\n   return {messages: messages.fetch()}\r\n },\r\n // ...\r\n})\r\n```\r\n\r\n## `accounts-generic`\r\n\r\nI don't know a lot about how the accounts package works so I could be wrong, but it seems to me like a generic accounts package could expose an API to use any storage method on the client and the server.\r\n\r\n```js\r\n// on the server\r\nUsers = new Mongo.Collection('custom-users')\r\nAccounts.set(function(record) {\r\n Users.upsert({_id: record._id}, record)\r\n})\r\nAccounts.get(function(userId) {\r\n Users.findOne(userId)\r\n})\r\n// on the client\r\nAccounts.set(function(cookie) {\r\n saveCookie(cookie)\r\n})\r\nAccounts.get(function() {\r\n return getCookie()\r\n})\r\n```\r\n\r\nThe ddp packages should be able to use these methods to authenticate users and provide `this.userId` in `Meteor.methods` and `Meteor.publish`. In addition to being able to use any storage method you want, I think devs with existing stacks would be much more open to adopting Meteor if they didn't have to migrate their entire user database to use Meteor's accounts packages.\r\n\r\n## Implications\r\n\r\n- Using `ddp-pub-mongo` and `accounts-mongo`, we should get the same exact functionality we have right now.\r\n- It should possible to user `mongo` on the server without `minimongo` on the client using `ddp-sub` instead of `ddp-sub-mongo`.\r\n\r\n### P.S. Latency Compensation\r\n\r\nThis is the only ugly part of all of this. Mongo and Minimongo do a great job at latency compensation and reactivity. Its a huge feat, but it doesnt seem likely to happen for other databases anytime soon. I dont have a perfect solution here but I have something that works.\r\n\r\nBasically, I buffer the subscription data with a ReactiveVar:\r\n\r\n```jsx\r\nvar messages = new ReactiveVar([])\r\nTracker.autorun(() => {\r\n  messagesSub = Meteor.subscribe('messages', roomId)\r\n  Tracker.autorun(() => {\r\n    messages.set(this.messagesSub.fetch())\r\n  })\r\n})\r\n```\r\n\r\nThen, I can latency compensate with the ReactiveVar however I want:\r\n\r\n```jsx\r\nfunction submitMsg(roomId, text) {\r\n  var prevMsgs = messages.get()\r\n  messages.set([{roomId, text, unverified:true}].concat(prevMsgs))\r\n  Meteor.call('addMsg', roomId, text, function(err) {\r\n    if (err) {\r\n      messages.set(prevMsgs)\r\n    }\r\n    // without an error, the subscription should update the messages\r\n  })\r\n}\r\n```\r\n\r\n---\r\n\r\nAnyways, I just wanted to get a discussion started here. What are you thoughts?\r\n"},"comment":{"url":"https://api.github.com/repos/meteor/meteor/issues/comments/129968828","html_url":"https://github.com/meteor/meteor/issues/4915#issuecomment-129968828","issue_url":"https://api.github.com/repos/meteor/meteor/issues/4915","id":129968828,"user":{"login":"rclai","id":1316261,"avatar_url":"https://avatars.githubusercontent.com/u/1316261?v=3","gravatar_id":"","url":"https://api.github.com/users/rclai","html_url":"https://github.com/rclai","followers_url":"https://api.github.com/users/rclai/followers","following_url":"https://api.github.com/users/rclai/following{/other_user}","gists_url":"https://api.github.com/users/rclai/gists{/gist_id}","starred_url":"https://api.github.com/users/rclai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rclai/subscriptions","organizations_url":"https://api.github.com/users/rclai/orgs","repos_url":"https://api.github.com/users/rclai/repos","events_url":"https://api.github.com/users/rclai/events{/privacy}","received_events_url":"https://api.github.com/users/rclai/received_events","type":"User","site_admin":false},"created_at":"2015-08-11T17:01:48Z","updated_at":"2015-08-11T17:01:48Z","body":"What? You can call `.fetch()` on the current `Meteor.subscribe`?"}}