{"action":"closed","issue":{"url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/5030","labels_url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/5030/labels{/name}","comments_url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/5030/comments","events_url":"https://api.github.com/repos/HabitRPG/habitrpg/issues/5030/events","html_url":"https://github.com/HabitRPG/habitrpg/issues/5030","id":68585242,"number":5030,"title":"Mongoose Dictionary Type","user":{"login":"lefnire","id":195202,"avatar_url":"https://avatars.githubusercontent.com/u/195202?v=3","gravatar_id":"","url":"https://api.github.com/users/lefnire","html_url":"https://github.com/lefnire","followers_url":"https://api.github.com/users/lefnire/followers","following_url":"https://api.github.com/users/lefnire/following{/other_user}","gists_url":"https://api.github.com/users/lefnire/gists{/gist_id}","starred_url":"https://api.github.com/users/lefnire/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lefnire/subscriptions","organizations_url":"https://api.github.com/users/lefnire/orgs","repos_url":"https://api.github.com/users/lefnire/repos","events_url":"https://api.github.com/users/lefnire/events{/privacy}","received_events_url":"https://api.github.com/users/lefnire/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"paglias","id":589911,"avatar_url":"https://avatars.githubusercontent.com/u/589911?v=3","gravatar_id":"","url":"https://api.github.com/users/paglias","html_url":"https://github.com/paglias","followers_url":"https://api.github.com/users/paglias/followers","following_url":"https://api.github.com/users/paglias/following{/other_user}","gists_url":"https://api.github.com/users/paglias/gists{/gist_id}","starred_url":"https://api.github.com/users/paglias/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/paglias/subscriptions","organizations_url":"https://api.github.com/users/paglias/orgs","repos_url":"https://api.github.com/users/paglias/repos","events_url":"https://api.github.com/users/paglias/events{/privacy}","received_events_url":"https://api.github.com/users/paglias/received_events","type":"User","site_admin":false},"milestone":null,"comments":11,"created_at":"2015-04-15T06:03:06Z","updated_at":"2015-07-10T13:24:49Z","closed_at":"2015-07-10T13:24:49Z","body":"To complete the tasks=>collection migration, we'll need a new Mongoose type `Dictionary`. Really this type would come in handy in many other situations, but it's quite required for the T=>C migration.\r\n\r\nHere's the story. In MongoDB, storing arrays turns out to be stupid. Fine for very small arrays (eg, `user.preferences.hooks`); terrible for very large or commonly-accessed arrays (eg, `user.habits`). What happens is, in order to prevent race conditions on array indices, Mongoose adds an extra `._v` parameter to the query on array updates. This slows down queries tremendously, creating major scalability bottlenecks at the DB layer. The solution is to store arrays as _objects_. Then in order to treat those objects as lists, you add a `sort` field for sorting, and some app logic (typically using Lodash's collections API). More and more you see vendors stop supporting arrays at the DB layer and instead use objects (eg, [Firebase's post](https://www.firebase.com/blog/2014-04-28-best-practices-arrays-in-firebase.html)).\r\n\r\nSo as an example, instead of storing `user.habits` as an array:\r\n\r\n```javascript\r\nhabits: [{\r\n  value: Number,\r\n  text: String,\r\n  ...\r\n}]\r\n```\r\n\r\nwe want to store it as an object:\r\n\r\n```javascript\r\nhabits: {\r\n  <habit-id>: {\r\n    value: Number,\r\n    text: String,\r\n    sort: Number,\r\n    ...\r\n  }\r\n}\r\n```\r\n\r\nWhich is fine and dandy. The problem here is that `<habit-id>`. Those familiar with Mongoose know you can't do that - you can't have dictionaries, where a dynamic key maps to an object with a defined schema. What you _can_ do is have `habits: {type: Schema.Types.Mixed, 'default': {} }`, but then you don't get Schema validation, default values, `path.markModified()`, etc. and you have to validate your dictionaries manually. You can see this at play presently in a few situations: [quests](https://github.com/HabitRPG/habitrpg/blob/develop/website/src/models/user.js#L50), [filters](https://github.com/HabitRPG/habitrpg/blob/develop/website/src/models/user.js#L91), [messages](https://github.com/HabitRPG/habitrpg/blob/develop/website/src/models/user.js#L384), and many more. In these situations we handle validation in app logic, and trigger `.markModified()` manually.\r\n\r\nSo from my searching, there's no Mongoose support for this style of Dictionary attributes, nor any existing Mongoose plugins for such (someone may wanna double check in case my Google Fu is failing me). The task at hand then is to create a new [Mongoose plugin](http://mongoosejs.com/docs/plugins.html) to add a new type called Dictionary. You can see how to add new types with [this example](https://github.com/bnoguchi/mongoose-types/blob/master/lib/types/url.js). Existing [types here](https://github.com/Automattic/mongoose/tree/master/lib/types), a likely type to copy/paste from is [Array](https://github.com/Automattic/mongoose/blob/master/lib/types/array.js). Model defs then might look like the following:\r\n\r\n```javascript\r\nhabits: {type: Dictionary, schema: {\r\n  value: Number,\r\n  text: String,\r\n  ...\r\n}}\r\n```\r\n\r\nwhere the key is dynamic, the schema is handled as normally the case in Mongoose (pre-save validation, type-checking, etc), and `.markModified()` is handled automatically. Further, it may be useful to add support \"value dictionaries\" (for lack of a better word) where the dictionary doesn't map to an object, but a single value. Eg, `user.filters` is an object like `{<uuid>:true, <uuid>:false}` ([code](https://github.com/HabitRPG/habitrpg/blob/develop/website/src/models/user.js#L91)). In this case, we may extend our Dictionary type so that it would work as such:\r\n\r\n```javascript\r\nfilters: {type:Dictionary, schema:Boolean}\r\n```\r\n\r\nIdeally this dictionary type would be open-sourced to the mongoose plugin community (see [\"Community!\"](http://mongoosejs.com/docs/plugins.html)), maybe as mongoose-dictionary-type or something.\r\n\r\n<bountysource-plugin>\r\n---\r\nWant to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/11182846-mongoose-dictionary-type?utm_campaign=plugin&utm_content=tracker%2F68393&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F68393&utm_medium=issues&utm_source=github).\r\n</bountysource-plugin>"}}