{"action":"created","issue":{"url":"https://api.github.com/repos/uxbox/front/issues/60","labels_url":"https://api.github.com/repos/uxbox/front/issues/60/labels{/name}","comments_url":"https://api.github.com/repos/uxbox/front/issues/60/comments","events_url":"https://api.github.com/repos/uxbox/front/issues/60/events","html_url":"https://github.com/uxbox/front/pull/60","id":106882054,"number":60,"title":"Overhaul","user":{"login":"dialelo","id":409039,"avatar_url":"https://avatars.githubusercontent.com/u/409039?v=3","gravatar_id":"","url":"https://api.github.com/users/dialelo","html_url":"https://github.com/dialelo","followers_url":"https://api.github.com/users/dialelo/followers","following_url":"https://api.github.com/users/dialelo/following{/other_user}","gists_url":"https://api.github.com/users/dialelo/gists{/gist_id}","starred_url":"https://api.github.com/users/dialelo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dialelo/subscriptions","organizations_url":"https://api.github.com/users/dialelo/orgs","repos_url":"https://api.github.com/users/dialelo/repos","events_url":"https://api.github.com/users/dialelo/events{/privacy}","received_events_url":"https://api.github.com/users/dialelo/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-09-16T23:08:39Z","updated_at":"2015-09-17T17:44:29Z","closed_at":"2015-09-17T16:10:35Z","pull_request":{"url":"https://api.github.com/repos/uxbox/front/pulls/60","html_url":"https://github.com/uxbox/front/pull/60","diff_url":"https://github.com/uxbox/front/pull/60.diff","patch_url":"https://github.com/uxbox/front/pull/60.patch"},"body":"- [x] These changes depend the unreleased cats 1.0.0 version\r\n\r\nThis PR is a overhaul of state management, code organization and event management. I've broken a few things in the process, which I document later in this PR. I will create issues with such problems when this PR gets merged and I think it will be better to merge it before fixing other bugs that we may encounter testing this.\r\n\r\n## Code organization\r\n\r\nCode has been splitted without mercy, I've moved all the UI-related code under a package named ui to better separate the \"domain logic\" from views. Also, UI relations and dependencies are very different from those in the domain.\r\n\r\nThese are the current directories, I've added a comment in each explaining what is in there:\r\n\r\n```\r\n├── data          ;; db creation and restore, schema\r\n├── log           ;; event log and log record mechanism\r\n├── projects      ;; project actions, data and queries\r\n├── shapes        ;; shape types, actions, data and queries\r\n├── users         ;; user actions, data, and queries\r\n└── ui\r\n    ├── canvas    ;; same as before, moved one level up\r\n    ├── icons\r\n    ├── icon_sets\r\n    ├── streams   ;; UI streams of events (more on this later)\r\n    └── workspace\r\n```\r\n\r\n## State management\r\n\r\n### Farewell, global atom\r\n\r\nI've gotten rid of the global atom. It is not bad per se but when having lots of relational data (projects have pages, pages have shapes, pages have comments, shapes have comments, ...) it gets really tedious to manually manage references when writing to the atom; querying the data is another pain when having a map of maps in the global atom.\r\n\r\nThe code under the old uxbox.storage was doing a lot of writes when processing events and keeping materialized views of the data in several atoms. I've replaced that with usage of a [datascript](https://github.com/tonsky/datascript) DB which buys us (optional) schemas, serializable state, querying and several types of indexes for optimizing reads. \r\n\r\n### Database\r\n\r\nThe database creation logic is in `uxbox.data`, there are two namespaces in there:\r\n\r\n- [`schema`](https://github.com/uxbox/front/tree/datascript-storage-and-querying/src/uxbox/data/schema.cljs): the schema of all the data we're going to put in the database. This is good for mantainabilty\r\n  and documentation since every entity attributes and references are documented.\r\n- [`db`](https://github.com/uxbox/front/tree/datascript-storage-and-querying/src/uxbox/data/db.cljs): db creation from schema, serialize/deserialize to arbitrary storages (transient, local storage,\r\n  session storage, ...)\r\n\r\nNow the views instead of receiving the global atom (or cursor to the global atom) for getting application state they get a connection they can use for querying the data and recording events on the log, more on those two things later.\r\n\r\nTake a look at the [`uxbox.data` tests](https://github.com/uxbox/front/tree/datascript-storage-and-querying/test/uxbox/test/data_test.cljs) to get a better idea.\r\n\r\n### The log\r\n\r\nThe log is the log of events that happen in the application. Events are thinks like \"create project\", \"create shape\" and such; they haven't changed very much. Events have a type (namespaced keyword like :uxbox/create-project), payload (an arbitrary data structure), a timestamp and and author (full user info at the moment). UI events such as mouse clicks are not logged at the moment but there is no reason they couldn't be, for being able to replay UI event streams that have happened in any client.\r\n\r\nThe [`uxbox.log.core`](https://github.com/uxbox/front/tree/datascript-storage-and-querying/src/uxbox/log/core.cljs) namespace contains:\r\n- all event types (this is mostly for reference), examples are `:uxbox/create-project` or `:uxbox/toggle-shape-visibility`.\r\n- A constructor for events given a type, payload, author and timestamp.\r\n- A function `record!` that, given a connection to a database, event type and payload changes the database\r\n  according to the event. It also records the event into the database.\r\n- A function `persist!` that, given a connection to a database and event changes the database according to\r\n  the event.\r\n- a multimethod `to-datoms` that, given an event an a immutable DB returns a vector of datascript datoms\r\n  that would change the state of the DB after this event.\r\n\r\nI've gotten rid of the asynchronous pubsub for recording events and event interpretation (changing the database) is synchronous. Now actions use `uxbox.log.core/record!` for appending events to the log and causing state changes.\r\n\r\nAn example of how recording log events looks like:\r\n\r\n```clojure\r\n(uxbox.log.core/record! :uxbox/change-page [page-id :page/title \"Foo\"])\r\n```\r\n\r\n## Data\r\n\r\nThere are three \"domain\" packages in UXBox currently:\r\n\r\n- `uxbox.projects`: project and page management.\r\n- `uxbox.shapes`: shape types and functions.\r\n- `uxbox.users`: user data.\r\n\r\nAll the data structures have a schema in `uxbox.data.schema`.\r\n\r\n### `uxbox.projects`\r\n\r\nThe package about projects and pages. It has three modules:\r\n\r\n- `uxbox.projects.actions`: high-level API for recording project events on the log.\r\n- `uxbox.projects.data`: high-level constructors of project data and conversion of project\r\n  log events to datoms.\r\n- `uxbox.projects.queries`: queries for project data.\r\n\r\n### `uxbox.shapes`\r\n\r\nThis is very interesting, it has `actions` and `queries` like `uxbox.projects` but also a lot of shape types like line, circle, rectangle and several protocols for drawing them. I want to make further changes on how shapes, drawing tools and icons work but that's for another PR :stuck_out_tongue:\r\n\r\nOnly small refactorings here. I'm going to work on improvements in this area after this has been merged.\r\n\r\n### `uxbox.users`\r\n\r\nA fake query of the current user right now. In the future it will be used for querying current user and other user's data.\r\n\r\n## Components\r\n\r\nThe UI has been moved under `uxbox.ui`.\r\n\r\nThe main application component is mounted once, listening for changes in location (now an atom in `uxbox.navigation`) and rendering top-level components depending on URL. It is passed the connection and the location. Location is not passed to top-level components, just the connection (when required) and params.\r\n\r\nAll components under the main component are reactive and update themselves.\r\n\r\nThe main changes here are:\r\n- UI and transient state is now local to the components since global atom has dissapeared. We can\r\n  store even component state in datascript if we want to.\r\n- There are components that react to queries, this is acomplished with mixins found in `uxbox.ui.mixin`.\r\n\r\n### Queries\r\n\r\nThis is an example of a component that reacts to a query, note that it must receive the connection (`conn`) as the first argument and the components is stateful, defined with `rum/defcs`. The results of pulling the query data are passed with the given key (`:events`) in the state inside an atom. The view will re-render whenever the result of the query changes.\r\n\r\nNote that the activity timeline is querying the event log directly and massaging the data for displaying it.\r\n\r\n```clojure\r\n(rum/defcs activity-timeline < (mx/pull-query\r\n                                :events\r\n                                q/all-events\r\n                                '[:event/type\r\n                                  :event/payload\r\n                                  :event/author\r\n                                  :event/timestamp])\r\n  [{events :events} conn]\r\n  [:aside#activity-bar.activity-bar\r\n   [:div.activity-bar-inside\r\n    [:h4 \"ACTIVITY\"]\r\n    (for [[date items] (activities-by-date @events)]\r\n      (concat\r\n       [[:span.date-ribbon\r\n         {:key date}\r\n         (day date)]]\r\n       (map #(activity-item (materialize conn %)) items)))]])\r\n```\r\n\r\n### Commands\r\n\r\nBefore, the pub-sub system decided when an element was selected or had to be drawn. Now those values (moving shapes, currently drawing shape) are calculated using event streams (more on this later) and the results (move selected shapes, finish drawing stroke for currently drawing shape) are delivered to atoms. The results are commited by the views using commands.\r\n\r\nHere is an excerpt from the `uxbox.ui.canvas` namespace:\r\n\r\n\r\n```clojure\r\n(def draw! (s/pipe-to-atom cs/draw-stream))\r\n\r\n(rum/defc canvas < ;; ...\r\n                   (mx/cmds-mixin\r\n                    [::draw draw! (fn [[conn page] shape]\r\n                                    (actions/draw-shape conn page shape))]\r\n\r\n                    [::move move! (fn [[conn] selections]\r\n                                    (actions/update-shapes conn selections))])\r\n  [conn\r\n   page\r\n   shapes]\r\n   ;; ...\r\n)\r\n```\r\n\r\nThe `draw!` atom contains the values pushed into the `cs/draw-stream` stream. Whenever a value must be drawn, such value will appear in `cs/draw-stream` and be piped in the `draw!` atom. This will cause the the callback function to be called with the `canvas` view's arguments, having access to the connection and current context\r\nto perform the action.\r\n\r\nThe canvas view pushes primitive mouse up and down events into streams that are manipulated until deriving the draw stream. It contains shapes after the user has drawn them in a single keystroke.\r\n\r\n## Event management\r\n\r\n### Basic abstractions\r\n\r\nI've used the [BaconJS library](https://baconjs.github.io/) for compositional event streams. It has three main abstractions, we can observe for values in every one of them:\r\n\r\n- Property: A continuous value that can be sampled and combined\r\n- EventStream: A stream of succesful or error values. It can end.\r\n- Bus: A bus where we can put values, errors, end it and plug input streams to.\r\n\r\nThere are wrapper functions under `uxbox.streams` that cover a fair amount of the Bacon API but there is much left, we can implement wrapper functions as needed. The implementation is [well tested](https://github.com/uxbox/front/tree/datascript-storage-and-querying/test/uxbox/test/streams_test.cljs).\r\n\r\nIt includes a `transform` function for applying a transducer to an observable :tada:\r\n\r\n### Streams\r\n\r\nI used streams implementing multiselect, dragging and drawing (see `uxbox.ui.canvas.streams`). The canvas view feeds events to a bus and it can be combined with plenty of combinators that BaconJS provides.\r\n\r\nLet's see an example of how the drawing stream is calculated.\r\n\r\nFirst of all, drawing happens when there is a selected tool and the user clicks on the canvas. In that moment we want to grab the selected tool and the current coordinates and create a shape for drawing. Let's put that into an stream:\r\n\r\n```clojure\r\n(def start-drawing-stream\r\n  (s/sampled-by\r\n   (s/combine\r\n    (fn [tool coords]\r\n      (tools/start-drawing tool coords))\r\n    selected-tool-stream\r\n    canvas-coordinates-stream)\r\n   (s/and tool-selected? mouse-down?)))\r\n```\r\n\r\nAfter we have the shape to start drawing, the user will likely move the mouse while is down and adjust the size. We can scan the values that happen during that moment and update the shape during the stroke:\r\n\r\n```clojure\r\n(def drawing-stream\r\n  (s/flat-map start-drawing-stream\r\n              (fn [shape]\r\n                (let [stroke (s/take-while (clamp-coords canvas-coordinates)\r\n                                           mouse-down?)]\r\n                  (s/scan (fn [s [x y]]\r\n                            (shapes/draw s x y))\r\n                          shape\r\n                          stroke)))))\r\n```\r\n\r\nThe `flat-map` operation is like a nested for loop for event streams. Every time a shape is found in `start-drawing-stream` starts to return values from a newly created stream. The stroke is captured as the mouse coordinates during the time the mouse is down. Scanning all mouse coordinates we are able to update the shape as it's being moved in the screen.\r\n\r\nNow we want to derive a stream that gives us the values of the shapes that we are drawing when we commit them with a mouse up, easy:\r\n\r\n```clojure\r\n(def draw-stream\r\n  (s/sampled-by drawing-stream\r\n    (s/true? mouse-up?)))\r\n\r\n(def draw! (s/pipe-to-atom draw-stream))\r\n```\r\n\r\nAnd we also want to derive the stream of the \"current drawing shape\". After mouse up we don't want to display the current drawing shape anymore since it will be already drawn.\r\n\r\n```clojure\r\n(def draw-in-progress\r\n  (s/merge\r\n    drawing-stream\r\n    (s/map (constantly nil)  mouse-up-stream)))\r\n```\r\n\r\nThis is later used in the canvas view:\r\n\r\n```clojure\r\n(def drawing (s/pipe-to-atom cs/draw-in-progress))\r\n\r\n;; in canvas component\r\n(when-let [shape (rum/react drawing)]\r\n  (shapes/shape->drawing-svg shape))\r\n```\r\n\r\nI think this will be a good tool for more complex interactions.\r\n\r\n## More\r\n\r\n### Regressions\r\n\r\nThere are a few regressions that I know of, I will be opening issues after this gets merged:\r\n\r\n- Icon selection & drawing\r\n- Element options\r\n- Page count in dashboard\r\n- Zoom (implement with event streams)\r\n- Key bindings (should be easy to add back with streams)\r\n- Components tool (did it work?)\r\n- Rectangle and icon drawing with negative deltas\r\n\r\n### Further work\r\n\r\n- Put icon sets and drawing tool metadata in DB (for example, order is hardcoded and we may want\r\n  to make the UI configurable)\r\n- Put shape menu info in DB\r\n- Improvements to shape types, separation of protocols about 2d drawing (domain) and SVG drawing (UI).\r\n- Update the canvas-coordinates stream to take into account user scrolls\r\n"},"comment":{"url":"https://api.github.com/repos/uxbox/front/issues/comments/141163540","html_url":"https://github.com/uxbox/front/pull/60#issuecomment-141163540","issue_url":"https://api.github.com/repos/uxbox/front/issues/60","id":141163540,"user":{"login":"dialelo","id":409039,"avatar_url":"https://avatars.githubusercontent.com/u/409039?v=3","gravatar_id":"","url":"https://api.github.com/users/dialelo","html_url":"https://github.com/dialelo","followers_url":"https://api.github.com/users/dialelo/followers","following_url":"https://api.github.com/users/dialelo/following{/other_user}","gists_url":"https://api.github.com/users/dialelo/gists{/gist_id}","starred_url":"https://api.github.com/users/dialelo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dialelo/subscriptions","organizations_url":"https://api.github.com/users/dialelo/orgs","repos_url":"https://api.github.com/users/dialelo/repos","events_url":"https://api.github.com/users/dialelo/events{/privacy}","received_events_url":"https://api.github.com/users/dialelo/received_events","type":"User","site_admin":false},"created_at":"2015-09-17T17:44:29Z","updated_at":"2015-09-17T17:44:29Z","body":"\\o/"}}