{"action":"created","issue":{"url":"https://api.github.com/repos/equalitie/ceno/issues/9","labels_url":"https://api.github.com/repos/equalitie/ceno/issues/9/labels{/name}","comments_url":"https://api.github.com/repos/equalitie/ceno/issues/9/comments","events_url":"https://api.github.com/repos/equalitie/ceno/issues/9/events","html_url":"https://github.com/equalitie/ceno/issues/9","id":54829121,"number":9,"title":"CeNo-node, Cache, and Bundler interface specification","user":{"login":"redwire","id":682761,"avatar_url":"https://avatars.githubusercontent.com/u/682761?v=3","gravatar_id":"","url":"https://api.github.com/users/redwire","html_url":"https://github.com/redwire","followers_url":"https://api.github.com/users/redwire/followers","following_url":"https://api.github.com/users/redwire/following{/other_user}","gists_url":"https://api.github.com/users/redwire/gists{/gist_id}","starred_url":"https://api.github.com/users/redwire/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redwire/subscriptions","organizations_url":"https://api.github.com/users/redwire/orgs","repos_url":"https://api.github.com/users/redwire/repos","events_url":"https://api.github.com/users/redwire/events{/privacy}","received_events_url":"https://api.github.com/users/redwire/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-01-20T00:25:11Z","updated_at":"2015-01-20T20:55:29Z","closed_at":null,"body":"Here is a dump of the email I wrote to @billdoran, @vmon, and @misaakidis about the specification for the interaction between the three major components of CeNo that I have designed.\r\n\r\nThe sequence diagram referenced can be found [in my freenetinterface branch](https://raw.githubusercontent.com/redwire/ceno/d83adfdb07db7231a10005aab13b192bb4de3201/ceno-node/doc/commSeqDiag.png)\r\n\r\nI've just finished designing what I think is a very reasonable\r\n(relatively abstract) protocol to facilitate communication between the\r\nCeNo client, Cache server, and Bundler/Transport.  I've designed the\r\nprotocol to use HTTP, since that is how most of the stack works already.\r\n\r\nI've attached an image of a sequence diagram I did up to outline the\r\nflow of the interactions.  Please have a quick look over it and refer\r\nback to it as I go on to explain the steps and decisions I made\r\nthroughout the rest of this email.\r\n\r\nFirst, some definitions:\r\n\r\n1. The \"User\" in this case is the web browser- the human being who\r\ninitiates a request for a web page.\r\n2. The \"CeNo client\" is the CeNo proxy server that serves bundled pages\r\nin response to requests from User.\r\n3. The \"Cache server\" is the server in charge of handling requests for\r\nbundles, which currently runs on Freenet.\r\n4. The \"Bundler or Transport\" is the bridge server that fetches pages\r\nand compiles them into bundled documents.\r\n\r\nSome formalities:\r\n\r\n1. I adhere to HTTP's expectations that:\r\n    1. GET requests are used to simply fetch documents/data; these\r\nrequests should not lead to changes on the server they are being\r\nrequested from.\r\n    2. POST requests are used to make requests to pages that *do* lead\r\nto changes on the server they are sent to.\r\n2. JSON is used to encode responses that the CeNo client will have to\r\nhandle, since it is Javascript already and that is pretty standard\r\nAPI-esque convention for Node.JS.\r\n3. This document describes the more complicated scenario where a bundle\r\nis *not* found stored on the Cache server.  Quite clearly, the case\r\nwhere a bundle exists is finished after the first request to the Cache\r\nserver.\r\n\r\nNow then, a detailed explanation of the process.\r\n\r\n1. User sends a request for a page from CeNo client on the index route\r\n(/), with the URL of the page encoded in the query string.\r\n2. CeNoClient requests the bundle for the URL being requested from the\r\nCache server on the index route (/), using the same encoding.\r\n3. Cache server checks if a bundle exists.\r\n4. Cache server responds to CeNo client to inform it that no bundle was\r\nfound (DNF).\r\n5. CeNo client sends User a \"please wait\" page which will periodically\r\nre-request the original page requested by User. More on this later.\r\n6. CeNo client also sends a request to Transport on the index route (/)\r\nagain with the URL encoded in the query string to initiate a bundle\r\ncreation process. This is a GET request because it does not require the\r\nTransport store data- only that it start doing some work.\r\n7. Transport responds to CeNo client with an ID corresponding to the\r\nprocess handling its request.\r\n8. Transport creates a bundle for the requested page.\r\n9. Transport POSTs to Cache server on the store route (/store) to have\r\nit store the new bundle.\r\n10. Cache server stores the bundle.\r\n11. Cache server responds to Transport to inform it that it has\r\ncompleted storing the bundle.\r\n12. Transport POSTs to CeNo client on the done route (/done) to inform\r\nit that a process has completed.\r\n13. CeNo client responds to Transport to confirm that it has\r\nacknowledged that it is no longer waiting for the newly completed process.\r\n14. User re-requests the page they originally requested just as in step 1.\r\n15. As in step 2, CeNo client requests the bundle from Cache server.\r\n16. Cache server checks for the bundle requested.\r\n17. Cache server responds to CeNo client with the bundled page.\r\n18. CeNo client serves the bundle to User.\r\n\r\nClarification:\r\n\r\n1. At step 7, CeNo client will store the process ID it receives so that\r\nit does not have to re-request the same page of either Cache server or\r\nTransport should the same page be requested by User again.\r\n2. Transport sends bundles to Cache server before CeNo client for three\r\nreasons:\r\n    1. So that it can respond (with a process ID) quickly to CeNo\r\nclient, making CeNo client able to keep track of requests being\r\nprocessed and to leave it free to handle other User requests.\r\n    2. So that the bundle can be guaranteed to be retrievable by Cache\r\nserver once CeNo client is informed that it has been prepared.\r\n    3. So that CeNo client does not have to take any extra special steps\r\nto request the bundle from Transport, and can instead re-request the\r\nbundle from Cache server whenever User re-requests it.\r\n3. Bundling is going to be a long process no matter what and the\r\nexpectation is that, over time, Bundler will be invoked less and less\r\nfor more popular, cached pages. All interactions are designed to be as\r\nshort and to-the-point as possible so that HTTP can be used seamlessly.\r\n4. This design leaves CeNo client responsible for doing the bare minimum\r\namount of work/waiting possible so that it is free to quickly respond to\r\nUser in the case that it must reserve the \"please wait\" page.\r\n5. Cache server is designed to behave most like a standard web service. \r\nIt is possible to have bundles served to or requested from it from any\r\nkind of clients with this approach.\r\n\r\nThat's it for now, everyone.  If anyone feels so inclined, feel free to\r\nturn this email into an issue on Github for clearer discussion.  If you\r\nhave any questions, feel free to either email me back or hit me up on\r\nIRC during my usual work hours (3:30 PM to 7:30 PM EST on Monday to Friday).\r\n\r\nMy focus is on the \"please wait\" page to be served to User as well as\r\nthe functionality of both CeNo client and Transport.  The majority of\r\nthe functionality of both of these things is already complete, and I\r\nwill begin implementing means of handling the protocol I have defined\r\nhere.  If any changes must be made, it will still be quite easy to do so."},"comment":{"url":"https://api.github.com/repos/equalitie/ceno/issues/comments/70731340","html_url":"https://github.com/equalitie/ceno/issues/9#issuecomment-70731340","issue_url":"https://api.github.com/repos/equalitie/ceno/issues/9","id":70731340,"user":{"login":"redwire","id":682761,"avatar_url":"https://avatars.githubusercontent.com/u/682761?v=3","gravatar_id":"","url":"https://api.github.com/users/redwire","html_url":"https://github.com/redwire","followers_url":"https://api.github.com/users/redwire/followers","following_url":"https://api.github.com/users/redwire/following{/other_user}","gists_url":"https://api.github.com/users/redwire/gists{/gist_id}","starred_url":"https://api.github.com/users/redwire/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redwire/subscriptions","organizations_url":"https://api.github.com/users/redwire/orgs","repos_url":"https://api.github.com/users/redwire/repos","events_url":"https://api.github.com/users/redwire/events{/privacy}","received_events_url":"https://api.github.com/users/redwire/received_events","type":"User","site_admin":false},"created_at":"2015-01-20T20:55:29Z","updated_at":"2015-01-20T20:55:29Z","body":"Those are some really good point, @misaakidis.\r\n\r\nI have a concern with the second of your extra thoughts regarding how quickly CeNo client would be expected to respond to User.  I definitely agree that we would like to have connections between User and CeNo client effectively closed as soon as possible so that CeNo client does not become overwhelmed. However, it my concern that if we have CeNo client always serve the \"please wait\" page immediately, then we have two issues with regards to how it later handles the bundle it requests of Cache server.\r\n\r\n1. CeNo client will be made busy by requests to Cache server regardless of when it makes them. If we have CeNo client immediately respond to User with the \"please wait\" page, CeNo client would have to keep the bundle data it eventually receives in memory until User requests the page again. What if this request never comes? What if it just takes longer than usual for it to come?\r\n2. Related to point 1 above, if we took the approach of further reducing the requirement for synchronization between CeNo client and Cache server so that Cache server would send a POST to CeNo client when it has retrieved a bundle from cache upon every request, Cache server would end up doing *a lot* more work than in the scheme I have outlined in cases where User has unusual timing for re-requests.\r\n\r\nI believe I understand the motivation for your suggestion, but I worry about how it would scale as CeNo is used by more and more individuals with variably reliable network connections.\r\n\r\nI suppose your third point addresses my concerns by introducing a Queue or Promise into the setting. I feel like we might be more specifically concerned with the ephemeral nature of HTTP requests and wonder what you think of the idea of not using HTTP at all."}}