{"action":"created","issue":{"url":"https://api.github.com/repos/go-distributed/meritop/issues/107","labels_url":"https://api.github.com/repos/go-distributed/meritop/issues/107/labels{/name}","comments_url":"https://api.github.com/repos/go-distributed/meritop/issues/107/comments","events_url":"https://api.github.com/repos/go-distributed/meritop/issues/107/events","html_url":"https://github.com/go-distributed/meritop/issues/107","id":54328923,"number":107,"title":"Defragment framework interfaces into context based calls","user":{"login":"fengjingchao","id":920884,"avatar_url":"https://avatars.githubusercontent.com/u/920884?v=3","gravatar_id":"","url":"https://api.github.com/users/fengjingchao","html_url":"https://github.com/fengjingchao","followers_url":"https://api.github.com/users/fengjingchao/followers","following_url":"https://api.github.com/users/fengjingchao/following{/other_user}","gists_url":"https://api.github.com/users/fengjingchao/gists{/gist_id}","starred_url":"https://api.github.com/users/fengjingchao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fengjingchao/subscriptions","organizations_url":"https://api.github.com/users/fengjingchao/orgs","repos_url":"https://api.github.com/users/fengjingchao/repos","events_url":"https://api.github.com/users/fengjingchao/events{/privacy}","received_events_url":"https://api.github.com/users/fengjingchao/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":17,"created_at":"2015-01-14T14:39:14Z","updated_at":"2015-01-16T04:08:49Z","closed_at":null,"body":"Problem Description\r\n---\r\nThe latest build failure was caused in the race:\r\n\r\n1. Everything worked correctly at epoch 6\r\n2. After finished sending the data to parent and before new epoch set, a slave died. It restarted in a new node and repeated the same computation.\r\n3. In regression framework,\r\n```Go\r\nfunc (t *dummySlave) ParentDataReady(parentID uint64, req string, resp []byte) {\r\n        ...\r\n\tif len(children) != 0 {\r\n\t\tt.framework.FlagMetaToChild(\"ParamReady\")\r\n\t} else {\r\n\t\t// On leaf node, we can immediately return by and flag parent\r\n\t\t// that this node is ready.\r\n\t\tt.framework.FlagMetaToParent(\"GradientReady\")\r\n\t}\r\n}\r\n```\r\n\r\nIn FlagMetaToX, it prepended *epoch* to the string and set it in etcd. However, the current implementation is wrong about synchronizing epoch here:\r\n\r\n```Go\r\nfunc (f *framework) FlagMetaToChild(meta string) {\r\n\tvalue := fmt.Sprintf(\"%d-%s\", f.epoch, meta)\r\n\tf.etcdClient.Set (ChildMetaPath, value, 0 )\r\n}\r\n```\r\nThe problem is, a new epoch 7 was set, and it flagged meta with the epoch 7. But it's actually dealing with data in epoch 6!\r\n\r\nAnalysis\r\n---\r\n\r\nApparently there are many ways to resolve it. We can stop all work on hand at the end of each epoch. But this is not code scalable -- if we write more remote requests, we also need to consistently close them, and heavy burden on debugging. Or we can just let user pass the epoch to `FlagMetaToChild`. This violates our design, and I am aware that keeping track of epoch, etc. info should be done in framework, not user level.\r\n\r\nProposed Solution\r\n---\r\nAfter a second thought to framework interface, actually there are two types of work in it. One type is epoch specific: FlagMetaToX, DataRequest, (maybe) IncEpoch. The rest of APIs are the other type, epoch agnostic.\r\n\r\nI propose to defragment FlagMetatoX, DataRequest into a different interface:\r\n```Go\r\ntype Context:\r\n  GetEpoch, GetFromID\r\n  FlagMetatoChild, FlagMetatoParent, DataRequest\r\n```\r\nAnd change Task interface:\r\n```Go\r\nfunc (t *dummySlave) ParentDataReady(ctx Context, req string, resp []byte) {\r\n \tif len(children) != 0 {\r\n\t\tctx.FlagMetaToChild(\"ParamReady\")\r\n\t} else {\r\n\t\t// On leaf node, we can immediately return by and flag parent\r\n\t\t// that this node is ready.\r\n\t\tctx.FlagMetaToParent(\"GradientReady\")\r\n\t}\r\n}\r\n```\r\nIn this way, the context will help us track epoch and it's easy for use to synchronized them under the framework hood."},"comment":{"url":"https://api.github.com/repos/go-distributed/meritop/issues/comments/70205887","html_url":"https://github.com/go-distributed/meritop/issues/107#issuecomment-70205887","issue_url":"https://api.github.com/repos/go-distributed/meritop/issues/107","id":70205887,"user":{"login":"xiaoyunwu","id":5473617,"avatar_url":"https://avatars.githubusercontent.com/u/5473617?v=3","gravatar_id":"","url":"https://api.github.com/users/xiaoyunwu","html_url":"https://github.com/xiaoyunwu","followers_url":"https://api.github.com/users/xiaoyunwu/followers","following_url":"https://api.github.com/users/xiaoyunwu/following{/other_user}","gists_url":"https://api.github.com/users/xiaoyunwu/gists{/gist_id}","starred_url":"https://api.github.com/users/xiaoyunwu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiaoyunwu/subscriptions","organizations_url":"https://api.github.com/users/xiaoyunwu/orgs","repos_url":"https://api.github.com/users/xiaoyunwu/repos","events_url":"https://api.github.com/users/xiaoyunwu/events{/privacy}","received_events_url":"https://api.github.com/users/xiaoyunwu/received_events","type":"User","site_admin":false},"created_at":"2015-01-16T04:08:49Z","updated_at":"2015-01-16T04:08:49Z","body":"Is it possible to do away with the concept of context?\r\n\r\nI still believe we can handle it without task implementation ever know\r\nabout context or epoch, am I missing something?\r\n\r\nXiaoyun\r\n\r\nOn Thu, Jan 15, 2015 at 6:37 PM, Hongchao Deng <notifications@github.com>\r\nwrote:\r\n\r\n> Let's talk with examples.\r\n>\r\n> Let's assume we have encoded context into req:\r\n>\r\n> func (t *dummySlave) ParentDataReady(parentID uint64, req Request, resp Resp) {\r\n>         ...\r\n>     if len(children) != 0 {\r\n>         req.Context.FlagMetaToChild(\"ParamReady\")\r\n>     } else {\r\n>         req.Context.FlagMetaToParent(\"GradientReady\")\r\n>     }\r\n> }\r\n>\r\n> So my question is, what APIs should be included in Context not Framework?\r\n> My list:\r\n>\r\n>    - FlagMetaToX\r\n>    - DataRequest\r\n>    - IncEpoch\r\n>\r\n> —\r\n> Reply to this email directly or view it on GitHub\r\n> <https://github.com/go-distributed/meritop/issues/107#issuecomment-70200220>\r\n> .\r\n>"}}