{"action":"created","issue":{"url":"https://api.github.com/repos/ipfs/go-ipfs/issues/1291","labels_url":"https://api.github.com/repos/ipfs/go-ipfs/issues/1291/labels{/name}","comments_url":"https://api.github.com/repos/ipfs/go-ipfs/issues/1291/comments","events_url":"https://api.github.com/repos/ipfs/go-ipfs/issues/1291/events","html_url":"https://github.com/ipfs/go-ipfs/issues/1291","id":80656632,"number":1291,"title":"Spec for adder APIs (both low-level and high-level)","user":{"login":"wking","id":209920,"avatar_url":"https://avatars.githubusercontent.com/u/209920?v=3","gravatar_id":"","url":"https://api.github.com/users/wking","html_url":"https://github.com/wking","followers_url":"https://api.github.com/users/wking/followers","following_url":"https://api.github.com/users/wking/following{/other_user}","gists_url":"https://api.github.com/users/wking/gists{/gist_id}","starred_url":"https://api.github.com/users/wking/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wking/subscriptions","organizations_url":"https://api.github.com/users/wking/orgs","repos_url":"https://api.github.com/users/wking/repos","events_url":"https://api.github.com/users/wking/events{/privacy}","received_events_url":"https://api.github.com/users/wking/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-05-25T20:27:07Z","updated_at":"2015-06-03T21:44:45Z","closed_at":"2015-06-02T17:30:03Z","body":"A clear picture of the intended adder API is blocking some of the later commits in #1136.  This issue floats my first pass at an adder API that I think strikes a good balance between power and simplicity (mostly by offloading the complicated bits to a per-DAG-node callback).\r\n\r\n# Where it lives\r\n\r\nI'd suggest moving `core/coreunix` to `core/unixfs` to match\r\n`unixfs/` and `shell/unixfs/`.\r\n\r\n# High-level UI (shell/unixfs/)\r\n\r\n    // NodeCallback is called after each DAG node is created.  The\r\n    // arguments are:\r\n    //\r\n    //   nIn: The just-created node\r\n    //   p: The path from the add root to the just-created node.  This\r\n    //     is empty for the root node.  For large files and\r\n    //     directories that are chunked and trickled, each chunk and\r\n    //     fanout node will have the same path argument.\r\n    //   f: A File reference for the file used to create nIn.  Don't\r\n    //     seek this (which could throw off the chunker), but you can\r\n    //     use it to extract metadata about the visited file including\r\n    //     its name, permissions, etc.  This will be nil for\r\n    //     io.Reader-based adders.\r\n    //   top: Whether or not nIn is the tip of a trickle DAG or an\r\n    //     unchunked object.  This allows you to distinguish those\r\n    //     nodes (which are referenced from a link with a user-visible\r\n    //     name)\r\n    //\r\n    // The returned values are:\r\n    //\r\n    //   nOut: The node to insert into the constructed DAG.  Return nIn\r\n    //     to use the just-created node without changes or nil to drop\r\n    //     the just-created node.  You're also free to return another\r\n    //     node of your choosing (e.g. a new node wrapping the\r\n    //     just-created node or a completely independent node).\r\n    //   err: Any errors serious enough to abort the addition.\r\n    type NodeCallback func(nIn *dag.Node, p *path.Path, f *os.File, top bool) (nOut *dag.Node, err error)\r\n\r\n    // Add recursively adds files from a File type, which can point to\r\n    // either a directory or a file.  The arguments are:\r\n    //\r\n    //   ctx: A Context for cancelling or timing out a recursive\r\n    //     addition.\r\n    //   n: And IPFS node for storing newly-created DAG nodes.\r\n    //   f: An open file pointing at the root of the filesystem to be\r\n    //     added.\r\n    //   cb: An optional hook for post-DAG-node processing.  Set to nil\r\n    //     if you don't need it.\r\n    //\r\n    // The returned values are:\r\n    //\r\n    //   root: The root of the just-added DAG.\r\n    //   err: Any errors serious enough to abort the addition.\r\n    Add(ctx context.Context, n *core.IpfsNode, f *os.File, cb *NodeCallback) (root *dag.Node, err error)\r\n\r\n    // AddFromReader recursively adds a file from an io.Reader.  It is\r\n    // otherwise identical to Add().\r\n    AddFromReader(ctx context.Context, n *core.IpfsNode, r io.Reader, cb *NodeCallback) (root *dag.Node, err error)\r\n\r\nMost additions will be recursive and load data from a [*File][File]\r\n(which can be a directory or a file).  Alternatively, the\r\n`*FromReader` variants accept a [Reader][2].\r\n\r\nWe need a way to get information about progress of a running addition\r\nback to other goroutines.  Choices for this include [the channel\r\nmessages proposed in #1121][channel] or additional arguments to [a\r\nper-chunk callback like that proposed in #1274][callback].  The main\r\ndifference between a callback and a channel is whether or not we want\r\nsynchronous collaboration between the adder and the hook.  Since I\r\nthink we want the option for synchronous collaboration\r\n(e.g. optionally inserting a metadata node on top of each file node).\r\nFor situations where asynchronous communication makes more sense, the\r\nuser can provide a synchronous callback that just pushes a message to\r\na channel (so the callback-based API supports the channel-based\r\nworkflow).\r\n\r\nActions like wrapping an added file in another Merkle object to hold a\r\nfilename is left to the caller and the callback API.\r\n\r\n# Low-level UI (core/unixfs/)\r\n\r\nThese should look just like the high-level API, except instead of\r\npassing in an IpfsNode and using that node's default DAG service,\r\ntrickler, and splitter, we pass each of those in explicitly:\r\n\r\n    Add(ctx context.Context, ds dag.DAGService, t trickle.Trickler, s chunk.BlockSplitter, f *os.File, cb *NodeCallback) (root *dag.Node, err error)\r\n    AddFromReader(ctx context.Context, ds dag.DAGService, t trickle.Trickler, s chunk.BlockSplitter, r io.Reader, cb *NodeCallback) (root *dag.Node, error)\r\n\r\nWe don't currently have a public `Trickler` interface, but I think we should\r\nadd one so folks can easily plug in alternative trickler implementations.\r\n\r\nI'm not familiar enough with Go at the moment to know which arguments\r\nare best passed by reference and which should be passed by value.  If\r\nI was writing this in C, everything except the Boolean ‘top’ would be\r\npassed by reference, but I'll have to read around to figure out what's\r\nidiomatic in Go.\r\n\r\n[File]: https://golang.org/pkg/os/#File\r\n[Reader]: https://golang.org/pkg/io/#Reader\r\n[channel]: https://github.com/ipfs/go-ipfs/issues/1121#issuecomment-104073727\r\n[callback]: https://github.com/ipfs/go-ipfs/pull/1274\r\n"},"comment":{"url":"https://api.github.com/repos/ipfs/go-ipfs/issues/comments/108624573","html_url":"https://github.com/ipfs/go-ipfs/issues/1291#issuecomment-108624573","issue_url":"https://api.github.com/repos/ipfs/go-ipfs/issues/1291","id":108624573,"user":{"login":"jbenet","id":138401,"avatar_url":"https://avatars.githubusercontent.com/u/138401?v=3","gravatar_id":"","url":"https://api.github.com/users/jbenet","html_url":"https://github.com/jbenet","followers_url":"https://api.github.com/users/jbenet/followers","following_url":"https://api.github.com/users/jbenet/following{/other_user}","gists_url":"https://api.github.com/users/jbenet/gists{/gist_id}","starred_url":"https://api.github.com/users/jbenet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbenet/subscriptions","organizations_url":"https://api.github.com/users/jbenet/orgs","repos_url":"https://api.github.com/users/jbenet/repos","events_url":"https://api.github.com/users/jbenet/events{/privacy}","received_events_url":"https://api.github.com/users/jbenet/received_events","type":"User","site_admin":false},"created_at":"2015-06-03T21:44:45Z","updated_at":"2015-06-03T21:44:45Z","body":"@wking sorry maybe i missed it-- the motivation for `pre-` and `post-` callbacks?"}}