Sunday, July 19, 2015

Thoughts on Conduits




The Conduit model



Things to know about Conduit



Useful functions



countC = sumC <| mapC (const 1)


awaitJust act = do
    x <- await
    whenJust x act


zipFromC !i = awaitJust $ \a -> do
    yield (i,a)
    zipFromC (succ i)



Why not Pipes?




John Wiegley said...

Both pipes and conduit start from the same foundation, which is equivalent to an effect-bearing Free construction over a get/put term algebra:

data TermAlg i o r = Get (i -> r) | Put o r
type Pipe i o = FreeT (TermAlg i o)

Where the two libraries differ is what functionality they add to this term algebra. For pipes, it is only bidirectional communication:

data TermAlg i' i o' o r
= Get i' (i -> r)
| Put o (o' -> r)

For Conduit, the algebra remains unidirectional, but three other features are added: prompt finalization, notification of upstream termination, and support for leftovers:

data TermAlg l i o u m r
= Get (i -> r) (u -> r)
| Put o (m ()) r
| Leftover l r

I wouldn't necessarily call these additions "unprincipled", except insofar as they extend the semantics of the term algebra in ways that no longer allow certain mathematical constructions, like categories, to be formed. Obeying fewer laws means the type is less constrained in its behavior, which does reduce reasoning power.

Where I think the real question lies is: How much functionality should one "bake in" when other approaches are possible? For example, leftover support can be added externally using StateT, and is the approach taken by pipes-parse. Yes, it is more difficult to learn at the beginning, but isn't Haskell about wearing the hair shirt, and communicating as much of a program's behavior in types as we can get away with, to write better programs?

Conduit aims at a certain intuitive simplicity by providing a semantically rich core type, and for its chosen crowd of users, this may be exactly what they want. I tend to prefer the core simplicity of pipes, and like the fact that to get fancy behavior out of my pipelines, I need to use fancier types.

Neil Mitchell said...

John: Thanks for that explanation. I suggest you put it on a blog post somewhere (or School of Haskell or some such) as I think it would be useful for others - it's the clearest comparison I've seen, the kind of thing I was looking for. I think in my case I'm using the subset that makes use of neither of the extensions, so my pipe and conduit experiences are likely to be the same.

John Wiegley said...

Hi Neil, thanks for the suggestion, that sounds like a good idea.

In terms of your experience using both libraries probably being the same: Although you don't explicitly make use of conduit's extra features, you probably do so implicitly. For example, notification of upstream termination allows conduit to collect elements into lists using a Consumer (sinkLazy). In pipes, such collection must be done by a Producer to list function (toListM). Receiving Producers as arguments is a common pipes idiom for detecting Producer termination. You'll even see functions from Producer to Producer, that in conduit could just be Conduits.

An example where leftovers become significant is takeWhile, since pipes can't "put back" the element that fails the test. Currently, there is no way to access that element at all (although I've opened a pull request to remedy this). Even after my pull request, the user of takeWhile must consciously use that element first, before calling await to get the next element, which is a requirement *not* present in the types.

Lastly there is prompt finalization, which means that a file can be closed as soon as its last chunk is read. This can only sort of be done in pipes, by testing for EOF and closing the handle before transmitting the last chunk. However, even that doesn't help if downstream exits before upstream has a chance to read that chunk. Closure is then deferred until the entire SafeT session has ended, where the pipes bracket finalizer was registered. In conduit the right behavior happens naturally for any user of bracket, meaning there are fewer ways to get it wrong.

Whatever the Next Generation of streaming libraries may be, my hope is that as many gotchas like this are eliminated by the types as possible, while retaining maximal expressiveness about the semantics of those types (and so, more laws).

Neil Mitchell said...

John: Yet more really interesting information! Certainly worthy of wider publication, since these are exactly the things I wanted to know.