{"action":"created","issue":{"url":"https://api.github.com/repos/playframework/playframework/issues/3796","labels_url":"https://api.github.com/repos/playframework/playframework/issues/3796/labels{/name}","comments_url":"https://api.github.com/repos/playframework/playframework/issues/3796/comments","events_url":"https://api.github.com/repos/playframework/playframework/issues/3796/events","html_url":"https://github.com/playframework/playframework/pull/3796","id":53844195,"number":3796,"title":"Add Enumeratees for character encoding and decoding","user":{"login":"pauldraper","id":1403271,"avatar_url":"https://avatars.githubusercontent.com/u/1403271?v=3","gravatar_id":"","url":"https://api.github.com/users/pauldraper","html_url":"https://github.com/pauldraper","followers_url":"https://api.github.com/users/pauldraper/followers","following_url":"https://api.github.com/users/pauldraper/following{/other_user}","gists_url":"https://api.github.com/users/pauldraper/gists{/gist_id}","starred_url":"https://api.github.com/users/pauldraper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pauldraper/subscriptions","organizations_url":"https://api.github.com/users/pauldraper/orgs","repos_url":"https://api.github.com/users/pauldraper/repos","events_url":"https://api.github.com/users/pauldraper/events{/privacy}","received_events_url":"https://api.github.com/users/pauldraper/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-01-09T08:23:58Z","updated_at":"2015-01-12T01:53:18Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/playframework/playframework/pulls/3796","html_url":"https://github.com/playframework/playframework/pull/3796","diff_url":"https://github.com/playframework/playframework/pull/3796.diff","patch_url":"https://github.com/playframework/playframework/pull/3796.patch"},"body":"While building a few `Enumeratees` I ran into an obvious issue: **text-based formats deal in *characters*, not *bytes*.**\r\n\r\nTranslating between the two is not easy, as there are dozens of reasonably common encodings.\r\n\r\nI made a couple `Enumeratee`s to encode and decode between `Array[Byte]` and `String` (I could have used `Array[Char]` instead of `String`, but the latter is more common in APIs).\r\n\r\nThey use Java's [CharsetEncoder](http://docs.oracle.com/javase/6/docs/api/java/nio/charset/CharsetEncoder.html) and [CharsetDecoder](http://docs.oracle.com/javase/6/docs/api/java/nio/charset/CharsetDecoder.html) to incrementally encode and decode, handling cases where characters are split across chunks.\r\n\r\nThis allows for broad support, and now my text-based `Enumeratee`s can be used on any encoding Java understands.\r\n\r\nGiven the abundance of text-based formats, I believe this is a utility that most writers of `Enumeratees` in the Play framework will quickly find they need.\r\n\r\n---\r\n\r\nThere is one sticky spot, though it probably sounds worse than it actually is: I assume that if two `Array[Byte]`s can be decoded, the decoding of their concatenation is the concatenation of their decodings. Likewise, if two `String`s can be encoded, the encoding of their concatenation is the concatenation of their encodings.\r\n\r\nIf the `Enumeratee` (1) uses the mutable `CharsetEncoder` and `CharsetDecoder`, (2) processes input without permanently storing it, and (3) is not mutable itself, then I believe this assumption is a necessary one.\r\n\r\nTo my knowledge, every major encoding follows this, with one exception: the Unicode [BOM](https://en.wikipedia.org/wiki/Byte_order_mark) (byte-order-mark). Specifically, Unicode can have a 0xfeff marker at the beginning indicating byte-order. It's only a problem when decoding strings. Each time a `CharsetEncoder` starts, it can put a BOM. But you can't just concatenate the `Byte[Array]`s of multiple `CharsetEncoder`s together, as this mark is considered invalid anywhere except the beginning. Use of BOM is discouraged (and unncessary) for UTF-8 and UTF-32 and slightly discouraged for UTF-16. But, for example, OpenJDK prepends a BOM onto UTF-16 decodings. There's an ugly bit of code that removes it, that probably needs one final tweak.\r\n\r\n---\r\n\r\nThe `Enumeratee`s follow important guidelines: they're immutable and check whether the inner `Iteratee` is done or not. They have also be designed with performance in mind, so they operate on chunks of input, not individual `Byte`s or `Char`s.\r\n\r\nI'm interested in feedback on my approach and whether it would make sense to include in the Play framework."},"comment":{"url":"https://api.github.com/repos/playframework/playframework/issues/comments/69522858","html_url":"https://github.com/playframework/playframework/pull/3796#issuecomment-69522858","issue_url":"https://api.github.com/repos/playframework/playframework/issues/3796","id":69522858,"user":{"login":"jroper","id":105833,"avatar_url":"https://avatars.githubusercontent.com/u/105833?v=3","gravatar_id":"","url":"https://api.github.com/users/jroper","html_url":"https://github.com/jroper","followers_url":"https://api.github.com/users/jroper/followers","following_url":"https://api.github.com/users/jroper/following{/other_user}","gists_url":"https://api.github.com/users/jroper/gists{/gist_id}","starred_url":"https://api.github.com/users/jroper/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jroper/subscriptions","organizations_url":"https://api.github.com/users/jroper/orgs","repos_url":"https://api.github.com/users/jroper/repos","events_url":"https://api.github.com/users/jroper/events{/privacy}","received_events_url":"https://api.github.com/users/jroper/received_events","type":"User","site_admin":false},"created_at":"2015-01-12T01:53:18Z","updated_at":"2015-01-12T01:53:18Z","body":"Hi @pauldraper, looks good to me. We actually don't get that much demand for this - the reason being that generally by the time you've converted to characters, it means you want to parse it, and streaming parsing, especially asynchronous streaming parsing with iteratees, goes way above most peoples heads.  I've actually got a similar decoder [here](https://github.com/jroper/play-iteratees-extras/blob/master/src/main/scala/play/extras/iteratees/Encoding.scala), it takes a slightly different approach in how to allocate buffers (yours is technically more correct and probably more space efficient), but apart from that they follow quite the same approach.\r\n\r\nAnyway, I think there's no harm in merging this, so I'll do so."}}