{"action":"created","issue":{"url":"https://api.github.com/repos/rails/rails-html-sanitizer/issues/30","labels_url":"https://api.github.com/repos/rails/rails-html-sanitizer/issues/30/labels{/name}","comments_url":"https://api.github.com/repos/rails/rails-html-sanitizer/issues/30/comments","events_url":"https://api.github.com/repos/rails/rails-html-sanitizer/issues/30/events","html_url":"https://github.com/rails/rails-html-sanitizer/issues/30","id":56319269,"number":30,"title":"Encoding errors on ASCII-8BIT strings (eg: any string from the mysql adapter)","user":{"login":"kindjar","id":290336,"avatar_url":"https://avatars.githubusercontent.com/u/290336?v=3","gravatar_id":"","url":"https://api.github.com/users/kindjar","html_url":"https://github.com/kindjar","followers_url":"https://api.github.com/users/kindjar/followers","following_url":"https://api.github.com/users/kindjar/following{/other_user}","gists_url":"https://api.github.com/users/kindjar/gists{/gist_id}","starred_url":"https://api.github.com/users/kindjar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kindjar/subscriptions","organizations_url":"https://api.github.com/users/kindjar/orgs","repos_url":"https://api.github.com/users/kindjar/repos","events_url":"https://api.github.com/users/kindjar/events{/privacy}","received_events_url":"https://api.github.com/users/kindjar/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-02-03T01:11:03Z","updated_at":"2015-02-24T18:38:06Z","closed_at":null,"body":"The sanitizer seems to have issues when its input is a string in `ASCII-8BIT` encoding:\r\n\r\n```ruby\r\nirb(main):006:0* Rails::Html::WhiteListSanitizer.new.sanitize(\"tooth\".encode('ASCII-8BIT'))\r\noutput error : unknown encoding ASCII-8BIT\r\n=> \"\"\r\nirb(main):007:0>\r\n```\r\n\r\nWhile `ASCII-8BIT` isn't the default encoding these days, it seems that strings coming from the `mysql` adapter (but not the `mysql2` adapter) are **always** in `ASCII-8BIT` encoding, even when the table is using charset `utf8`:\r\n\r\n```ruby\r\nirb(main):004:0> Day.connection.charset\r\n=> \"utf8\"\r\nirb(main):005:0> Day.last.notes.encoding\r\n=> #<Encoding:ASCII-8BIT>\r\n```\r\n\r\nThis means that using the sanitizer on any string from the database when using the `mysql` adapter will result in errors. I chased the error down to Nokogiri's `NodeSet#to_s` method, but wasn't sure what the right approach was for addressing the issue.\r\n\r\nSwitching to the `mysql2` adapter makes the issue go away, since it produces all strings in UTF-8. However, folks who've been using the `mysql` gem (for legacy reasons or whatever) could run into headaches trying to upgrade to Rails 4.2 because of this (it hit me by way of the `highlight` method in `ActionView::Helpers::TextHelper`)."},"comment":{"url":"https://api.github.com/repos/rails/rails-html-sanitizer/issues/comments/75817436","html_url":"https://github.com/rails/rails-html-sanitizer/issues/30#issuecomment-75817436","issue_url":"https://api.github.com/repos/rails/rails-html-sanitizer/issues/30","id":75817436,"user":{"login":"kindjar","id":290336,"avatar_url":"https://avatars.githubusercontent.com/u/290336?v=3","gravatar_id":"","url":"https://api.github.com/users/kindjar","html_url":"https://github.com/kindjar","followers_url":"https://api.github.com/users/kindjar/followers","following_url":"https://api.github.com/users/kindjar/following{/other_user}","gists_url":"https://api.github.com/users/kindjar/gists{/gist_id}","starred_url":"https://api.github.com/users/kindjar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kindjar/subscriptions","organizations_url":"https://api.github.com/users/kindjar/orgs","repos_url":"https://api.github.com/users/kindjar/repos","events_url":"https://api.github.com/users/kindjar/events{/privacy}","received_events_url":"https://api.github.com/users/kindjar/received_events","type":"User","site_admin":false},"created_at":"2015-02-24T18:38:05Z","updated_at":"2015-02-24T18:38:05Z","body":"@kaspth Yes #33 does fix the issue I was seeing, and since my data was actually UTF-8 already (it's just that the mysql database adapter tags it wrong), everything is cool. \r\n\r\nForcing ASCII-8BIT (which is just a label for a collection of different character encodings) to be interpreted as UTF-8 could cause problems for people whose data is in some non-UTF-8-compatible encoding (like, say, Windows-1252). It would be nice to get a note into the release notes to warn people about the `mysql` adapter, though."}}