{"action":"created","issue":{"url":"https://api.github.com/repos/berkshelf/berkshelf-api/issues/112","labels_url":"https://api.github.com/repos/berkshelf/berkshelf-api/issues/112/labels{/name}","comments_url":"https://api.github.com/repos/berkshelf/berkshelf-api/issues/112/comments","events_url":"https://api.github.com/repos/berkshelf/berkshelf-api/issues/112/events","html_url":"https://github.com/berkshelf/berkshelf-api/issues/112","id":32175497,"number":112,"title":"Can't process certain cookbooks","user":{"login":"stuartphilp","id":2964961,"avatar_url":"https://avatars.githubusercontent.com/u/2964961?v=3","gravatar_id":"","url":"https://api.github.com/users/stuartphilp","html_url":"https://github.com/stuartphilp","followers_url":"https://api.github.com/users/stuartphilp/followers","following_url":"https://api.github.com/users/stuartphilp/following{/other_user}","gists_url":"https://api.github.com/users/stuartphilp/gists{/gist_id}","starred_url":"https://api.github.com/users/stuartphilp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stuartphilp/subscriptions","organizations_url":"https://api.github.com/users/stuartphilp/orgs","repos_url":"https://api.github.com/users/stuartphilp/repos","events_url":"https://api.github.com/users/stuartphilp/events{/privacy}","received_events_url":"https://api.github.com/users/stuartphilp/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/berkshelf/berkshelf-api/labels/Bug","name":"Bug","color":"F01A00"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":49,"created_at":"2014-04-24T18:53:54Z","updated_at":"2015-02-26T00:25:31Z","closed_at":null,"body":"Occasionally the cache builder will fail to process cookbooks with invalid characters in their metadata.json:\r\n\r\n``` \r\nProcessing metadata for 500 cookbooks with 5278 remaining on opscode\r\n2014-04-24_18:38:09.38282 I, [2014-04-24T13:38:00.353518 #8728]  INFO -- : Saving the cache to: /etc/berkshelf/api-server/cerch\r\n2014-04-24_18:38:09.38283 I, [2014-04-24T13:38:00.353977 #8728]  INFO -- : Cache saved!\r\n2014-04-24_18:38:09.38284 E, [2014-04-24T13:38:09.381809 #8728] ERROR -- : Actor crashed!\r\n2014-04-24_18:38:09.38285 Encoding::InvalidByteSequenceError: \"\\xE2\" on US-ASCII\r\n2014-04-24_18:38:09.38285       /opt/berkshelf-api/v1.3.1/vendor/bundle/ruby/1.9.1/gems/json-1.8.1/lib/json/common.rb:155:in `encode'\r\n2014-04-24_18:38:09.38285       /opt/berkshelf-api/v1.3.1/vendor/bundle/ruby/1.9.1/gems/json-1.8.1/lib/json/common.rb:155:in `initialize'\r\n2014-04-24_18:38:09.38285       /opt/berkshelf-api/v1.3.1/vendor/bundle/ruby/1.9.1/gems/json-1.8.1/lib/json/common.rb:155:in `new'\r\n2014-04-24_18:38:09.38285       /opt/berkshelf-api/v1.3.1/vendor/bundle/ruby/1.9.1/gems/json-1.8.1/lib/json/common.rb:155:in `parse'\r\n2014-04-24_18:38:09.38286       /opt/berkshelf-api/v1.3.1/vendor/bundle/ruby/1.9.1/gems/ridley-3.1.0/lib/ridley/chef/cookbook/metadata.rb:458:in `from_json'\r\n2014-04-24_18:38:09.38286       /opt/berkshelf-api/v1.3.1/vendor/bundle/ruby/1.9.1/gems/ridley-3.1.0/lib/ridley/chef/cookbook/metadata.rb:29:in `from_json'\r\n2014-04-24_18:38:09.38286       /opt/berkshelf-api/v1.3.1/lib/berkshelf/api/cache_builder/worker/opscode.rb:59:in `load_metadata'\r\n2014-04-24_18:38:09.38286       /opt/berkshelf-api/v1.3.1/lib/berkshelf/api/cache_builder/worker/opscode.rb:39:in `block in metadata'\r\n2014-04-24_18:38:09.38287       /opt/chef/embedded/lib/ruby/1.9.1/tmpdir.rb:83:in `mktmpdir'\r\n2014-04-24_18:38:09.38287       /opt/berkshelf-api/v1.3.1/lib/berkshelf/api/cache_builder/worker/opscode.rb:37:in `metadata'\r\n2014-04-24_18:38:09.38287       /opt/berkshelf-api/v1.3.1/vendor/bundle/ruby/1.9.1/gems/celluloid-0.16.0.pre/lib/celluloid/calls.rb:26:in `public_send'\r\n2014-04-24_18:38:09.38287       /opt/berkshelf-api/v1.3.1/vendor/bundle/ruby/1.9.1/gems/celluloid-0.16.0.pre/lib/celluloid/calls.rb:26:in `dispatch'\r\n2014-04-24_18:38:09.38288       /opt/berkshelf-api/v1.3.1/vendor/bundle/ruby/1.9.1/gems/celluloid-0.16.0.pre/lib/celluloid/calls.rb:63:in `dispatch'\r\n2014-04-24_18:38:09.38288       /opt/berkshelf-api/v1.3.1/vendor/bundle/ruby/1.9.1/gems/celluloid-0.16.0.pre/lib/celluloid/cell.rb:60:in `block in invoke'\r\n2014-04-24_18:38:09.38288       /opt/berkshelf-api/v1.3.1/vendor/bundle/ruby/1.9.1/gems/celluloid-0.16.0.pre/lib/celluloid/cell.rb:71:in `block in task'\r\n2014-04-24_18:38:09.38288       /opt/berkshelf-api/v1.3.1/vendor/bundle/ruby/1.9.1/gems/celluloid-0.16.0.pre/lib/celluloid/actor.rb:362:in `block in task'\r\n2014-04-24_18:38:09.38289       /opt/berkshelf-api/v1.3.1/vendor/bundle/ruby/1.9.1/gems/celluloid-0.16.0.pre/lib/celluloid/tasks.rb:55:in `block in initialize'\r\n2014-04-24_18:38:09.38289       /opt/berkshelf-api/v1.3.1/vendor/bundle/ruby/1.9.1/gems/celluloid-0.16.0.pre/lib/celluloid/tasks/task_fiber.rb:15:in `block in create'\r\n```\r\n\r\nPerhaps the [opscode worker](https://github.com/berkshelf/berkshelf-api/blob/master/lib/berkshelf/api/cache_builder/worker/opscode.rb#L60) could handle this more gracefully"},"comment":{"url":"https://api.github.com/repos/berkshelf/berkshelf-api/issues/comments/76095440","html_url":"https://github.com/berkshelf/berkshelf-api/issues/112#issuecomment-76095440","issue_url":"https://api.github.com/repos/berkshelf/berkshelf-api/issues/112","id":76095440,"user":{"login":"lamont-granquist","id":454857,"avatar_url":"https://avatars.githubusercontent.com/u/454857?v=3","gravatar_id":"","url":"https://api.github.com/users/lamont-granquist","html_url":"https://github.com/lamont-granquist","followers_url":"https://api.github.com/users/lamont-granquist/followers","following_url":"https://api.github.com/users/lamont-granquist/following{/other_user}","gists_url":"https://api.github.com/users/lamont-granquist/gists{/gist_id}","starred_url":"https://api.github.com/users/lamont-granquist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lamont-granquist/subscriptions","organizations_url":"https://api.github.com/users/lamont-granquist/orgs","repos_url":"https://api.github.com/users/lamont-granquist/repos","events_url":"https://api.github.com/users/lamont-granquist/events{/privacy}","received_events_url":"https://api.github.com/users/lamont-granquist/received_events","type":"User","site_admin":false},"created_at":"2015-02-26T00:25:31Z","updated_at":"2015-02-26T00:25:31Z","body":"lol\r\n\r\nYeah, so there's various ways to hose locales.  Easiest way is if someone just doesn't like the way `ls -la` sorts output in the `en_US.UTF_8` locale with dotfiles all scattered through the listing alphabetically -- and what people do is `export LC_ALL=C` or something like that and then they break UTF-8 for everything they run -- which works for 'muricans until they try to parse something with an umlaut in someone's name and then they think the software they ran is buggy (and that is exactly how I ran for years myself as a proud 'murican).  We also ran into people running on windows in Japan with some kind of strange non-UTF-8 charset which they had to have default on the system but which broke `knife upload` as well.\r\n\r\nSo, anyway, it turned out that we couldn't really fight the latter battle, and while we could fight the former battle that went nowhere because while locales aren't actually all that hard, its just hard enough that users don't care.  So what Chef 12 does now is force the default external encoding to UTF-8:\r\n\r\nhttps://github.com/chef/chef/blob/68bab0b362c47e5a75492e57094f72ecee1171ae/lib/chef/application.rb#L193-L195\r\n\r\nhttps://github.com/chef/chef/blob/master/lib/chef/config.rb#L698-L705\r\n\r\nI still sort of think its philosophically wrong to be doing that, but in practice nobody seems to care and UTF-8 is mostly taking over the world, and by standing ground you just wind up in a mutual firing squad with users on one side and software devs on the other.   Plus if anyone ever does write a cookbook or whatever in Shift-JIS then they can still put a magic comment at the start of it to change the encoding.\r\n\r\nIf you look further into what Chef does with locales there's also this huge pile of code here which I do *not* think you need:\r\n\r\nhttps://github.com/chef/chef/blob/master/lib/chef/config.rb#L646-L696\r\n\r\nThat has to do with the fact that we explicitly need an *English* locale because in many cases in providers we parse the output of commands and grep for localized strings and if the system is in `de_DE.UTF-8` you'll get back German which will break the providers -- so we need to force external commands to English and UTF-8 (except for when service providers are firing up someone's software which executes in the system locale).   I don't think you need that.\r\n\r\nSo, TL;DR add a line:\r\n\r\n```\r\nEncoding.default_external = Encoding::UTF_8\r\n```\r\n\r\nMake it configurable for bonus points, but chances are nobody will care."}}