{"action":"created","issue":{"url":"https://api.github.com/repos/jruby/jruby/issues/2856","labels_url":"https://api.github.com/repos/jruby/jruby/issues/2856/labels{/name}","comments_url":"https://api.github.com/repos/jruby/jruby/issues/2856/comments","events_url":"https://api.github.com/repos/jruby/jruby/issues/2856/events","html_url":"https://github.com/jruby/jruby/issues/2856","id":69701417,"number":2856,"title":"Some data can cause String#encode to hang","user":{"login":"marshalium","id":237643,"avatar_url":"https://avatars.githubusercontent.com/u/237643?v=3","gravatar_id":"","url":"https://api.github.com/users/marshalium","html_url":"https://github.com/marshalium","followers_url":"https://api.github.com/users/marshalium/followers","following_url":"https://api.github.com/users/marshalium/following{/other_user}","gists_url":"https://api.github.com/users/marshalium/gists{/gist_id}","starred_url":"https://api.github.com/users/marshalium/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marshalium/subscriptions","organizations_url":"https://api.github.com/users/marshalium/orgs","repos_url":"https://api.github.com/users/marshalium/repos","events_url":"https://api.github.com/users/marshalium/events{/privacy}","received_events_url":"https://api.github.com/users/marshalium/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-04-20T23:13:29Z","updated_at":"2015-04-29T04:47:55Z","closed_at":null,"body":"I have code that attempts to remove invalid characters by converting an input (supposedly) UTF-8 string to UTF-16 and back to UTF-8.\r\n\r\nIf given a string of random binary data instead of mostly valid UTF-8, the first call to encode (UTF-8 -> UTF-16) can hang and appears to never return. I wrote a test script that demonstrates a case where this happens consistently. It does not happen with all random data, but it's pretty easy to find a case that does this by just randomly generating bytes.\r\n\r\n    #!/usr/bin/env ruby\r\n    \r\n    require 'base64'\r\n    \r\n    b64_str = <<-EOS\r\n    1UflCDMEvBUpiI0dm73ApUSGAZBpWnk3HnW2mt68zj5LtNPJJTJ7eZi3Fr8Y\r\n    zhIb3lANSO4OMtTCHZLzPREbFS4i42tNbpO49AeFnBF963sDMUzQCSXFNlKx\r\n    6zZlAz342wWs90mJkcJEvr6SyqzXyA98NNa3K6G1DOjWhTYb52pkAEB5CF3i\r\n    mL5FKPHrvT+grhY3fgUeSlWU/Ozux0vwtD6iaO5l/aGr9yzBD8mDyZASxNOe\r\n    g+Bq+q8DUuh97W5O6wMYs2/X+p10/mXwT87MvOk+hSFk8EUVPesaY4T306eq\r\n    o6eJbCsBVIrQRqKGnh5vKkzWoXdg+f7bdqhTn4Ut3L/8gRsTB0/dAp6p1S6F\r\n    YvZVqRiW75pIYhcrHmtJFAcV49fmyjgvyuSwfcVXuIzELA3scTXQoPjJZqLS\r\n    t8YZeHm2fsQgEX28QqbW27SBMPIF7juznZhge2c8TmNN3AWdZOBFY3qFYYKH\r\n    aMG20j9XMXxEY2dqrw33XBcc2D/czO0rdmzK/XBdotMWqpZAgyE0k86FfG35\r\n    Weqd/PcTOCbwHFLZUHcSO5+2bFeTGHlf77ThLkNGYM7+RuWY2IY9X3f3PMa7\r\n    bWo0UaFFrEzfq23DVz32Sm5YDt0ygwgLV7XhnJ8B7a5gNtg8c/WZvNnzYdbN\r\n    KWfaU79yQmgbIda8PO33hXxCygAQMsPON5lWqXrulxjge+LhlDiIQvKkZA4P\r\n    pDL9k6MI81YzkBWjFeOkK4LCWsdiQylk0w0xNR9/PhwNIWCWsatsR4YK9kEF\r\n    4dKF4OooLO1xw/4P0BkdvHSP1EQm/CvAMdi7ZW48/j4yJbwNnwl9OBAizout\r\n    ZJ4dSun5hEnuRovB1OeyOz7E6Nc17F8Qrq0WRgaR0KzMxuyK9teHIu5GSoYg\r\n    e57sUEmDxrD0h1f1Vq4uFgplW504zSXCl1An7F/FGOXihNStze+KP+gnqq2T\r\n    iQ5OlcVorCIur8RRQwzRGayBiSjmDVlmpzclnqw6TytWBeY/q6rtMmBLoV3g\r\n    OqAOgiRiM5DexPw0ZxLnuJA2TACkn3Yeikka72YONTgZeU9ZPdgYM+d4y+hX\r\n    AvwaiIdOJBAoOHizAFBs4RcrRst88DftnZZAZh5CenV12nn0YF1eaoLeHMdH\r\n    XWkBxn5Ihi6mMTRlZ2tviHMalkIyJtXrwoeqHuWqq2N3r+FGWIJAqbmE1wIr\r\n    GvbObjMyGsLs3KJd0qtDxamU+gicMpf4gzW/dbFLuFhOiR8asrFvEZETssBW\r\n    YzwBN+i6izcmhbo8BzFBf9T9jQl56XQHrR1sn4GmEaemJQuXRSxovu8pMgYR\r\n    SW9cG4sF2JepxwSkuXyFx1gp02R0CBHE3D6oul4k1PZ+/qs6DvoOL+RE87BI\r\n    0o7LttNbp9hOrGhx6g7m9NoIEkemmj805G7MlDzyR9PuDygcC8Qzf4a8aCku\r\n    fs7ayAW7+hZxyxryE7V6V1caVHiCi6XQPw5RLuH6ukDttb4QltTguIch/qNS\r\n    AUKtWMJxjKLJxkKWv+W9GLpbhiXsfQAu1Z6Tkln7x3rsCiJNZPv76UhmiPcP\r\n    buor2LV7nbw6bjFkYKMSNwdJJG5iVV54zao7EZ0boxVzXba+iQFCUesDSaDg\r\n    auBBx5N9bXQRsidr+oi9RHwu9KE5RtPtACXgFrvnqtw7CPd4UbuyXOe+GZUP\r\n    66FHyBQUHhW21xvkcLb1L2E52RQuleJu71SYVVqsob5d+SH2QMbUjUsbTmiZ\r\n    OBvsMDl62sVSKfGMZL41LcMYmfxA4fEXZehIYgaTwpTPFo/Opme/nbcBoszT\r\n    OZzb96IUnXh9ej+9qFC+fB6yB5FDHQd/LFiP7ULaDlIoAXZYXjidwFWFGXld\r\n    PG4oqarLnLAxqy31olu39HERJq7+Fxw97++CejMvuOkWYjGCdYcOkZ70CNl1\r\n    40jimz00oACiv3/QqxcRyDluXzbs9iweJ+KbdISwxrOqC17l11LENa9Wybeg\r\n    tAIkR5OzNKEDM7cYdnDjnt4JXGo/3OiNo06W4AcNH+I0a/gu6SwozDc+htyP\r\n    gfhpGwBaeL+KPOEvn6JXLyvE6DBGGPWBr0JXewgOqx/gyOIp1zlqB1peBNKl\r\n    gOx8jd+PoYvEAqTjmDKzXHICUE8s07YKbyHlmxI9srv9Ffhw88+EuHuHAhfr\r\n    91JFOeWo49nwXA8tKUVOckypJHgb0475NkLeAII5275RYXBU4EO9jjdtY8px\r\n    YpkhgP6UTXHA+rDbutlp9sDXL+iftBXfH8bsyJqbBSYcgYMPgF15I702CQuB\r\n    xiG3xlw7Uq0Xi/2DiWGn14ibS4zW5QMaQWTWVlEYRQVO7/AWMGWU6/Blg3yM\r\n    vCAwhIFY59ZbLbHoNABI1jXatNYIM7Q3W0kIsfqaxZXqAHDzW5YvXo16r1L2\r\n    mWuNFGmZBVMG/qTfXDk9boORvxG0eLnfVmou9orNSke6eWX3L8/EIuYrEyFx\r\n    d6o8q96iN/zbn720NNj0C3wCEvitk9+2RhMqAbT2VcXz/HcZV2GNnI0FP03r\r\n    0XyqgBXo4snc/BOcO/9FdJ5kkAORHCdlTETZJydyI1oVaatIZoRZrqqi/jg3\r\n    9tYuEIHyBavZ3n8V8lupuWwv3dQcDJAz9QgqYZ4Lv3FbulLWEIPwyxsK8+1O\r\n    BhrOQScB1WdstQNCg+3Ihqo+RxaCK6nDZZrDQMCXTPtfS58n+b7+5WUpPVN4\r\n    8y/sOrYIDt2tAeaprRUnIZMupHEh4dE=\r\n    EOS\r\n    \r\n    data = Base64.decode64(b64_str)\r\n    data.force_encoding(\"UTF-8\")\r\n    \r\n    puts \"Before calling encode: data.length=#{data.length}\"\r\n    data = data.encode(\"UTF-16\", :undef => :replace, :invalid => :replace, :replace => '')\r\n    puts \"After calling encode: data.length=#{data.length}\"\r\n\r\n\r\nHere's me running it on MRI Ruby. It took less than 1 second:\r\n\r\n    $ ruby --version\r\n    ruby 2.0.0p353 (2013-11-22 revision 43784) [x86_64-darwin14.0.0]\r\n    $ time ~/tmp/encoding_test.rb \r\n    Before calling encode: data.length=1965\r\n    After calling encode: data.length=2242\r\n    \r\n    real  0m0.095s\r\n    user  0m0.054s\r\n    sys 0m0.040s\r\n\r\n\r\nHere's me running it on the latest stable JRuby. I gave up after 4 minutes:\r\n\r\n    $ ruby --version\r\n    jruby 1.7.19 (1.9.3p551) 2015-01-29 20786bd on Java HotSpot(TM) 64-Bit Server VM 1.7.0_75-b13 +jit [darwin-x86_64]\r\n    $ time ~/tmp/encoding_test.rb \r\n    Before calling encode: data.length=1965\r\n    ^C\r\n    real  4m33.090s\r\n    user  4m33.700s\r\n    sys 0m0.770s\r\n\r\n\r\nHere's what that thread was doing before I killed it:\r\n\r\n    $ jstack 69020\r\n    \"main\" prio=5 tid=0x00007fd3aa802800 nid=0xf07 runnable [0x0000000109060000]\r\n       java.lang.Thread.State: RUNNABLE\r\n            at org.jruby.util.encoding.CharsetTranscoder$TranscoderEngine.encode(CharsetTranscoder.java:660)\r\n            at org.jruby.util.encoding.CharsetTranscoder$TranscoderEngine.transcode(CharsetTranscoder.java:530)\r\n            at org.jruby.util.encoding.CharsetTranscoder.primitiveConvert(CharsetTranscoder.java:333)\r\n            at org.jruby.util.encoding.CharsetTranscoder.transcode(CharsetTranscoder.java:294)\r\n            at org.jruby.util.encoding.CharsetTranscoder.transcode(CharsetTranscoder.java:236)\r\n            at org.jruby.util.io.EncodingUtils.transcodeLoop(EncodingUtils.java:874)\r\n            at org.jruby.util.io.EncodingUtils.strTranscode0(EncodingUtils.java:802)\r\n            at org.jruby.util.io.EncodingUtils.strTranscode(EncodingUtils.java:737)\r\n            at org.jruby.util.io.EncodingUtils.strEncode(EncodingUtils.java:708)\r\n            at org.jruby.RubyString.encode(RubyString.java:7619)\r\n            at org.jruby.RubyString$INVOKER$i$encode.call(RubyString$INVOKER$i$encode.gen)\r\n            at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:346)\r\n            at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:204)\r\n            at Users.mscorcio.tmp.encoding_test.__file__(/Users/mscorcio/tmp/encoding_test.rb:58)\r\n            at Users.mscorcio.tmp.encoding_test.load(/Users/mscorcio/tmp/encoding_test.rb)\r\n            at org.jruby.Ruby.runScript(Ruby.java:866)\r\n            at org.jruby.Ruby.runScript(Ruby.java:859)\r\n            at org.jruby.Ruby.runNormally(Ruby.java:728)\r\n            at org.jruby.Ruby.runFromMain(Ruby.java:577)\r\n            at org.jruby.Main.doRunFromMain(Main.java:395)\r\n            at org.jruby.Main.internalRun(Main.java:290)\r\n            at org.jruby.Main.run(Main.java:217)\r\n            at org.jruby.Main.main(Main.java:197)\r\n"},"comment":{"url":"https://api.github.com/repos/jruby/jruby/issues/comments/97303463","html_url":"https://github.com/jruby/jruby/issues/2856#issuecomment-97303463","issue_url":"https://api.github.com/repos/jruby/jruby/issues/2856","id":97303463,"user":{"login":"headius","id":10135,"avatar_url":"https://avatars.githubusercontent.com/u/10135?v=3","gravatar_id":"","url":"https://api.github.com/users/headius","html_url":"https://github.com/headius","followers_url":"https://api.github.com/users/headius/followers","following_url":"https://api.github.com/users/headius/following{/other_user}","gists_url":"https://api.github.com/users/headius/gists{/gist_id}","starred_url":"https://api.github.com/users/headius/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/headius/subscriptions","organizations_url":"https://api.github.com/users/headius/orgs","repos_url":"https://api.github.com/users/headius/repos","events_url":"https://api.github.com/users/headius/events{/privacy}","received_events_url":"https://api.github.com/users/headius/received_events","type":"User","site_admin":false},"created_at":"2015-04-29T04:47:55Z","updated_at":"2015-04-29T04:47:55Z","body":"I could not get master to fail in any case, but this isn't surprising...the transcoder is now identical to MRI's. \r\n\r\nJRuby 1.7 worked ok on Java 8u40, but on Java 7u67 I was able to reproduce your results with both your original script and on the longer-running random search.\r\n\r\nSo it seems there may be a Java bug here. I will investigate a bit to see if I can improve my transcoder to avoid this problem."}}