{"action":"created","issue":{"url":"https://api.github.com/repos/joyent/node/issues/25583","labels_url":"https://api.github.com/repos/joyent/node/issues/25583/labels{/name}","comments_url":"https://api.github.com/repos/joyent/node/issues/25583/comments","events_url":"https://api.github.com/repos/joyent/node/issues/25583/events","html_url":"https://github.com/joyent/node/issues/25583","id":91329699,"number":25583,"title":"Segfault on node 0.12.4, to do with unicode and buffers","user":{"login":"myndzi","id":1588419,"avatar_url":"https://avatars.githubusercontent.com/u/1588419?v=3","gravatar_id":"","url":"https://api.github.com/users/myndzi","html_url":"https://github.com/myndzi","followers_url":"https://api.github.com/users/myndzi/followers","following_url":"https://api.github.com/users/myndzi/following{/other_user}","gists_url":"https://api.github.com/users/myndzi/gists{/gist_id}","starred_url":"https://api.github.com/users/myndzi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/myndzi/subscriptions","organizations_url":"https://api.github.com/users/myndzi/orgs","repos_url":"https://api.github.com/users/myndzi/repos","events_url":"https://api.github.com/users/myndzi/events{/privacy}","received_events_url":"https://api.github.com/users/myndzi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-06-26T18:38:13Z","updated_at":"2015-07-02T13:57:54Z","closed_at":null,"body":"I have a core dump now from a binary compiled with symbols, but I'm not sure what to do with it. I've managed to dump the data from the buffer being operated on to a file, but I cannot reproduce the crash in javascript code. I'm uncertain how exactly things got passed through and along. The backtrace looks like this:\r\n\r\n    #0  v8::base::OS::Abort () at ../deps/v8/src/base/platform/platform-posix.cc:278\r\n    #1  0x08bfd8b4 in V8_Fatal (file=0x8d58527 \"../deps/v8/src/unicode.cc\", line=320, format=0x8d58516 \"CHECK(%s) failed\") at ../deps/v8/src/base/logging.cc:87\r\n    #2  0x08a8d695 in unibrow::Utf8DecoderBase::WriteUtf16Slow (stream=0xb24203d \"<data>\"..., data=0x5b4d7c3e, data_length=1) at ../deps/v8/src/unicode.cc:320\r\n    #3  0x0879f02e in unibrow::Utf8Decoder<512u>::WriteUtf16 (this=0xa10ba40, data=0x5b4d3ff8, length=7714) at ../deps/v8/src/unicode-inl.h:197\r\n    #4  0x0878ed6d in v8::internal::Factory::NewStringFromUtf8 (this=0xa0fe080, string=..., pretenure=v8::internal::NOT_TENURED) at ../deps/v8/src/factory.cc:263\r\n    #5  0x0862c24c in v8::(anonymous namespace)::NewString (factory=0xa0fe080, type=v8::String::kNormalString, string=...) at ../deps/v8/src/api.cc:5333\r\n    #6  0x0863b37d in v8::(anonymous namespace)::NewString<char> (v8_isolate=0xa0fe080, location=0x8c4d06c \"v8::String::NewFromUtf8()\", env=0x8c4d058 \"String::NewFromUtf8\", data=0xb2401a0 \"<data>\"..., type=v8::String::kNormalString, length=7834) at ../deps/v8/src/api.cc:5378\r\n    #7  0x0862c37d in v8::String::NewFromUtf8 (isolate=0xa0fe080, data=0xb2401a0 \"<data>\"..., type=v8::String::kNormalString, length=7834) at ../deps/v8/src/api.cc:5397\r\n    #8  0x08b99bfd in node::StringBytes::Encode (isolate=0xa0fe080, buf=0xb2401a0 \"<data>\"..., buflen=7834, encoding=node::UTF8) at ../src/string_bytes.cc:727\r\n    #9  0x08b69f43 in node::Buffer::StringSlice<(node::encoding)1> (args=...) at ../src/node_buffer.cc:266\r\n    #10 0x08b685a5 in node::Buffer::Utf8Slice (args=...) at ../src/node_buffer.cc:281\r\n    #11 0x08649a59 in v8::internal::FunctionCallbackArguments::Call (this=0xbf7fe95c, f=0x8b68594 <node::Buffer::Utf8Slice(v8::FunctionCallbackInfo<v8::Value> const&)>) at ../deps/v8/src/arguments.cc:33\r\n    #12 0x0867f8a5 in v8::internal::HandleApiCallHelper<false> (args=..., isolate=0xa0fe080) at ../deps/v8/src/builtins.cc:1144\r\n    #13 0x0867a996 in v8::internal::Builtin_Impl_HandleApiCall (args=..., isolate=0xa0fe080) at ../deps/v8/src/builtins.cc:1161\r\n    #14 0x0867a975 in v8::internal::Builtin_HandleApiCall (args_length=4, args_object=0xbf7fea34, isolate=0xa0fe080) at ../deps/v8/src/builtins.cc:1160\r\n    #15 0x5340a3f6 in ?? ()\r\n    #16 0x5347d7ea in ?? ()\r\n    #17 0x5340b93b in ?? ()\r\n    #18 0x5563ccc4 in ?? ()\r\n    #19 0x534332f7 in ?? ()\r\n    #20 0x53430b73 in ?? ()\r\n    #21 0x5f01ad25 in ?? ()\r\n    #22 0x53b36df8 in ?? ()\r\n    #23 0xb152490f in ?? ()\r\n    #24 0x5340b93b in ?? ()\r\n    #25 0x2b9bc4b4 in ?? ()\r\n    #26 0x2b97af56 in ?? ()\r\n    #27 0x53bf31e2 in ?? ()\r\n    #28 0x5340b93b in ?? ()\r\n    #29 0x53b3bb00 in ?? ()\r\n    #30 0x53447015 in ?? ()\r\n    #31 0x5342574a in ?? ()\r\n    #32 0x0877cdbf in v8::internal::Invoke (is_construct=false, function=..., receiver=..., argc=2, args=0xbf7feeec) at ../deps/v8/src/execution.cc:91\r\n    #33 0x0877d148 in v8::internal::Execution::Call (isolate=0xa0fe080, callable=..., receiver=..., argc=2, argv=0xbf7feeec, convert_receiver=true) at ../deps/v8/src/execution.cc:141\r\n    #34 0x086287ff in v8::Function::Call (this=0xa128928, recv=..., argc=2, argv=0xbf7feeec) at ../deps/v8/src/api.cc:4020\r\n    #35 0x08b4bae2 in node::AsyncWrap::MakeCallback (this=0xa13c8b8, cb=..., argc=2, argv=0xbf7feeec) at ../src/async-wrap.cc:136\r\n    #36 0x08b4d650 in node::AsyncWrap::MakeCallback (this=0xa13c8b8, symbol=..., argc=2, argv=0xbf7feeec) at ../src/async-wrap-inl.h:99\r\n    #37 0x08b8f596 in node::ZCtx::After (work_req=0xa13c92c, status=0) at ../src/node_zlib.cc:341\r\n    #38 0x08be72c6 in uv__queue_done (w=0xa13c958, err=0) at ../deps/uv/src/threadpool.c:257\r\n    #39 0x08be720a in uv__work_done (handle=0x917c3a0 <default_loop_struct+96>) at ../deps/uv/src/threadpool.c:236\r\n    #40 0x08be9106 in uv__async_event (loop=0x917c340 <default_loop_struct>, w=0x917c42c <default_loop_struct+236>, nevents=1) at ../deps/uv/src/unix/async.c:92\r\n    #41 0x08be9263 in uv__async_io (loop=0x917c340 <default_loop_struct>, w=0x917c430 <default_loop_struct+240>, events=1) at ../deps/uv/src/unix/async.c:132\r\n    #42 0x08bfa7d0 in uv__io_poll (loop=0x917c340 <default_loop_struct>, timeout=0) at ../deps/uv/src/unix/linux-core.c:324\r\n    #43 0x08be9c87 in uv_run (loop=0x917c340 <default_loop_struct>, mode=UV_RUN_ONCE) at ../deps/uv/src/unix/core.c:324\r\n    #44 0x08b61c41 in node::Start (argc=2, argv=0xa0fe008) at ../src/node.cc:3722\r\n    #45 0x08b88b58 in main (argc=2, argv=0xbf802664) at ../src/node_main.cc:65\r\n\r\n(I've omitted the actual data snippits for readability)\r\n\r\nAs best as I can tell, the buffer being worked with is split on the first byte of a four-byte utf-8 sequence representing a > 0xFFFF value (U+1F495); it's part of an HTTP response. The error is obvious:\r\n\r\nhttps://github.com/joyent/node/blob/v0.12.4/deps/v8/src/unicode.cc#L317-L321\r\n\r\nAn assumption is made here, that if there's a UTF-16 surrogate pair, both parts exist, but that's not what's happening. It seems assumed that such a case won't reach this code. I was unable to follow the code here well enough to come up with a way to reproduce the problem in JS code:\r\n\r\nhttps://github.com/joyent/node/blob/v0.12.4/deps/v8/src/factory.cc#L231-L265\r\n\r\nPlease let me know what if anything I can do to help nail this down further!"},"comment":{"url":"https://api.github.com/repos/joyent/node/issues/comments/118041728","html_url":"https://github.com/joyent/node/issues/25583#issuecomment-118041728","issue_url":"https://api.github.com/repos/joyent/node/issues/25583","id":118041728,"user":{"login":"myndzi","id":1588419,"avatar_url":"https://avatars.githubusercontent.com/u/1588419?v=3","gravatar_id":"","url":"https://api.github.com/users/myndzi","html_url":"https://github.com/myndzi","followers_url":"https://api.github.com/users/myndzi/followers","following_url":"https://api.github.com/users/myndzi/following{/other_user}","gists_url":"https://api.github.com/users/myndzi/gists{/gist_id}","starred_url":"https://api.github.com/users/myndzi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/myndzi/subscriptions","organizations_url":"https://api.github.com/users/myndzi/orgs","repos_url":"https://api.github.com/users/myndzi/repos","events_url":"https://api.github.com/users/myndzi/events{/privacy}","received_events_url":"https://api.github.com/users/myndzi/received_events","type":"User","site_admin":false},"created_at":"2015-07-02T13:57:54Z","updated_at":"2015-07-02T13:57:54Z","body":"For what it's worth, even if it doesn't crash, your data will get messed up if it cuts in the middle of a UTF-8 sequence, if you're stringifying each chunk. This could happen because people don't know how to work with buffers and are just using setEncoding, or they're piping to some streaming module that does the same (htmlparser2 operates with strings). I wrote this: https://www.npmjs.com/package/utf8-align-stream which should ensure that a Node stream always outputs whole UTF-8 sequences when they're available by conditionally buffering the last <= 5 bytes. This will make the strings compose correctly as well as prevent this crash. "}}