{"action":"created","issue":{"url":"https://api.github.com/repos/aerospike/aerospike-server/issues/103","labels_url":"https://api.github.com/repos/aerospike/aerospike-server/issues/103/labels{/name}","comments_url":"https://api.github.com/repos/aerospike/aerospike-server/issues/103/comments","events_url":"https://api.github.com/repos/aerospike/aerospike-server/issues/103/events","html_url":"https://github.com/aerospike/aerospike-server/issues/103","id":115852777,"number":103,"title":"Segmentation fault","user":{"login":"pjrtb","id":10600841,"avatar_url":"https://avatars.githubusercontent.com/u/10600841?v=3","gravatar_id":"","url":"https://api.github.com/users/pjrtb","html_url":"https://github.com/pjrtb","followers_url":"https://api.github.com/users/pjrtb/followers","following_url":"https://api.github.com/users/pjrtb/following{/other_user}","gists_url":"https://api.github.com/users/pjrtb/gists{/gist_id}","starred_url":"https://api.github.com/users/pjrtb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pjrtb/subscriptions","organizations_url":"https://api.github.com/users/pjrtb/orgs","repos_url":"https://api.github.com/users/pjrtb/repos","events_url":"https://api.github.com/users/pjrtb/events{/privacy}","received_events_url":"https://api.github.com/users/pjrtb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2015-11-09T10:57:45Z","updated_at":"2015-11-10T10:19:29Z","closed_at":null,"body":"Hi,\r\n\r\nI have two crashing instances of aerospike CE (3.6.0, downgraded from 3.63 which also was crashing).\r\n\r\n```\r\nldd $(which asd)\r\n        linux-vdso.so.1 =>  (0x00007fff5cdef000)\r\n        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fca955d0000)\r\n        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fca953c8000)\r\n        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fca951c4000)\r\n        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fca94fab000)\r\n        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fca94ca5000)\r\n        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fca94a8f000)\r\n        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fca946ca000)\r\n        /lib64/ld-linux-x86-64.so.2 (0x00007fca957ee000)\r\n```\r\n\r\nOS: Ubuntu 14.04.3 LTS\r\nlibc6 - 2.19-0ubuntu6.6\r\nlibgcc1 - 1:4.9.1-0ubuntu1\r\nzlib1g - 1:1.2.8.dfsg-1ubuntu1\r\n\r\nbacktrace from gdb:\r\n```\r\n#0  0x000000000058fc9d in je_bitmap_set (bit=18446744073709551555, binfo=0x9fb380 <je_arena_bin_info+768>,\r\n    bitmap=0x7fffd619d010) at include/jemalloc/internal/bitmap.h:101\r\n#1  je_bitmap_sfu (binfo=0x9fb380 <je_arena_bin_info+768>, bitmap=0x7fffd619d010)\r\n    at include/jemalloc/internal/bitmap.h:140\r\n#2  arena_run_reg_alloc (bin_info=0x9fb358 <je_arena_bin_info+728>, run=0x7fffd619d000) at src/arena.c:291\r\n#3  je_arena_tcache_fill_small (arena=0x7ffff65f0280, tbin=0x7fffc6450108, binind=7, prof_accumbytes=<optimized out>)\r\n    at src/arena.c:1479\r\n#4  0x00000000005a810f in je_tcache_alloc_small_hard (tcache=<optimized out>, tbin=0x7fffc6450108,\r\n    binind=<optimized out>) at src/tcache.c:72\r\n#5  0x0000000000585c94 in je_tcache_alloc_small (size=112, tcache=0x7fffc6450000, zero=false)\r\n    at include/jemalloc/internal/tcache.h:303\r\n#6  je_arena_malloc (zero=false, size=<optimized out>, arena=0x0, try_tcache=true)\r\n    at include/jemalloc/internal/arena.h:957\r\n#7  je_imalloct (arena=0x0, try_tcache=true, size=<optimized out>)\r\n    at include/jemalloc/internal/jemalloc_internal.h:771\r\n#8  je_imalloc (size=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:780\r\n#9  malloc (size=<optimized out>) at src/jemalloc.c:929\r\n#10 0x00000000004ff12d in cf_malloc_at (sz=105, file=0x665270 \"base/thr_demarshal.c\", line=591) at alloc.c:1470\r\n#11 0x000000000048c614 in thr_demarshal (arg=<optimized out>) at base/thr_demarshal.c:591\r\n#12 0x00007ffff7bc4182 in start_thread (arg=0x7ffd5c75c700) at pthread_create.c:312\r\n#13 0x00007ffff6db047d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111\r\n```\r\n\r\ntail from log:\r\n```\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:rw_dup_prole:2136) [REC NOT FOUND:1]<rw_dup_prole()>PID(193) Pstate(1):<Digest>:0xc120db8e859f5a7def2b706fae709671703eac15\r\nNov 09 2015 09:55:30 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 23932 changed from arena #150 ==> #150\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:as_rw_start:1197) [PROCESS KEY] {iassd:2045} Self(bb9d6288b902500) Read(0):<Digest>:0xfdb7814a944d0ca24849c2555e1898f8d86b9495\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:internal_rw_start:807) {iassd:2045} as_write_start: duplicate partitions encountered 1 bb9abb08a902500\r\nNov 09 2015 09:55:30 GMT: DEBUG (demarshal): (thr_demarshal.c:thr_demarshal:472) receiving pipelined request\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:internal_rw_start:1103) write: sending request to 1 nodes\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:send_messages:366) resending rw request: {iassd:2045} node bb9abb08a902500 digest a20c4d944a81b7fd\r\nNov 09 2015 09:55:30 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 23931 changed from arena #24 ==> #150\r\nNov 09 2015 09:55:30 GMT: DEBUG (tsvc): (thr_tsvc.c:process_transaction:542) [WRITE CASE]: Partition CK(ba4b040ae66096e)\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:as_rw_start:1197) [PROCESS KEY] {iassd:488} Self(bb9d6288b902500) Read(0):<Digest>:0xe8910facf3ff072ac4b56048e9e3274628aa291c\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:internal_rw_start:807) {iassd:488} as_write_start: duplicate partitions encountered 1 bb9abb08a902500\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:internal_rw_start:1103) write: sending request to 1 nodes\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:send_messages:366) resending rw request: {iassd:488} node bb9abb08a902500 digest 2a07fff3ac0f91e8\r\nNov 09 2015 09:55:30 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 23925 changed from arena #150 ==> #150\r\nNov 09 2015 09:55:30 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 23937 changed from arena #150 ==> #150\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:write_process:2851) Write Pickled: PID(3793) PState(1) Gen(63):<Digest>:0xd1fef5ffb532187028513ad9755abcd71b93fef6\r\nNov 09 2015 09:55:30 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 23933 changed from arena #150 ==> #150\r\nNov 09 2015 09:55:30 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 23929 changed from arena #150 ==> #24\r\nNov 09 2015 09:55:30 GMT: DEBUG (tsvc): (thr_tsvc.c:process_transaction:542) [WRITE CASE]: Partition CK(ba4b040ae66096e)\r\nNov 09 2015 09:55:30 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 23936 changed from arena #150 ==> #150\r\nNov 09 2015 09:55:30 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 23934 changed from arena #150 ==> #150\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:as_rw_start:1197) [PROCESS KEY] {iassd:3450} Self(bb9d6288b902500) Read(0):<Digest>:0x7a6db9b11841c4459d456acce542e90adb7411c0\r\nNov 09 2015 09:55:30 GMT: DEBUG (demarshal): (thr_demarshal.c:thr_demarshal:472) receiving pipelined request\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:internal_rw_start:807) {iassd:3450} as_write_start: duplicate partitions encountered 1 bb9abb08a902500\r\nNov 09 2015 09:55:30 GMT: DEBUG (tsvc): (thr_tsvc.c:process_transaction:542) [WRITE CASE]: Partition CK(ba4b040ae66096e)\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:internal_rw_start:1103) write: sending request to 1 nodes\r\nNov 09 2015 09:55:30 GMT: DEBUG (tsvc): (thr_tsvc.c:process_transaction:542) [WRITE CASE]: Partition CK(ba4b040ae66096e)\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:send_messages:366) resending rw request: {iassd:3450} node bb9abb08a902500 digest 45c44118b1b96d7a\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:as_rw_start:1197) [PROCESS KEY] {iassd:921} Self(bb9d6288b902500) Read(0):<Digest>:0x99a303476d8fdf374b8657b8c9c93fc28cbbfa7b\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:rw_process_ack:1911) {iassd:586} rw_process_ack: Processing response 377401554d18024a WRITE\r\nNov 09 2015 09:55:30 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 23931 changed from arena #150 ==> #150\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:internal_rw_start:807) {iassd:921} as_write_start: duplicate partitions encountered 1 bb9abb08a902500\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:as_rw_start:1197) [PROCESS KEY] {iassd:501} Self(bb9d6288b902500) Read(0):<Digest>:0xf521b2d962948be061e553698318ccb4bf6c735f\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:internal_rw_start:1103) write: sending request to 1 nodes\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:internal_rw_start:807) {iassd:501} as_write_start: duplicate partitions encountered 1 bb9abb08a902500\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:send_messages:366) resending rw request: {iassd:921} node bb9abb08a902500 digest 37df8f6d4703a399\r\nNov 09 2015 09:55:30 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 23932 changed from arena #150 ==> #150\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:internal_rw_start:1103) write: sending request to 1 nodes\r\nNov 09 2015 09:55:30 GMT: DEBUG (cf:jem): (jem.c:jem_set_arena:182) TID 23931 changed from arena #150 ==> #150\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:send_messages:366) resending rw request: {iassd:501} node bb9abb08a902500 digest e08b9462d9b221f5\r\nNov 09 2015 09:55:30 GMT: DEBUG (tsvc): (thr_tsvc.c:process_transaction:542) [WRITE CASE]: Partition CK(ba4b040ae66096e)\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:write_process:2851) Write Pickled: PID(832) PState(1) Gen(279):<Digest>:0x404362dd40db2e5c86bb442f23db18157a9e822d\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:rw_process_ack:1911) {iassd:2045} rw_process_ack: Processing response a20c4d944a81b7fd WRITE\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:as_rw_start:1197) [PROCESS KEY] {iassd:1651} Self(bb9d6288b902500) Read(0):<Digest>:0x73263891a6e839bf38e5c7a91fd6dfc4b430a733\r\nNov 09 2015 09:55:30 GMT: DEBUG (record): (record.c:as_partition_vinfoset_unpickle:1764) received vinfo set with 0 size, unusual RW\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:finish_rw_process_dup_ack:1627) finish_rw_process_dup_ack: received message with record properties\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:internal_rw_start:807) {iassd:1651} as_write_start: duplicate partitions encountered 1 bb9abb08a902500\r\nNov 09 2015 09:55:30 GMT: DEBUG (record): (record.c:as_record_flatten:1215) flatten start: \r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:internal_rw_start:1103) write: sending request to 1 nodes\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:send_messages:366) resending rw request: {iassd:1651} node bb9abb08a902500 digest bf39e8a691382673\r\nNov 09 2015 09:55:30 GMT: DEBUG (record): (record.c:as_partition_vinfoset_unpickle:1764) received vinfo set with 0 size, unusual RW\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:write_process:2851) Write Pickled: PID(2006) PState(1) Gen(3):<Digest>:0xd6376836cda8c43b02014a510396fcf23e4c711a\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:internal_rw_start:1103) write: sending request to 1 nodes\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:finish_rw_process_dup_ack:1627) finish_rw_process_dup_ack: received message with record properties\r\nNov 09 2015 09:55:30 GMT: DEBUG (rw): (thr_rw.c:send_messages:366) resending rw request: {iassd:2045} node bb9abb08a902500 digest a20c4d944a81b7fd\r\n```"},"comment":{"url":"https://api.github.com/repos/aerospike/aerospike-server/issues/comments/155379088","html_url":"https://github.com/aerospike/aerospike-server/issues/103#issuecomment-155379088","issue_url":"https://api.github.com/repos/aerospike/aerospike-server/issues/103","id":155379088,"user":{"login":"xiaosuo","id":105033,"avatar_url":"https://avatars.githubusercontent.com/u/105033?v=3","gravatar_id":"","url":"https://api.github.com/users/xiaosuo","html_url":"https://github.com/xiaosuo","followers_url":"https://api.github.com/users/xiaosuo/followers","following_url":"https://api.github.com/users/xiaosuo/following{/other_user}","gists_url":"https://api.github.com/users/xiaosuo/gists{/gist_id}","starred_url":"https://api.github.com/users/xiaosuo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiaosuo/subscriptions","organizations_url":"https://api.github.com/users/xiaosuo/orgs","repos_url":"https://api.github.com/users/xiaosuo/repos","events_url":"https://api.github.com/users/xiaosuo/events{/privacy}","received_events_url":"https://api.github.com/users/xiaosuo/received_events","type":"User","site_admin":false},"created_at":"2015-11-10T10:19:29Z","updated_at":"2015-11-10T10:19:29Z","body":"I have the same problem, and I have batch requests. At the same time, the same issue happens with 3.6.3."}}