{"action":"created","issue":{"url":"https://api.github.com/repos/mozilla/rr/issues/1481","labels_url":"https://api.github.com/repos/mozilla/rr/issues/1481/labels{/name}","comments_url":"https://api.github.com/repos/mozilla/rr/issues/1481/comments","events_url":"https://api.github.com/repos/mozilla/rr/issues/1481/events","html_url":"https://github.com/mozilla/rr/issues/1481","id":75658441,"number":1481,"title":"trying to reverse-step a recording of QEMU asserts: FATAL /home/petmay01/src/rr/src/ReplaySession.cc:315:cont_syscall_boundary() errno: 0 'Success'","user":{"login":"pm215","id":1682716,"avatar_url":"https://avatars.githubusercontent.com/u/1682716?v=3","gravatar_id":"","url":"https://api.github.com/users/pm215","html_url":"https://github.com/pm215","followers_url":"https://api.github.com/users/pm215/followers","following_url":"https://api.github.com/users/pm215/following{/other_user}","gists_url":"https://api.github.com/users/pm215/gists{/gist_id}","starred_url":"https://api.github.com/users/pm215/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pm215/subscriptions","organizations_url":"https://api.github.com/users/pm215/orgs","repos_url":"https://api.github.com/users/pm215/repos","events_url":"https://api.github.com/users/pm215/events{/privacy}","received_events_url":"https://api.github.com/users/pm215/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2015-05-12T16:27:00Z","updated_at":"2015-05-17T21:34:16Z","closed_at":null,"body":"I have a recording of QEMU which will replay forwards fine. However attempting a reverse-step causes an rr assertion:\r\n\r\n```\r\ne104462:trusty:qemu$ ~/src/rr-obj64/bin/rr replay ~/.rr/qemu-system-arm-6\r\nGNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1\r\nCopyright (C) 2014 Free Software Foundation, Inc.\r\nLicense GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\r\nThis is free software: you are free to change and redistribute it.\r\nThere is NO WARRANTY, to the extent permitted by law.  Type \"show copying\"\r\nand \"show warranty\" for details.\r\nThis GDB was configured as \"x86_64-linux-gnu\".\r\nType \"show configuration\" for configuration details.\r\nFor bug reporting instructions, please see:\r\n<http://www.gnu.org/software/gdb/bugs/>.\r\nFind the GDB manual and other documentation resources online at:\r\n<http://www.gnu.org/software/gdb/documentation/>.\r\nFor help, type \"help\".\r\nType \"apropos word\" to search for commands related to \"word\"...\r\nReading symbols from /home/petmay01/linaro/qemu-from-laptop/qemu/build/a64-targets-nodbg/arm-softmmu/qemu-system-arm...done.\r\nReally redefine built-in command \"checkpoint\"? (y or n) [answered Y; input not from terminal]\r\nReally redefine built-in command \"checkpoint\"? (y or n) [answered Y; input not from terminal]\r\nReally redefine built-in command \"restart\"? (y or n) [answered Y; input not from terminal]\r\n/proc/28299/fd/4:53: Error in sourced command file:\r\nUndefined maintenance set command: \"target-async 0\".  Try \"help maintenance set\".\r\nRemote debugging using 127.0.0.1:28300\r\nwarning: limiting remote suggested packet size (206696 bytes) to 16384\r\nReading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.19.so...done.\r\ndone.\r\nLoaded symbols for /lib64/ld-linux-x86-64.so.2\r\n0x00002aaaaaaafa86 in _dl_start (arg=0x7fffffffe330) at rtld.c:380\r\n380     rtld.c: No such file or directory.\r\n(gdb) handle SIGUSR1 pass noprint nostop\r\nSignal        Stop      Print   Pass to program Description\r\nSIGUSR1       No        No      Yes             User defined signal 1\r\n(gdb) break spitz_out_switch\r\nBreakpoint 1 at 0x5555556b8490: file /home/petmay01/linaro/qemu-from-laptop/qemu/hw/arm/spitz.c, line 774.\r\n(gdb) c\r\nContinuing.\r\naudio: Could not init `oss' audio driver\r\n[New Thread 28035.28037]\r\n[Switching to Thread 28035.28037]\r\n\r\nBreakpoint 1, spitz_out_switch (opaque=0x5555563664b0, line=0, level=1)\r\n    at /home/petmay01/linaro/qemu-from-laptop/qemu/hw/arm/spitz.c:774\r\n774     {\r\n(gdb) rs\r\n[FATAL /home/petmay01/src/rr/src/ReplaySession.cc:315:cont_syscall_boundary() errno: 0 'Success'] \r\n (task 28324 (rec:28037) at time 6373)\r\n ->  Assertion `false' failed to hold. Replay got unrecorded signal 11\r\nLaunch gdb with\r\n  gdb /home/petmay01/linaro/qemu-from-laptop/qemu/build/a64-targets-nodbg/arm-softmmu/qemu-system-arm\r\nand attach to the rr debug server with:\r\n  target remote :28324\r\n```\r\n\r\nThis is with rr built from git commit 89e53a6176a058270a2fde91c894edb740ce3a4c.\r\n\r\nReverse-step in a simple hello-world app seems to work OK, so it must be something specific to what QEMU is doing.\r\n\r\nDump of the event log around time 6373 in case that is helpful:\r\n\r\n```\r\n  global_time:6371, event:`SYSCALL: ppoll' (state:ENTERING_SYSCALL) tid:28035, ticks:43500534\r\n rax:0xffffffffffffffda rbx:0x55555634fa00 rcx:0xffffffffffffffff rdx:0x7fffffffdf60 rsi:0x5 rdi:0x5555569cea70 rbp:0x7fffffffdfb4 rsp:0x7fffffffdf40 r8:0x8 r9:0x0 r10:0x0 r11:0x293 r12:0x17ce62c r13:0xf r14:0x0 r15:0x555556331b30 rip:0x2aaab26181ef eflags:0x293 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10f}\r\n{\r\n  global_time:6372, event:`SYSCALL: futex' (state:EXITING_SYSCALL) tid:28037, ticks:20375577966\r\n rax:0x0 rbx:0x55555637bbb0 rcx:0xffffffffffffffff rdx:0x2 rsi:0x80 rdi:0x555555e9e100 rbp:0x555555e9a860 rsp:0x2aaac37e08f8 r8:0x555555e9e100 r9:0xb r10:0x0 r11:0x246 r12:0x555555e451a8 r13:0x555555dda2e0 r14:0x555556373960 r15:0x80000000 rip:0x2aaab231bf2c eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca}\r\n{\r\n  global_time:6373, event:`SYSCALL: clock_gettime' (state:ENTERING_SYSCALL) tid:28037, ticks:20375583681\r\n rax:0xffffffffffffffda rbx:0xfffffffffffffe50 rcx:0xffffffffffffffff rdx:0xffffffffffd047b0 rsi:0x2aaac37e01c0 rdi:0x1 rbp:0x0 rsp:0x2aaac37e0078 r8:0x1 r9:0xffffffff r10:0x55555637cbb0 r11:0x216 r12:0x2aaac37e01c0 r13:0x1 r14:0x4 r15:0x2aaac37e0168 rip:0x70000018 eflags:0x216 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe4}\r\n{\r\n  global_time:6374, event:`SYSCALL: clock_gettime' (state:EXITING_SYSCALL) tid:28037, ticks:20375583681\r\n rax:0x0 rbx:0xfffffffffffffe50 rcx:0xffffffffffffffff rdx:0xffffffffffd047b0 rsi:0x2aaac37e01c0 rdi:0x1 rbp:0x0 rsp:0x2aaac37e0078 r8:0x1 r9:0xffffffff r10:0x55555637cbb0 r11:0x216 r12:0x2aaac37e01c0 r13:0x1 r14:0x4 r15:0x2aaac37e0168 rip:0x70000018 eflags:0x216 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe4}\r\n{\r\n  global_time:6375, event:`SYSCALL: clock_gettime' (state:ENTERING_SYSCALL) tid:28037, ticks:20375584796\r\n rax:0xffffffffffffffda rbx:0xfffffffffffffe50 rcx:0xffffffffffffffff rdx:0xffffffffffd047b0 rsi:0x2aaac37e01c0 rdi:0x1 rbp:0x0 rsp:0x2aaac37e0078 r8:0x1 r9:0xffffffff r10:0x5555556566b0 r11:0x216 r12:0x2aaac37e01c0 r13:0x1 r14:0x4 r15:0x2aaac37e0168 rip:0x70000018 eflags:0x216 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe4}\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/mozilla/rr/issues/comments/102860677","html_url":"https://github.com/mozilla/rr/issues/1481#issuecomment-102860677","issue_url":"https://api.github.com/repos/mozilla/rr/issues/1481","id":102860677,"user":{"login":"pm215","id":1682716,"avatar_url":"https://avatars.githubusercontent.com/u/1682716?v=3","gravatar_id":"","url":"https://api.github.com/users/pm215","html_url":"https://github.com/pm215","followers_url":"https://api.github.com/users/pm215/followers","following_url":"https://api.github.com/users/pm215/following{/other_user}","gists_url":"https://api.github.com/users/pm215/gists{/gist_id}","starred_url":"https://api.github.com/users/pm215/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pm215/subscriptions","organizations_url":"https://api.github.com/users/pm215/orgs","repos_url":"https://api.github.com/users/pm215/repos","events_url":"https://api.github.com/users/pm215/events{/privacy}","received_events_url":"https://api.github.com/users/pm215/received_events","type":"User","site_admin":false},"created_at":"2015-05-17T21:34:16Z","updated_at":"2015-05-17T21:34:16Z","body":"That pull request addresses the lack of PROT_GROWSDOWN support (though you should check it carefully before merging because I'm kind of blundering about as a non-native-speaker of C++ :-)). \r\n\r\nHowever the replay then falls over later on with a different mismatch:\r\n```\r\n[FATAL /home/petmay01/src/rr/src/AddressSpace.cc:947:assert_segments_match() errno: 0 'Success'] \r\n (task 21845 (rec:17396) at time 2844)\r\n ->  Assertion `same_mapping' failed to hold. \r\nCached mapping 0x55555600c000-0x55555668e000 0x3 f:0x2 should be 0x55555600c000-0x55555668d000 0x3 f:0x2\r\n```\r\nA quick look at the log suggests this is mishandling of brk() moving the program break down in memory: previously we did\r\n```\r\n[AddressSpace] brk(0x55555668e000)\r\n```\r\nand passed the consistency checks with\r\n```\r\n[AddressSpace] examining /proc/maps segment 0x55555600c000-0x55555668e000 0x3 f:0x2\r\n```\r\nbut then\r\n```\r\n[AddressSpace] brk(0x55555668d000)\r\n[AddressSpace] mmap(0x5555564a2000, 2011136, 0x3, 0x22, 0\r\n[AddressSpace] munmap(0x5555564a2000, 2011136)\r\n[AddressSpace]   unmapping (0x5555564a2000-0x55555668d000 0 f:0) ...\r\n[AddressSpace]   erased (0x5555564a2000-0x55555668e000 0x3 f:0x22) ...\r\n[AddressSpace]   mapping 0x5555564a2000-0x55555668d000 0x3 f:0x22\r\n[AddressSpace]     (flags or prot differ)\r\n[AddressSpace]     adjacent!\r\n[AddressSpace]     (not adjacent in memory)\r\n[AddressSpace]   coalescing 0x5555564a2000-0x55555668e000 0x3 f:0x22\r\n```\r\nand then we fail the consistency check because the kernel has truncated the mapping to the new program break but our cached version is still at the old location.\r\n"}}