{"action":"closed","issue":{"url":"https://api.github.com/repos/openvswitch/ovs-issues/issues/90","labels_url":"https://api.github.com/repos/openvswitch/ovs-issues/issues/90/labels{/name}","comments_url":"https://api.github.com/repos/openvswitch/ovs-issues/issues/90/comments","events_url":"https://api.github.com/repos/openvswitch/ovs-issues/issues/90/events","html_url":"https://github.com/openvswitch/ovs-issues/issues/90","id":92410053,"number":90,"title":"datapath-windows: Memory leak in OVS datapath","user":{"login":"svinturis","id":8489740,"avatar_url":"https://avatars.githubusercontent.com/u/8489740?v=3","gravatar_id":"","url":"https://api.github.com/users/svinturis","html_url":"https://github.com/svinturis","followers_url":"https://api.github.com/users/svinturis/followers","following_url":"https://api.github.com/users/svinturis/following{/other_user}","gists_url":"https://api.github.com/users/svinturis/gists{/gist_id}","starred_url":"https://api.github.com/users/svinturis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/svinturis/subscriptions","organizations_url":"https://api.github.com/users/svinturis/orgs","repos_url":"https://api.github.com/users/svinturis/repos","events_url":"https://api.github.com/users/svinturis/events{/privacy}","received_events_url":"https://api.github.com/users/svinturis/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2015-07-01T15:12:40Z","updated_at":"2015-07-03T07:38:04Z","closed_at":"2015-07-03T07:38:04Z","body":"Detected after enabling Driver Verifier on OVS.\r\n\r\nBelow is the stacktrace:\r\n\r\n    *******************************************************************************\r\n    *                                                                             *\r\n    *                        Bugcheck Analysis                                    *\r\n    *                                                                             *\r\n    *******************************************************************************\r\n    \r\n    DRIVER_VERIFIER_DETECTED_VIOLATION (c4)\r\n    A device driver attempting to corrupt the system has been caught.  This is\r\n    because the driver was specified in the registry as being suspect (by the\r\n    administrator) and the kernel has enabled substantial checking of this driver.\r\n    If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA will\r\n    be among the most commonly seen crashes.\r\n    Arguments:\r\n    Arg1: 0000000000000062, A driver has forgotten to free its pool allocations prior to unloading.\r\n    Arg2: ffffe0000382f2e8, name of the driver having the issue.\r\n    Arg3: ffffe00003693990, verifier internal structure with driver information.\r\n    Arg4: 0000000000000001, total # of (paged+nonpaged) allocations that weren't freed.\r\n    \tType !verifier 3 drivername.sys for info on the allocations\r\n    \tthat were leaked that caused the bugcheck.\r\n    \r\n    Debugging Details:\r\n    ------------------\r\n    \r\n    \r\n    BUGCHECK_STR:  0xc4_62\r\n    \r\n    IMAGE_NAME:  OVSExt.sys\r\n    \r\n    DEBUG_FLR_IMAGE_TIMESTAMP:  5593e682\r\n    \r\n    MODULE_NAME: OVSExt\r\n    \r\n    FAULTING_MODULE: fffff800023aa000 OVSExt\r\n    \r\n    VERIFIER_DRIVER_ENTRY: dt nt!_MI_VERIFIER_DRIVER_ENTRY ffffe00003693990\r\n    Symbol nt!_MI_VERIFIER_DRIVER_ENTRY not found.\r\n    \r\n    DEFAULT_BUCKET_ID:  WIN8_DRIVER_FAULT\r\n    \r\n    PROCESS_NAME:  services.exe\r\n    \r\n    CURRENT_IRQL:  2\r\n    \r\n    ANALYSIS_VERSION: 6.3.9600.17237 (debuggers(dbg).140716-0327) amd64fre\r\n    \r\n    LAST_CONTROL_TRANSFER:  from fffff80360deb7c6 to fffff80360d68c90\r\n    \r\n    STACK_TEXT:  \r\n    ffffd000`2489db18 fffff803`60deb7c6 : 00000000`00000000 00000000`00000000 ffffd000`2489dc80 fffff803`60d10654 : nt!DbgBreakPointWithStatus\r\n    ffffd000`2489db20 fffff803`60deb0d7 : 00000000`00000003 00000000`00000062 fffff803`60d70070 00000000`000000c4 : nt!KiBugCheckDebugBreak+0x12\r\n    ffffd000`2489db80 fffff803`60d621a4 : 00000000`000000c4 fffff803`60cc6f6c ffffe000`00053630 fffff803`60cc9b73 : nt!KeBugCheck2+0x8ab\r\n    ffffd000`2489e290 fffff803`612816a8 : 00000000`000000c4 00000000`00000062 ffffe000`0382f2e8 ffffe000`03693990 : nt!KeBugCheckEx+0x104\r\n    ffffd000`2489e2d0 fffff803`61285c4a : fffff800`023aa000 ffffe000`0382f230 00000000`00000000 00000000`ffffffff : nt!VerifierBugCheckIfAppropriate+0x3c\r\n    ffffd000`2489e310 fffff803`60dbe2e8 : 00000000`00000000 00000000`00001000 0000007f`00000001 ffffe000`0323a1d0 : nt!VfPoolCheckForLeaks+0x4a\r\n    ffffd000`2489e350 fffff803`61273060 : fffff803`60ebecc0 fffff803`60ebecc0 ffffe000`0382f230 00000000`00000001 : nt! ?? ::FNODOBFM::`string'+0x4bd58\r\n    ffffd000`2489e3e0 fffff803`610e66c6 : 00000000`00000000 ffffe000`0382f230 00000000`00000000 ffffe000`01a70c60 : nt!VfDriverUnloadImage+0x34\r\n    ffffd000`2489e410 fffff803`610e6630 : 00000000`00000000 ffffe000`0382f230 ffffe000`01a70c60 ffffe000`01a70c30 : nt!MiUnloadSystemImage+0x7e\r\n    ffffd000`2489e490 fffff803`610e6578 : 00000000`00000000 ffffe000`001b5080 ffffe000`01a70c60 00000000`00010286 : nt!MmUnloadSystemImage+0x20\r\n    ffffd000`2489e4c0 fffff803`60fc3f28 : 00000000`00000000 ffffe000`01a70c60 ffffe000`001b5080 ffffc000`00086448 : nt!IopDeleteDriver+0x40\r\n    ffffd000`2489e500 fffff803`60c7a05f : 00000000`00000000 ffffd000`2489e800 ffffe000`01a70c60 00000000`c0000001 : nt!ObpRemoveObjectRoutine+0x64\r\n    ffffd000`2489e560 fffff803`610eaa2e : ffffe000`01a70c60 ffffd000`2489e9a0 00000000`c0000001 00000000`c0000001 : nt!ObfDereferenceObject+0x8f\r\n    ffffd000`2489e5a0 fffff803`60d6d8b3 : 00000000`00000000 ffffe000`07be5880 000000de`89eaea90 00000000`00000000 : nt!IopUnloadDriver+0x262\r\n    ffffd000`2489e780 fffff803`60d65d00 : fffff803`611aee61 ffffe000`07be5880 ffffd000`2489eb80 000000de`89eaea90 : nt!KiSystemServiceCopyEnd+0x13\r\n    ffffd000`2489e918 fffff803`611aee61 : ffffe000`07be5880 ffffd000`2489eb80 000000de`89eaea90 00000000`000001c8 : nt!KiServiceLinkage\r\n    ffffd000`2489e920 fffff803`60d6d8b3 : ffffe000`07be5880 ffffe000`07be5880 000000de`89978420 00000000`00000000 : nt! ?? ::NNGAKEGL::`string'+0x70261\r\n    ffffd000`2489eb00 00007ffd`997994fa : 00007ff6`b4d11c90 00000000`00000078 00000000`00000078 000000de`89978420 : nt!KiSystemServiceCopyEnd+0x13\r\n    000000de`89eaea68 00007ff6`b4d11c90 : 00000000`00000078 00000000`00000078 000000de`89978420 00007ff6`b4ce03f4 : ntdll!NtUnloadDriver+0xa\r\n    000000de`89eaea70 00007ff6`b4d11b68 : 00000000`0000000a 00000000`00000000 00007ff6`b4d23368 000000de`89eaebe0 : services!ScUnloadDriver+0xb4\r\n    000000de`89eaeab0 00007ff6`b4cf85d6 : 00000000`00000000 00000000`00000000 000000de`899fa3c0 367abb81`0000000d : services!ScControlDriver+0x110\r\n    000000de`89eaeae0 00007ff6`b4cd65b8 : 00000000`00000001 00000000`00000001 00007ff6`b4cdf0fc 00007ffd`00000000 : services!RI_ScGetCurrentGroupStateW+0x5272\r\n    000000de`89eaeba0 00007ffd`98da2385 : 00000000`00000000 000000de`89eaf0c8 000000de`89eaf210 00007ffd`98da2346 : services!RControlService+0x50\r\n    000000de`89eaec20 00007ffd`98daae16 : 000000de`89eaf0b0 00007ff6`b4cdf0f2 000000de`8a6e4430 00000000`00000001 : RPCRT4!Invoke+0x65\r\n    000000de`89eaec80 00007ffd`98da34ea : 00000000`00000019 000000de`899e32b0 00007ff6`b4ce1600 000000de`ffffffe5 : RPCRT4!NdrStubCall2+0x38b\r\n    000000de`89eaf300 00007ffd`98da2614 : 00007ff6`00000000 00000000`00000001 00000000`00000060 00007ffd`00000017 : RPCRT4!NdrServerCall2+0x1a\r\n    000000de`89eaf330 00007ffd`98da2517 : 00000000`00000000 000000de`89eaf421 000000de`89eaf4e8 000000de`89eaf3c0 : RPCRT4!DispatchToStubInCNoAvrf+0x14\r\n    000000de`89eaf380 00007ffd`98da30ad : 000000de`8994e800 00000000`00000000 00000000`00000000 000000de`8a6e42e0 : RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x177\r\n    000000de`89eaf480 00007ffd`98da2cc1 : 00000000`0006d39c 000000de`8a6e42e0 00000000`00000000 000000de`8994e800 : RPCRT4!LRPC_SCALL::DispatchRequest+0x1bd\r\n    000000de`89eaf580 00007ffd`98da2a97 : 00000011`00000000 00000000`00000003 000000de`00000000 00000000`00000000 : RPCRT4!LRPC_SCALL::HandleRequest+0x201\r\n    000000de`89eaf630 00007ffd`98da1d04 : 000000de`8994c540 000000de`00000000 00007ffd`98df9b24 000000de`8994c540 : RPCRT4!LRPC_SASSOCIATION::HandleRequest+0x237\r\n    000000de`89eaf6c0 00007ffd`98da1afe : 00000000`00000000 00000000`00000000 00000000`7ffe03c0 00007ffd`98df9b24 : RPCRT4!LRPC_ADDRESS::ProcessIO+0x36d\r\n    000000de`89eaf800 00007ffd`99724624 : 00000000`00000070 ffffffff`ffffffff 00000000`7ffe03b0 000000de`89eaf8b8 : RPCRT4!LrpcIoComplete+0xae\r\n    000000de`89eaf8a0 00007ffd`99722bfd : 00000000`00000000 00000000`00000000 00000000`00000000 000000de`8997c4e0 : ntdll!TppAlpcpExecuteCallback+0x204\r\n    000000de`89eaf910 00007ffd`98a31611 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x3ad\r\n    000000de`89eafd00 00007ffd`997764ad : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0xd\r\n    000000de`89eafd30 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d\r\n    \r\n    \r\n    STACK_COMMAND:  kb\r\n    \r\n    FOLLOWUP_NAME:  MachineOwner\r\n    \r\n    FAILURE_BUCKET_ID:  0xc4_62_VRF_LEAKED_POOL_IMAGE_OVSExt.sys\r\n    \r\n    BUCKET_ID:  0xc4_62_VRF_LEAKED_POOL_IMAGE_OVSExt.sys\r\n    \r\n    ANALYSIS_SOURCE:  KM\r\n    \r\n    FAILURE_ID_HASH_STRING:  km:0xc4_62_vrf_leaked_pool_image_ovsext.sys\r\n    \r\n    FAILURE_ID_HASH:  {4bcf06aa-3750-7f0f-72b4-698f3e1db606}\r\n    \r\n    Followup: MachineOwner\r\n    ---------\r\n    \r\n    3: kd> !verifier 3 ovsext.sys\r\n    \r\n    Verify Flags Level 0x01aeefbf\r\n    \r\n      STANDARD FLAGS:\r\n        [X] (0x00000000) Automatic Checks\r\n        [X] (0x00000001) Special pool\r\n        [X] (0x00000002) Force IRQL checking\r\n        [X] (0x00000008) Pool tracking\r\n        [X] (0x00000010) I/O verification\r\n        [X] (0x00000020) Deadlock detection\r\n        [X] (0x00000080) DMA checking\r\n        [X] (0x00000100) Security checks\r\n        [X] (0x00000800) Miscellaneous checks\r\n        [X] (0x00020000) DDI compliance checking\r\n    \r\n      ADDITIONAL FLAGS:\r\n        [X] (0x00000004) Randomized low resources simulation\r\n        [X] (0x00000200) Force pending I/O requests\r\n        [X] (0x00000400) IRP logging\r\n        [X] (0x00002000) Invariant MDL checking for stack\r\n        [X] (0x00004000) Invariant MDL checking for driver\r\n        [X] (0x00008000) Power framework delay fuzzing\r\n        [X] (0x00040000) Systematic low resources simulation\r\n        [X] (0x00080000) DDI compliance checking (additional)\r\n        [X] (0x00200000) NDIS/WIFI verification\r\n        [X] (0x00800000) Kernel synchronization delay fuzzing\r\n        [X] (0x01000000) VM switch verification\r\n    \r\n        [X] Indicates flag is enabled\r\n    \r\n    \r\n    Summary of All Verifier Statistics\r\n    \r\n      RaiseIrqls           0x151bd\r\n      AcquireSpinLocks     0x1f517\r\n      Synch Executions     0xa833\r\n      Trims                0x81d79\r\n    \r\n      Pool Allocations Attempted             0x28413d\r\n      Pool Allocations Succeeded             0x27f2d5\r\n      Pool Allocations Succeeded SpecialPool 0x27f2d5\r\n      Pool Allocations With NO TAG           0x0\r\n      Pool Allocations Failed                0x0\r\n    \r\n      Current paged pool allocations         0xa7 for 00002C5A bytes\r\n      Peak paged pool allocations            0xb3 for 000030FC bytes\r\n      Current nonpaged pool allocations      0x1220 for 00131E83 bytes\r\n      Peak nonpaged pool allocations         0x1442 for 001D4ADF bytes\r\n    \r\n    Driver Verification List\r\n    ------------------------\r\n    \r\n      MODULE: 0xffffe000000535e0 ovsext.sys (Loaded&Unloaded)\r\n    \r\n        Pool Allocation Statistics: ( NonPagedPool / PagedPool )\r\n    \r\n          Current Pool Allocations: ( 0x00000001 / 0x00000000 )\r\n          Current Pool Bytes:       ( 0x00000018 / 0x00000000 )\r\n          Peak Pool Allocations:    ( 0x00000211 / 0x00000000 )\r\n          Peak Pool Bytes:          ( 0x000a06c2 / 0x00000000 )\r\n          Contiguous Memory Bytes:       0x00000000\r\n          Peak Contiguous Memory Bytes:  0x00000000\r\n    \r\n        Pool Allocations:\r\n    \r\n          Address             Length      Tag   Caller Address    \r\n          ------------------  ----------  ----  ------------------\r\n          0xffffcf80036b4fe0  0x00000018  OVSD  0xfffff800023cbf11  OVSExt!OvsAllocateMemoryWithTag\r\n    \r\n        Contiguous allocations are not displayed with public symbols.\r\n    \r\n    3: kd> dt nt!_VF_SUSPECT_DRIVER_ENTRY ffffe000000535e0\r\n       +0x000 Links            : _LIST_ENTRY [ 0xffffe000`00053590 - 0xfffff803`60ebec60 ]\r\n       +0x010 Loads            : 2\r\n       +0x014 Unloads          : 1\r\n       +0x018 BaseName         : _UNICODE_STRING \"ovsext.sys\"\r\n    3: kd> ln 0xfffff800023cbf11\r\n    c:\\1.data\\cloudbase\\work\\git\\ovs\\datapath-windows\\ovsext\\util.c(33)\r\n    (fffff800`023cbea0)   OVSExt!OvsAllocateMemoryWithTag+0x71   |  (fffff800`023cbf20)   OVSExt!OvsAppendList\r\n    "}}