{"action":"created","issue":{"url":"https://api.github.com/repos/SnabbCo/snabbswitch/issues/673","labels_url":"https://api.github.com/repos/SnabbCo/snabbswitch/issues/673/labels{/name}","comments_url":"https://api.github.com/repos/SnabbCo/snabbswitch/issues/673/comments","events_url":"https://api.github.com/repos/SnabbCo/snabbswitch/issues/673/events","html_url":"https://github.com/SnabbCo/snabbswitch/issues/673","id":120548235,"number":673,"title":"Max PPS ramp up time depends on lookup table size","user":{"login":"mwiget","id":3111088,"avatar_url":"https://avatars.githubusercontent.com/u/3111088?v=3","gravatar_id":"","url":"https://api.github.com/users/mwiget","html_url":"https://github.com/mwiget","followers_url":"https://api.github.com/users/mwiget/followers","following_url":"https://api.github.com/users/mwiget/following{/other_user}","gists_url":"https://api.github.com/users/mwiget/gists{/gist_id}","starred_url":"https://api.github.com/users/mwiget/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mwiget/subscriptions","organizations_url":"https://api.github.com/users/mwiget/orgs","repos_url":"https://api.github.com/users/mwiget/repos","events_url":"https://api.github.com/users/mwiget/events{/privacy}","received_events_url":"https://api.github.com/users/mwiget/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-12-05T11:23:22Z","updated_at":"2015-12-10T06:21:37Z","closed_at":"2015-12-09T09:08:34Z","body":"Discovered an interesting behavior in snabb when using large per-packet lookup table: The larger the hash table, the longer it takes to reach maximum packets per second (PPS). I've used binding table size of 0 (bypassing the lookup), 50k, 100k and 200k. It takes up to 13 seconds on an Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz to reach the maximum PPS:\r\n\r\n![rampup-50k-packets](https://cloud.githubusercontent.com/assets/3111088/11607359/ac8eebb8-9b43-11e5-9354-b6665b5d29d8.png)\r\n\r\nFor my lightweight 4 over 6 AFTR prototype (https://github.com/mwiget/snabbswitch/tree/lwaftr/), I need to lookup the binding for each packet. The look up key for the encapsulation path (IPv4 packets) is the IPv4 destination address and TCP/UDP (or ICMP Id) 16 Bit mask. The key size is 32+16=48 Bits (fits in a Lua number) and the value is an IPv6 address. Initially I created another lookup table for the encapsulation path with the IPv6 address as key, but turns out there is an easier way: Packets from the lightweight B4 client contain the IPv4 packet within an IPv6 header, so its possible to do a look up based on the IPv4 header information and compare the IPv6 packet source address with the hash result.\r\n\r\nThe test result was generated with a little snabb program lwaftrbench: https://github.com/mwiget/snabbswitch/tree/lwaftr/src/program/lwaftrbench, which basically sends the content of two PCAP files in each direction thru the lwaftr app and reports statistics every second:\r\n\r\n```\r\nipv6_pcap_file ---> lwaftr ---> Statistics\r\n    Statistics <--- lwaftr <--- ipv4_pcap_file\r\n```\r\n\r\nThe slow ramp up effect can be seen when using an IPv4 PCAP file with 50k unique packets (hitting 50,000 unique binding table entries):\r\n\r\n```\r\ncd src/program/lwaftrbench/tests\r\n$ grep CPU /proc/cpuinfo |head -1\r\nmodel name\t: Intel(R) Xeon(R) CPU D-1540 @ 2.00GHz\r\n../../../snabb lwaftrbench -D 30 snabb-port-50k.cfg b4-icmp-request-0138.pcap lwaftr-ipv4-50k.pcap\r\nConfiguration file: snabb-port-50k.cfg\r\nrunning for 30 seconds\r\noffset=0 len=6 -> psmask=0xfc00\r\n50000 bindings parsed, using 1778 kBytes\r\nipv4_to_ipv6: 1496 bpp 0.013 MPPS, 0.152 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.013 MPPS, 0.010 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.012 MPPS, 0.142 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.012 MPPS, 0.009 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.009 MPPS, 0.113 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.009 MPPS, 0.007 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.010 MPPS, 0.125 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.010 MPPS, 0.008 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.607 MPPS, 7.266 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.607 MPPS, 0.476 Gbps.\r\nipv4_to_ipv6: 1496 bpp 1.232 MPPS, 14.748 Gbps.\r\nipv6_to_ipv4:   98 bpp 1.232 MPPS, 0.966 Gbps.\r\nipv4_to_ipv6: 1496 bpp 1.233 MPPS, 14.758 Gbps.\r\nipv6_to_ipv4:   98 bpp 1.233 MPPS, 0.967 Gbps.\r\nipv4_to_ipv6: 1496 bpp 1.234 MPPS, 14.764 Gbps.\r\nipv6_to_ipv4:   98 bpp 1.234 MPPS, 0.967 Gbps.\r\nipv4_to_ipv6: 1496 bpp 1.230 MPPS, 14.715 Gbps.\r\nipv6_to_ipv4:   98 bpp 1.230 MPPS, 0.964 Gbps.\r\n. . .\r\n```\r\n\r\nI'm curious to learn/find out what exactly happens here that will explain the jump in performance. My guess its memory caching by the CPU core. Running the same command on a server with no L3 cache  \r\nshows the same effect:\r\n\r\n```\r\n$ grep CPU /proc/cpuinfo |head -1\r\nmodel name\t: Intel(R) Atom(TM) CPU  C2750  @ 2.41GHz\r\n$ sudo ../../../snabb lwaftrbench -D 30 snabb-port-50k.cfg b4-icmp-request-0138.pcap lwaftr-ipv4-50k.pcap\r\nConfiguration file: snabb-port-50k.cfg\r\nrunning for 30 seconds\r\noffset=0 len=6 -> psmask=0xfc00\r\n50000 bindings parsed, using 1778 kBytes\r\nipv4_to_ipv6: 1496 bpp 0.005 MPPS, 0.058 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.005 MPPS, 0.004 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.005 MPPS, 0.056 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.005 MPPS, 0.004 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.002 MPPS, 0.018 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.002 MPPS, 0.001 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.004 MPPS, 0.052 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.004 MPPS, 0.003 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.002 MPPS, 0.024 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.002 MPPS, 0.002 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.001 MPPS, 0.015 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.001 MPPS, 0.001 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.005 MPPS, 0.055 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.005 MPPS, 0.004 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.001 MPPS, 0.018 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.001 MPPS, 0.001 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.003 MPPS, 0.040 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.003 MPPS, 0.003 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.002 MPPS, 0.027 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.002 MPPS, 0.002 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.001 MPPS, 0.017 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.001 MPPS, 0.001 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.004 MPPS, 0.051 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.004 MPPS, 0.003 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.002 MPPS, 0.021 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.002 MPPS, 0.001 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.001 MPPS, 0.015 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.001 MPPS, 0.001 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.004 MPPS, 0.052 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.004 MPPS, 0.003 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.002 MPPS, 0.021 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.002 MPPS, 0.001 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.262 MPPS, 3.130 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.262 MPPS, 0.205 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.424 MPPS, 5.077 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.424 MPPS, 0.333 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.425 MPPS, 5.082 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.425 MPPS, 0.333 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.424 MPPS, 5.073 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.424 MPPS, 0.332 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.424 MPPS, 5.076 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.424 MPPS, 0.332 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.424 MPPS, 5.077 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.424 MPPS, 0.333 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.424 MPPS, 5.075 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.424 MPPS, 0.332 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.424 MPPS, 5.075 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.424 MPPS, 0.332 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.425 MPPS, 5.081 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.425 MPPS, 0.333 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.424 MPPS, 5.073 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.424 MPPS, 0.332 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.425 MPPS, 5.081 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.425 MPPS, 0.333 Gbps.\r\nipv4_to_ipv6: 1496 bpp 0.424 MPPS, 5.076 Gbps.\r\nipv6_to_ipv4:   98 bpp 0.424 MPPS, 0.333 Gbps.\r\n```\r\n\r\nThe hash table key is built by the following function (https://github.com/mwiget/snabbswitch/blob/lwaftr/src/apps/lwaftr/lwaftr.lua)\r\n\r\n```\r\nlocal function ipv4idkey(ipv4, id, psmask)\r\n  local psid = bit.band(id,psmask)\r\n  local ipv4psid = bit.band(0xffffffffffffLL, ipv4)\r\n  local key = tonumber(bit.bor(bit.lshift(ipv4psid,16), psid))\r\n  return key\r\nend\r\n```\r\n\r\nand the IPv6 address is stored as cdata:\r\n\r\n```\r\nlocal in_addr6  = ffi.new(\"uint8_t[16]\")\r\nlocal result = C.inet_pton(AF_INET6, binding_ipv6_addr, in_addr6)\r\nlocal psidshifted = bit.lshift(psid,16 - psoffset - psid_len)\r\nlocal ipv4psid = ipv4idkey(ipv4, psidshifted, 0xffff)\r\nmap_ipv4psid_to_ipv6[ipv4psid] = in_addr6\r\n```\r\n\r\nIs this a result of L1/L2 cache? How can I find out? \r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/SnabbCo/snabbswitch/issues/comments/163513086","html_url":"https://github.com/SnabbCo/snabbswitch/issues/673#issuecomment-163513086","issue_url":"https://api.github.com/repos/SnabbCo/snabbswitch/issues/673","id":163513086,"user":{"login":"lukego","id":13791,"avatar_url":"https://avatars.githubusercontent.com/u/13791?v=3","gravatar_id":"","url":"https://api.github.com/users/lukego","html_url":"https://github.com/lukego","followers_url":"https://api.github.com/users/lukego/followers","following_url":"https://api.github.com/users/lukego/following{/other_user}","gists_url":"https://api.github.com/users/lukego/gists{/gist_id}","starred_url":"https://api.github.com/users/lukego/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lukego/subscriptions","organizations_url":"https://api.github.com/users/lukego/orgs","repos_url":"https://api.github.com/users/lukego/repos","events_url":"https://api.github.com/users/lukego/events{/privacy}","received_events_url":"https://api.github.com/users/lukego/received_events","type":"User","site_admin":false},"created_at":"2015-12-10T06:21:37Z","updated_at":"2015-12-10T06:21:37Z","body":":+1: "}}