{"action":"created","comment":{"url":"https://api.github.com/repos/SnabbCo/snabbswitch/pulls/comments/23495725","id":23495725,"diff_hunk":"@@ -0,0 +1,267 @@\n+-- The shmem base class provides a simple IPC mechanism to exchange\n+-- arbitrary cdata objects with other processes through a file-backed\n+-- shared memeory region.\n+--\n+-- A new empty region is created by calling the constructor with a\n+-- filename and optional directory, where the file should be created.\n+--\n+--  local shmem = require(\"lib.ipc.shmem.shmem\")\n+--  local foo = shmem:new({ filename = \"foo\", directory = \"/tmp\" })\n+--\n+-- If omitted, the directory defaults to \"/tmp/snabb-shmem\".  The\n+-- constructor creates an empty file called the \"data file\" with the\n+-- given name and maps it into the processe's virtual memory.  In\n+-- addition, it creates an \"index file\" by appending the suffix\n+-- \".index\" to the data file name.  The first line of the index file\n+-- contains a string that identifies the name space to which the\n+-- memory region belongs, followed by a colon, followed by an integer\n+-- version number.  The name space indicates how the rest of the index\n+-- file needs to be interpreted.  A name space is tied to the subclass\n+-- that implements it (but a subclass may inherit the name space of\n+-- its ancestor class).  The version number allows for changes of the\n+-- index format within the name space.\n+--\n+-- The base class provides the name space \"default\", i.e. the header\n+-- line of the index file for version 1 contains the string \"default:1\".\n+--\n+-- The default name space contains a single line in the index file for\n+-- every object stored in the memory region.  The order of the\n+-- descriptions must be the same as that of the objects.  A\n+-- description consists of an arbitrary name, followd by a colon,\n+-- followed by the length of the corresponding object in bytes.  For\n+-- example, the index file\n+--\n+--   default:1\n+--   foo:4\n+--   bar:7\n+--\n+-- Describes a memory region of length 11, which contains an object\n+-- named 'foo' that consists of 4 bytes starting at offset 0 and an\n+-- object named 'bar' consisting of 7 bytes starting at offset 4.  The\n+-- type of the object is implied by its name and is not part of the\n+-- description in the index. Each name must be unique.\n+--\n+-- An object is added to the region by calling the method register(),\n+-- which takes a string, a ctype object and an optional value as\n+-- arguments.  The ctype must refer to a complete type such that the\n+-- size of the object is fixed when it is added to the index.  The\n+-- following example adds an unsigned 32-bit number called \"counter\"\n+-- and a struct named \"bar\":\n+--\n+--  local counter_t = ffi.typeof(\"uint32_t\")\n+--  local bar_t = ffi.typeof(\"struct { uint8_t x; char string[10]; }\")","path":"src/lib/ipc/shmem/shmem.lua","position":52,"original_position":52,"commit_id":"ffb1ef8c278534954f0c0bdbfc3f6fc0c4b640b0","original_commit_id":"ffb1ef8c278534954f0c0bdbfc3f6fc0c4b640b0","user":{"login":"alexandergall","id":5851328,"avatar_url":"https://avatars.githubusercontent.com/u/5851328?v=3","gravatar_id":"","url":"https://api.github.com/users/alexandergall","html_url":"https://github.com/alexandergall","followers_url":"https://api.github.com/users/alexandergall/followers","following_url":"https://api.github.com/users/alexandergall/following{/other_user}","gists_url":"https://api.github.com/users/alexandergall/gists{/gist_id}","starred_url":"https://api.github.com/users/alexandergall/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexandergall/subscriptions","organizations_url":"https://api.github.com/users/alexandergall/orgs","repos_url":"https://api.github.com/users/alexandergall/repos","events_url":"https://api.github.com/users/alexandergall/events{/privacy}","received_events_url":"https://api.github.com/users/alexandergall/received_events","type":"User","site_admin":false},"body":"On Thu, Jan 22, 2015 at 6:12 PM, Luke Gorrie <notifications@github.com> wrote:\r\n>\r\n> In src/lib/ipc/shmem/shmem.lua:\r\n>\r\n> > +--\r\n> > +-- Describes a memory region of length 11, which contains an object\r\n> > +-- named 'foo' that consists of 4 bytes starting at offset 0 and an\r\n> > +-- object named 'bar' consisting of 7 bytes starting at offset 4.  The\r\n> > +-- type of the object is implied by its name and is not part of the\r\n> > +-- description in the index. Each name must be unique.\r\n> > +--\r\n> > +-- An object is added to the region by calling the method register(),\r\n> > +-- which takes a string, a ctype object and an optional value as\r\n> > +-- arguments.  The ctype must refer to a complete type such that the\r\n> > +-- size of the object is fixed when it is added to the index.  The\r\n> > +-- following example adds an unsigned 32-bit number called \"counter\"\r\n> > +-- and a struct named \"bar\":\r\n> > +--\r\n> > +--  local counter_t = ffi.typeof(\"uint32_t\")\r\n> > +--  local bar_t = ffi.typeof(\"struct { uint8_t x; char string[10]; }\")\r\n>\r\n> I really like the generality of this design. Great that it is so easy to store structured data instead of single scalar values, and that the way this data is interpreted is flexible and open. (This data structure is starting to remind me of Forth's Dictionary.)\r\n\r\nReminds me of a Forth extension module I once had for my C64 :)  I did\r\nsome programming in Forth, but forgot most of it in the 30+ years\r\nsince :(\r\n\r\n>\r\n> How do ensure that readers see a consistent value? The counter module depends on x86_64 loading and storing 64-bit values atomically so that reads and writes won't conflict. Do we need some kind of protocol to extend this to structured data, in case a write happens part-way through a read? (I wonder what would work if so.. a double buffer?)\r\n\r\nI thought about that. My initial idea is to use semaphores. This could\r\nbe a named semaphore or an unnamed one that is placed directly in the\r\nshared memory. I'm not sure about the performance implications, but\r\nimplementation would be simple, I guess.","created_at":"2015-01-24T10:52:49Z","updated_at":"2015-01-24T10:52:49Z","html_url":"https://github.com/SnabbCo/snabbswitch/pull/348#discussion_r23495725","pull_request_url":"https://api.github.com/repos/SnabbCo/snabbswitch/pulls/348","_links":{"self":{"href":"https://api.github.com/repos/SnabbCo/snabbswitch/pulls/comments/23495725"},"html":{"href":"https://github.com/SnabbCo/snabbswitch/pull/348#discussion_r23495725"},"pull_request":{"href":"https://api.github.com/repos/SnabbCo/snabbswitch/pulls/348"}}},"pull_request":{"url":"https://api.github.com/repos/SnabbCo/snabbswitch/pulls/348","id":27835402,"html_url":"https://github.com/SnabbCo/snabbswitch/pull/348","diff_url":"https://github.com/SnabbCo/snabbswitch/pull/348.diff","patch_url":"https://github.com/SnabbCo/snabbswitch/pull/348.patch","issue_url":"https://api.github.com/repos/SnabbCo/snabbswitch/issues/348","number":348,"state":"open","locked":false,"title":"IPC method using file-backed shared memory","user":{"login":"alexandergall","id":5851328,"avatar_url":"https://avatars.githubusercontent.com/u/5851328?v=3","gravatar_id":"","url":"https://api.github.com/users/alexandergall","html_url":"https://github.com/alexandergall","followers_url":"https://api.github.com/users/alexandergall/followers","following_url":"https://api.github.com/users/alexandergall/following{/other_user}","gists_url":"https://api.github.com/users/alexandergall/gists{/gist_id}","starred_url":"https://api.github.com/users/alexandergall/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexandergall/subscriptions","organizations_url":"https://api.github.com/users/alexandergall/orgs","repos_url":"https://api.github.com/users/alexandergall/repos","events_url":"https://api.github.com/users/alexandergall/events{/privacy}","received_events_url":"https://api.github.com/users/alexandergall/received_events","type":"User","site_admin":false},"body":"I'd like to pick up on a suggestion for sharing counters with other processes by Luke in the thread \r\nhttps://groups.google.com/d/msg/snabb-devel/-Ka_gPfd88k/tL0wTqxcizcJ\r\n\r\nI commented back then that I was thinking about something like that to implement an interface to a SNMP agent. I have done this by generalzing Luke's code from https://github.com/lukego/snabbswitch/blob/counter/src/core/counter.lua\r\n\r\nI think that this (or something similar) might be worth adding to the Snabb core.\r\n\r\nThe \"mib\" subclass supports all basic SMIv2 types.  And, yes, I have working code for my VPN app and a SNMP sub-agent (written in Perl) that allows me to monitor my system with our standard monitoring system via SNMP (based on Nagios). I'm really pleased with that :)\r\n\r\nI have also implemented Luke's \"counter\" module on top of it, which is a trivial extension of the shmem base class. The difference to his code, apart from using uint64 instead of double (open for discussion), is that he pre-allocates a memory region of a certain size and falls back to heap memory when it is exhausted, while my code starts with a zero-length mapping and extends it when new objects are registered. It will fail hard (assert)  when the underlying mmap() fails or there is a problem with the data or index file. Registering is expected to not be used in the fast path.\r\n\r\nI suggest to put this under lib/ipc/shmem in anticipation of other IPC mechanisms that we may add in the future. But I don't have a strong opinion about where to put it.\r\n","created_at":"2015-01-22T11:29:01Z","updated_at":"2015-01-24T10:52:49Z","closed_at":null,"merged_at":null,"merge_commit_sha":"464e25888dce43eccf60de727dada6d533442e03","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/SnabbCo/snabbswitch/pulls/348/commits","review_comments_url":"https://api.github.com/repos/SnabbCo/snabbswitch/pulls/348/comments","review_comment_url":"https://api.github.com/repos/SnabbCo/snabbswitch/pulls/comments/{number}","comments_url":"https://api.github.com/repos/SnabbCo/snabbswitch/issues/348/comments","statuses_url":"https://api.github.com/repos/SnabbCo/snabbswitch/statuses/ffb1ef8c278534954f0c0bdbfc3f6fc0c4b640b0","head":{"label":"alexandergall:shmem","ref":"shmem","sha":"ffb1ef8c278534954f0c0bdbfc3f6fc0c4b640b0","user":{"login":"alexandergall","id":5851328,"avatar_url":"https://avatars.githubusercontent.com/u/5851328?v=3","gravatar_id":"","url":"https://api.github.com/users/alexandergall","html_url":"https://github.com/alexandergall","followers_url":"https://api.github.com/users/alexandergall/followers","following_url":"https://api.github.com/users/alexandergall/following{/other_user}","gists_url":"https://api.github.com/users/alexandergall/gists{/gist_id}","starred_url":"https://api.github.com/users/alexandergall/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexandergall/subscriptions","organizations_url":"https://api.github.com/users/alexandergall/orgs","repos_url":"https://api.github.com/users/alexandergall/repos","events_url":"https://api.github.com/users/alexandergall/events{/privacy}","received_events_url":"https://api.github.com/users/alexandergall/received_events","type":"User","site_admin":false},"repo":{"id":15471078,"name":"snabbswitch","full_name":"alexandergall/snabbswitch","owner":{"login":"alexandergall","id":5851328,"avatar_url":"https://avatars.githubusercontent.com/u/5851328?v=3","gravatar_id":"","url":"https://api.github.com/users/alexandergall","html_url":"https://github.com/alexandergall","followers_url":"https://api.github.com/users/alexandergall/followers","following_url":"https://api.github.com/users/alexandergall/following{/other_user}","gists_url":"https://api.github.com/users/alexandergall/gists{/gist_id}","starred_url":"https://api.github.com/users/alexandergall/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexandergall/subscriptions","organizations_url":"https://api.github.com/users/alexandergall/orgs","repos_url":"https://api.github.com/users/alexandergall/repos","events_url":"https://api.github.com/users/alexandergall/events{/privacy}","received_events_url":"https://api.github.com/users/alexandergall/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/alexandergall/snabbswitch","description":"The Snabb Switch Project","fork":true,"url":"https://api.github.com/repos/alexandergall/snabbswitch","forks_url":"https://api.github.com/repos/alexandergall/snabbswitch/forks","keys_url":"https://api.github.com/repos/alexandergall/snabbswitch/keys{/key_id}","collaborators_url":"https://api.github.com/repos/alexandergall/snabbswitch/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/alexandergall/snabbswitch/teams","hooks_url":"https://api.github.com/repos/alexandergall/snabbswitch/hooks","issue_events_url":"https://api.github.com/repos/alexandergall/snabbswitch/issues/events{/number}","events_url":"https://api.github.com/repos/alexandergall/snabbswitch/events","assignees_url":"https://api.github.com/repos/alexandergall/snabbswitch/assignees{/user}","branches_url":"https://api.github.com/repos/alexandergall/snabbswitch/branches{/branch}","tags_url":"https://api.github.com/repos/alexandergall/snabbswitch/tags","blobs_url":"https://api.github.com/repos/alexandergall/snabbswitch/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/alexandergall/snabbswitch/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/alexandergall/snabbswitch/git/refs{/sha}","trees_url":"https://api.github.com/repos/alexandergall/snabbswitch/git/trees{/sha}","statuses_url":"https://api.github.com/repos/alexandergall/snabbswitch/statuses/{sha}","languages_url":"https://api.github.com/repos/alexandergall/snabbswitch/languages","stargazers_url":"https://api.github.com/repos/alexandergall/snabbswitch/stargazers","contributors_url":"https://api.github.com/repos/alexandergall/snabbswitch/contributors","subscribers_url":"https://api.github.com/repos/alexandergall/snabbswitch/subscribers","subscription_url":"https://api.github.com/repos/alexandergall/snabbswitch/subscription","commits_url":"https://api.github.com/repos/alexandergall/snabbswitch/commits{/sha}","git_commits_url":"https://api.github.com/repos/alexandergall/snabbswitch/git/commits{/sha}","comments_url":"https://api.github.com/repos/alexandergall/snabbswitch/comments{/number}","issue_comment_url":"https://api.github.com/repos/alexandergall/snabbswitch/issues/comments/{number}","contents_url":"https://api.github.com/repos/alexandergall/snabbswitch/contents/{+path}","compare_url":"https://api.github.com/repos/alexandergall/snabbswitch/compare/{base}...{head}","merges_url":"https://api.github.com/repos/alexandergall/snabbswitch/merges","archive_url":"https://api.github.com/repos/alexandergall/snabbswitch/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/alexandergall/snabbswitch/downloads","issues_url":"https://api.github.com/repos/alexandergall/snabbswitch/issues{/number}","pulls_url":"https://api.github.com/repos/alexandergall/snabbswitch/pulls{/number}","milestones_url":"https://api.github.com/repos/alexandergall/snabbswitch/milestones{/number}","notifications_url":"https://api.github.com/repos/alexandergall/snabbswitch/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/alexandergall/snabbswitch/labels{/name}","releases_url":"https://api.github.com/repos/alexandergall/snabbswitch/releases{/id}","created_at":"2013-12-27T09:42:11Z","updated_at":"2015-01-21T16:36:35Z","pushed_at":"2015-01-22T11:10:48Z","git_url":"git://github.com/alexandergall/snabbswitch.git","ssh_url":"git@github.com:alexandergall/snabbswitch.git","clone_url":"https://github.com/alexandergall/snabbswitch.git","svn_url":"https://github.com/alexandergall/snabbswitch","homepage":"","size":1966,"stargazers_count":0,"watchers_count":0,"language":"Lua","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"SnabbCo:master","ref":"master","sha":"e5235a666b4591ff89abc3dce149681c3a884eac","user":{"login":"SnabbCo","id":1913310,"avatar_url":"https://avatars.githubusercontent.com/u/1913310?v=3","gravatar_id":"","url":"https://api.github.com/users/SnabbCo","html_url":"https://github.com/SnabbCo","followers_url":"https://api.github.com/users/SnabbCo/followers","following_url":"https://api.github.com/users/SnabbCo/following{/other_user}","gists_url":"https://api.github.com/users/SnabbCo/gists{/gist_id}","starred_url":"https://api.github.com/users/SnabbCo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SnabbCo/subscriptions","organizations_url":"https://api.github.com/users/SnabbCo/orgs","repos_url":"https://api.github.com/users/SnabbCo/repos","events_url":"https://api.github.com/users/SnabbCo/events{/privacy}","received_events_url":"https://api.github.com/users/SnabbCo/received_events","type":"Organization","site_admin":false},"repo":{"id":4857740,"name":"snabbswitch","full_name":"SnabbCo/snabbswitch","owner":{"login":"SnabbCo","id":1913310,"avatar_url":"https://avatars.githubusercontent.com/u/1913310?v=3","gravatar_id":"","url":"https://api.github.com/users/SnabbCo","html_url":"https://github.com/SnabbCo","followers_url":"https://api.github.com/users/SnabbCo/followers","following_url":"https://api.github.com/users/SnabbCo/following{/other_user}","gists_url":"https://api.github.com/users/SnabbCo/gists{/gist_id}","starred_url":"https://api.github.com/users/SnabbCo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SnabbCo/subscriptions","organizations_url":"https://api.github.com/users/SnabbCo/orgs","repos_url":"https://api.github.com/users/SnabbCo/repos","events_url":"https://api.github.com/users/SnabbCo/events{/privacy}","received_events_url":"https://api.github.com/users/SnabbCo/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/SnabbCo/snabbswitch","description":"Snabb Switch: Fast open source packet processing","fork":false,"url":"https://api.github.com/repos/SnabbCo/snabbswitch","forks_url":"https://api.github.com/repos/SnabbCo/snabbswitch/forks","keys_url":"https://api.github.com/repos/SnabbCo/snabbswitch/keys{/key_id}","collaborators_url":"https://api.github.com/repos/SnabbCo/snabbswitch/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/SnabbCo/snabbswitch/teams","hooks_url":"https://api.github.com/repos/SnabbCo/snabbswitch/hooks","issue_events_url":"https://api.github.com/repos/SnabbCo/snabbswitch/issues/events{/number}","events_url":"https://api.github.com/repos/SnabbCo/snabbswitch/events","assignees_url":"https://api.github.com/repos/SnabbCo/snabbswitch/assignees{/user}","branches_url":"https://api.github.com/repos/SnabbCo/snabbswitch/branches{/branch}","tags_url":"https://api.github.com/repos/SnabbCo/snabbswitch/tags","blobs_url":"https://api.github.com/repos/SnabbCo/snabbswitch/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/SnabbCo/snabbswitch/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/SnabbCo/snabbswitch/git/refs{/sha}","trees_url":"https://api.github.com/repos/SnabbCo/snabbswitch/git/trees{/sha}","statuses_url":"https://api.github.com/repos/SnabbCo/snabbswitch/statuses/{sha}","languages_url":"https://api.github.com/repos/SnabbCo/snabbswitch/languages","stargazers_url":"https://api.github.com/repos/SnabbCo/snabbswitch/stargazers","contributors_url":"https://api.github.com/repos/SnabbCo/snabbswitch/contributors","subscribers_url":"https://api.github.com/repos/SnabbCo/snabbswitch/subscribers","subscription_url":"https://api.github.com/repos/SnabbCo/snabbswitch/subscription","commits_url":"https://api.github.com/repos/SnabbCo/snabbswitch/commits{/sha}","git_commits_url":"https://api.github.com/repos/SnabbCo/snabbswitch/git/commits{/sha}","comments_url":"https://api.github.com/repos/SnabbCo/snabbswitch/comments{/number}","issue_comment_url":"https://api.github.com/repos/SnabbCo/snabbswitch/issues/comments/{number}","contents_url":"https://api.github.com/repos/SnabbCo/snabbswitch/contents/{+path}","compare_url":"https://api.github.com/repos/SnabbCo/snabbswitch/compare/{base}...{head}","merges_url":"https://api.github.com/repos/SnabbCo/snabbswitch/merges","archive_url":"https://api.github.com/repos/SnabbCo/snabbswitch/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/SnabbCo/snabbswitch/downloads","issues_url":"https://api.github.com/repos/SnabbCo/snabbswitch/issues{/number}","pulls_url":"https://api.github.com/repos/SnabbCo/snabbswitch/pulls{/number}","milestones_url":"https://api.github.com/repos/SnabbCo/snabbswitch/milestones{/number}","notifications_url":"https://api.github.com/repos/SnabbCo/snabbswitch/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/SnabbCo/snabbswitch/labels{/name}","releases_url":"https://api.github.com/repos/SnabbCo/snabbswitch/releases{/id}","created_at":"2012-07-02T09:06:38Z","updated_at":"2015-01-23T19:33:25Z","pushed_at":"2015-01-21T11:58:02Z","git_url":"git://github.com/SnabbCo/snabbswitch.git","ssh_url":"git@github.com:SnabbCo/snabbswitch.git","clone_url":"https://github.com/SnabbCo/snabbswitch.git","svn_url":"https://github.com/SnabbCo/snabbswitch","homepage":"","size":4703,"stargazers_count":917,"watchers_count":917,"language":"Lua","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":101,"mirror_url":null,"open_issues_count":49,"forks":101,"open_issues":49,"watchers":917,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/SnabbCo/snabbswitch/pulls/348"},"html":{"href":"https://github.com/SnabbCo/snabbswitch/pull/348"},"issue":{"href":"https://api.github.com/repos/SnabbCo/snabbswitch/issues/348"},"comments":{"href":"https://api.github.com/repos/SnabbCo/snabbswitch/issues/348/comments"},"review_comments":{"href":"https://api.github.com/repos/SnabbCo/snabbswitch/pulls/348/comments"},"review_comment":{"href":"https://api.github.com/repos/SnabbCo/snabbswitch/pulls/comments/{number}"},"commits":{"href":"https://api.github.com/repos/SnabbCo/snabbswitch/pulls/348/commits"},"statuses":{"href":"https://api.github.com/repos/SnabbCo/snabbswitch/statuses/ffb1ef8c278534954f0c0bdbfc3f6fc0c4b640b0"}}}}