{"action":"created","issue":{"url":"https://api.github.com/repos/project-rig/rig/issues/142","labels_url":"https://api.github.com/repos/project-rig/rig/issues/142/labels{/name}","comments_url":"https://api.github.com/repos/project-rig/rig/issues/142/comments","events_url":"https://api.github.com/repos/project-rig/rig/issues/142/events","html_url":"https://github.com/project-rig/rig/issues/142","id":76617007,"number":142,"title":"Multiple connections for boards with multiple ethernet connected chips","user":{"login":"mundya","id":3439837,"avatar_url":"https://avatars.githubusercontent.com/u/3439837?v=3","gravatar_id":"","url":"https://api.github.com/users/mundya","html_url":"https://github.com/mundya","followers_url":"https://api.github.com/users/mundya/followers","following_url":"https://api.github.com/users/mundya/following{/other_user}","gists_url":"https://api.github.com/users/mundya/gists{/gist_id}","starred_url":"https://api.github.com/users/mundya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mundya/subscriptions","organizations_url":"https://api.github.com/users/mundya/orgs","repos_url":"https://api.github.com/users/mundya/repos","events_url":"https://api.github.com/users/mundya/events{/privacy}","received_events_url":"https://api.github.com/users/mundya/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/project-rig/rig/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/project-rig/rig/milestones/1","html_url":"https://github.com/project-rig/rig/milestones/1.0","labels_url":"https://api.github.com/repos/project-rig/rig/milestones/1/labels","id":945495,"number":1,"title":"1.0","description":"A working release for use with Nengo/SpiNNaker.","creator":{"login":"mundya","id":3439837,"avatar_url":"https://avatars.githubusercontent.com/u/3439837?v=3","gravatar_id":"","url":"https://api.github.com/users/mundya","html_url":"https://github.com/mundya","followers_url":"https://api.github.com/users/mundya/followers","following_url":"https://api.github.com/users/mundya/following{/other_user}","gists_url":"https://api.github.com/users/mundya/gists{/gist_id}","starred_url":"https://api.github.com/users/mundya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mundya/subscriptions","organizations_url":"https://api.github.com/users/mundya/orgs","repos_url":"https://api.github.com/users/mundya/repos","events_url":"https://api.github.com/users/mundya/events{/privacy}","received_events_url":"https://api.github.com/users/mundya/received_events","type":"User","site_admin":false},"open_issues":7,"closed_issues":103,"state":"open","created_at":"2015-01-21T16:25:41Z","updated_at":"2015-07-17T11:00:15Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2015-05-15T06:30:24Z","updated_at":"2015-07-17T18:26:53Z","closed_at":null,"body":"Not immediately required (it would be nice to have for Canada, possibly) but definitely necessary for Spaun.\r\n\r\nAs far as I can tell, the IP address of a chip can be read from the `sv` struct (`ip_addr`) and is, I assume, formatted the same way as IP addresses for IP tags (i.e., `!4B`, bewaring that the default appears to be 255.255.0.0).\r\nThe real challenge is in determining the co-ordinates of likely chips, though I guess a safe approach would be to always jump 8* (north or east) from the last found ethernet chip and then read `eth_addr` to get the co-ordinates of the nearest ethernet connected chip.\r\n\r\nI initially thought that this process should be part of `get_machine`... but now I'm not so sure.\r\n\r\nOnce we have a map of co-ordinates to ethernet addresses it becomes a drop in replacement to `MachineController.get_connection`, though we should also expose the mapping so that other software (like Nengo) can transmit SDP to SpiNNaker efficiently.\r\n\r\n\\* or 7?\r\n\r\nAlso: \"bewaring\" is an awesome word."},"comment":{"url":"https://api.github.com/repos/project-rig/rig/issues/comments/122365888","html_url":"https://github.com/project-rig/rig/issues/142#issuecomment-122365888","issue_url":"https://api.github.com/repos/project-rig/rig/issues/142","id":122365888,"user":{"login":"mossblaser","id":193514,"avatar_url":"https://avatars.githubusercontent.com/u/193514?v=3","gravatar_id":"","url":"https://api.github.com/users/mossblaser","html_url":"https://github.com/mossblaser","followers_url":"https://api.github.com/users/mossblaser/followers","following_url":"https://api.github.com/users/mossblaser/following{/other_user}","gists_url":"https://api.github.com/users/mossblaser/gists{/gist_id}","starred_url":"https://api.github.com/users/mossblaser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mossblaser/subscriptions","organizations_url":"https://api.github.com/users/mossblaser/orgs","repos_url":"https://api.github.com/users/mossblaser/repos","events_url":"https://api.github.com/users/mossblaser/events{/privacy}","received_events_url":"https://api.github.com/users/mossblaser/received_events","type":"User","site_admin":false},"created_at":"2015-07-17T18:26:53Z","updated_at":"2015-07-17T18:26:53Z","body":"The following is an informal design document for multi-connection support. Feedback highly encouraged.\r\n\r\nHigh-Throughput IO Plans\r\n========================\r\n\r\nGetting data in and out of SpiNNaker machines with high throughput is very\r\nimportant for large multi-board systems with potentially around six gigabytes\r\nof data per board to be loaded and only a single fairly measly 100MBit/s\r\ninterface each. Rig's new SCP I/O implementation with windowing enables the\r\nbest possible throughput into a single Ethernet connection. In this PR I hope\r\nto extend this to allowing multiple connections to several Ethernet connected\r\nchips to be used. It is not anticipated that anything but bulk memory\r\nread/writes will be required to perform high throughput I/O since all other\r\noperations are either broadcast-like or diagnostic (and thus fast enough at the\r\nmoment).\r\n\r\nHow many connections?\r\n---------------------\r\n\r\nThough ideally Rig should be able to drive all the Ethernet connections to an\r\narbitrarily large machine at once, in practice this is simply infeasible for\r\nthe following reasons:\r\n\r\n* The Python SCP implementation is likely to become CPU bound once a sufficient\r\n  number of SCP connections are in use since individual SCP packets are very\r\n  small.\r\n* A single Ethernet connection is more-or-less going to offer 1 GBit/s of I/O\r\n  which is only enough to saturate 10 boards' Ethernet connections anyway\r\n  (though in practice board Ethernet bandwidth is severely constrained anyway).\r\n\r\nI'm not sure which of these two issues will strike first. Initially I'll hope\r\nfor the best and assume we get I/O bound before CPU bound.\r\n\r\nWhich connection should be used?\r\n--------------------------------\r\n\r\nI would suggest that loading is always performed via the Ethernet connection on\r\nthe same board as the target processor. This choice has the advantage of being\r\neasily resolved via a lookup table. When a connection is not present, two\r\noptions exist:\r\n\r\n1. Fall back on (0, 0)\r\n2. Fall back on the nearest working connection\r\n3. Fall back on a randomly chosen connection\r\n4. Fall back on a cyclically chosen connection\r\n\r\nOption 1 is very easy but may lead to significant load imbalance: in principle\r\nN dead Ethernet connections could mean N times longer load times.\r\n\r\nOption 2 in principle is ideal but in practice is expensive since it requires\r\na fancy spatial lookup function to discover. A lookup table could be used but\r\nthis seems wasteful...\r\n\r\nOption 3 and 4 potentially work reasonably well and probably differ mainly in\r\nexpense of computation. It probably isn't worth worrying about the difference\r\nin cost...\r\n\r\nOverall, option 3 is probably the easiest to implement and stands the best\r\nchance of working.\r\n\r\nHow should multiple connections be used?\r\n----------------------------------------\r\n\r\nIn principle it could be done with a single select statement and some clever\r\nlogic. Alternatively, we can run many instances of the current SCPConnection\r\nobjects in separate threads. If we're truly I/O bound these two approaches\r\ndiffer only in the relative costs of Python vs OS level dispatching\r\nperformance.\r\n\r\nIf this were C, the select option would be the way to go but since this is\r\nPython there's a decent chance the difference in performance could be\r\ncomparable. The threading option is also potentially much easier to implement\r\nso I'll be going with that. If the SCP sending code proves too CPU intensive\r\n(sigh...) multiprocessing may be an option if there turns out to be some\r\nmoderately efficient way to share memory.\r\n\r\nI propose having a thread pool with each thread allocated a particular SCP\r\nconnection. When more SCP connections are present than threads, the remaining\r\nconnections (and their associated blocks of data) should be queued and given to\r\nthreads which have exhausted the traffic associated with their connection.\r\n\r\nAs for how to select the number of threads; initially this can just be fixed\r\nbut eventually some kind of control system could be set up...\r\n\r\nHow should data be supplied?\r\n----------------------------\r\n\r\nTwo obvious possibilities exist:\r\n\r\n1. Data/free memory could be supplied in full up-front.\r\n2. Callbacks could be arranged which return/handle data as it is fetched.\r\n\r\nThe former is potentially the easiest to implement and has the cleanest API.\r\nUnfortunately it may require your PC to have vast amounts of memory for very\r\nlarge problems. Further, there is lots of potential for subtle bugs arising\r\nfrom accidentally using the same memory twice or accessing uninitialised memory\r\nbuffers.\r\n\r\nThe later is much more fiddly to implement but obviously is more scalable since\r\nclever users can process their data as it arrives. It would be possible to wrap\r\nthis approach to look like the first approach for ease of implementation. This\r\napproach may also be marginally more costly due to the added function call\r\noverhead.\r\n\r\nI think due to safety and scalability I'll regrettably have to go with the\r\nsecond option as first class citizen and the first as an alternative API.\r\n\r\nHow should errors be treated?\r\n-----------------------------\r\n\r\nFor a large portion of applications, a single error during read/write is\r\nnon-recoverable. This is a semi-easy case to deal with, made super-easy if any\r\ndata received so far can be safely ignored, e.g. if all that will happen is an\r\nerror message and an exit.\r\n\r\nSadly, for larger-scale applications, a partial failure may not be as bad as\r\ncomplete failure. For example, when reading back the results of an experiment,\r\nif a single core has died during the experiment, the results on other cores may\r\nstill be valid and worth downloading. In such a circumstance it is still\r\nimportant to be able to determine *what* failed afterwards.\r\n\r\nThe correct solution here is probably to have two alternative modes of\r\noperation.\r\n\r\nHow should the API look?\r\n------------------------\r\n\r\nFiles on Linux are famously poor at dealing with multiple asynchronous\r\nreads/writes. On platforms where this is supported it is largely via\r\nnon-orthodox API. However this is done, it is probably going to result in the\r\nFile-like API being extended beyond typical file-like behaviour.\r\n\r\nAsyncIO/Trollius could be used here however it has famously dreadful\r\nperformance and even more dreadful documentation. This type of abstraction also\r\nhas a tendency to leak out into applications where it is unwanted.\r\n\r\nAn idea:\r\n\r\n    # Thin-wrapper around the current interface; overrides mc.read and mc.write\r\n    # such that they queue up reads/writes. For reads they also return a new\r\n    # buffer with garbage in it.\r\n    mc.begin_burst()\r\n    \r\n    mc.write(...)\r\n    data0 = mc.read(...)\r\n    \r\n    file_like.write(...)\r\n    file_like.write(...)\r\n    file_like.flush()\r\n    \r\n    data1 = file_like.read()\r\n    \r\n    # All the above actions actually take place now and in parallel. If\r\n    # something goes wrong, you get an exception and the above actions\r\n    # may/may-not have been called.\r\n    mc.end_burst()\r\n\r\nThe above works nicely for upgrading existing applications where you can afford\r\nthe memory overhead of having everything in RAM at once. To extend the APIs\r\nsomewhat we could make:\r\n\r\n* write() accept a callable and length in place of some bytes.\r\n* read() accept a callable as an extra argument and not return\r\n\r\nWith this modified API, callback style usage could be achieved at the expense\r\nof the file-like API being no-longer file-like. To properly support this, the\r\nwrite buffering/coalescing behaviour of MemoryIO will need modifying to support\r\ncoalescing callbacks. Outside burst-mode, the callbacks will simply get called\r\nimmediately.\r\n\r\nBy default, if an error occurs (including in a callback function) the end_burst\r\nmethod will simply propagate that error stopping all other activity as cleanly\r\nand quickly as possible. Some callbacks may not be called and some read data\r\nmay not have been placed in the reserved buffers (leaving their contents\r\nundefined).\r\n\r\nAlternatively, end_burst could accumulate exceptions and tracebacks, continuing\r\nas far as possible, and return a list of (callback, exception, traceback)\r\ntuples. Obviously, the callback object won't make sense unless you're using\r\nyour own callback system but given that this is a sufficiently advanced mode of\r\noperation, this is probably what should be used anyway.\r\n\r\nWhat will happen to the internals?\r\n----------------------------------\r\n\r\n0. The begin_burst function will simply set a flag.\r\n1. A set of read_into() methods will need to be added to allow creation of the\r\n   read buffer in advance.\r\n2. For each connection there will be a queue of read/write requests.\r\n3. All read/write requests should be wrapped in a lambda(?) and placed in the\r\n   buffer. For conventional reads, a buffer should be created and then\r\n   read_into called with that buffer.\r\n4. When not in a burst, the buffer will be emptied immediately (or a fast path\r\n   used...).\r\n5. The end_burst function will spawn some threads and in each thread work on\r\n   processing the queues.\r\n"}}