{"action":"created","issue":{"url":"https://api.github.com/repos/antonbarty/cheetah/issues/20","labels_url":"https://api.github.com/repos/antonbarty/cheetah/issues/20/labels{/name}","comments_url":"https://api.github.com/repos/antonbarty/cheetah/issues/20/comments","events_url":"https://api.github.com/repos/antonbarty/cheetah/issues/20/events","html_url":"https://github.com/antonbarty/cheetah/issues/20","id":75815377,"number":20,"title":"Do not reassemble & downsample when not requested","user":{"login":"biochem-fan","id":1611483,"avatar_url":"https://avatars.githubusercontent.com/u/1611483?v=3","gravatar_id":"","url":"https://api.github.com/users/biochem-fan","html_url":"https://github.com/biochem-fan","followers_url":"https://api.github.com/users/biochem-fan/followers","following_url":"https://api.github.com/users/biochem-fan/following{/other_user}","gists_url":"https://api.github.com/users/biochem-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/biochem-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/biochem-fan/subscriptions","organizations_url":"https://api.github.com/users/biochem-fan/orgs","repos_url":"https://api.github.com/users/biochem-fan/repos","events_url":"https://api.github.com/users/biochem-fan/events{/privacy}","received_events_url":"https://api.github.com/users/biochem-fan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":7,"created_at":"2015-05-13T02:26:48Z","updated_at":"2015-05-13T16:20:08Z","closed_at":null,"body":"I noticed that downsample() does not check whether downsampling is really needed. This slows down Cheetah by more than 50%!\r\n\r\nIn event.cpp, buffers for assembled image, downsampled image and radial average are always allocated. This consumes lots of memory and very slow. I thought that calloc uses zero-page and copy-on-write tricks but it seems not. Commenting out callocs for assembled image, downsampled image and radial average, and replacing calloc with malloc for raw images, FPS improved another 40%.\r\n\r\nSo, it would be nice if these buffers are allocated only when requested. I will do it if you want. If so, does it suffice to look at saveAssembled, saveAssembledAndDownsampled, saveRadialAverage in cPixelDetectorCommon to judge if these buffers are necessary?"},"comment":{"url":"https://api.github.com/repos/antonbarty/cheetah/issues/comments/101733704","html_url":"https://github.com/antonbarty/cheetah/issues/20#issuecomment-101733704","issue_url":"https://api.github.com/repos/antonbarty/cheetah/issues/20","id":101733704,"user":{"login":"antonbarty","id":674762,"avatar_url":"https://avatars.githubusercontent.com/u/674762?v=3","gravatar_id":"","url":"https://api.github.com/users/antonbarty","html_url":"https://github.com/antonbarty","followers_url":"https://api.github.com/users/antonbarty/followers","following_url":"https://api.github.com/users/antonbarty/following{/other_user}","gists_url":"https://api.github.com/users/antonbarty/gists{/gist_id}","starred_url":"https://api.github.com/users/antonbarty/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antonbarty/subscriptions","organizations_url":"https://api.github.com/users/antonbarty/orgs","repos_url":"https://api.github.com/users/antonbarty/repos","events_url":"https://api.github.com/users/antonbarty/events{/privacy}","received_events_url":"https://api.github.com/users/antonbarty/received_events","type":"User","site_admin":false},"created_at":"2015-05-13T16:20:08Z","updated_at":"2015-05-13T16:20:08Z","body":"Hi Carl\n\nThis raises an interesting point about the size of the arrays needed:\n\nThe mpCCD at SACLA has many more pixels than the back CSPAD at SLAC, for example. \nResults may vary depending on the size of the images being loaded (and therefore the memory footprint for all the different arrays, thus also what fits into memory caches).\n\n\n\n\n> On 13 May 2015, at 18:07, cnettel <notifications@github.com> wrote:\n> \n> I did previously also explore the calloc/malloc issue in different environments, with up to 20 threads. I could not reproduce numbers like that, at that point – and actually the overall processing rate was higher. Of course, that was quite some time back, so maybe the total memory allocation load is even higher now. (Processing rates in some use cases for me were closer to 60-100 Hz…) \n> \n> What input form was used? One important observation I made was that in some setups, the total amount of I/O caching could be extreme. In one cluster setup, GBs of input (!) ended up being in caches at different levels. This meant that running a benchmark ones, making a fix, compiling and running again almost always gave a positive result, only due to the repeat effect of caching. Since caches can go stale at different points as well, getting a reproducible, while still representative, benchmark scenario can be a bit of a challenge. \n> \n> I naturally don’t know if this might apply to your scenario, it’s just a word of caution. \n> \n> I agree that in theory calloc should be slow, although for large allocations there are also cases where calloc can rely on kernel services to provide already-zeroed pages. This stackoverflow thread provides some discussion of the issue. http://stackoverflow.com/questions/2688466/why-mallocmemset-is-slower-than-calloc (that also means that there can be threshold effects where “small” unused calloc allocations are slow, while “large” ones are fast) \n> \n> /Carl \n> \n> Från: biochem_fan [mailto:notifications@github.com] \n> Skickat: den 13 maj 2015 17:11 \n> Till: antonbarty/cheetah \n> Ämne: Re: [cheetah] Do not reassemble & downsample when not requested (#20) \n> \n> \n> Hi everyone, \n> \n> Thank you very much for comments. \n> \n> How do you typically run cheetah (how many threads, what hardware, what OS)? What kind of frame rates do you get? \n> \n> 12 cores@3.4<mailto:cores@3.4> GHz on Linux. \n> Initially, 10~12Hz. By commenting out downsample(), I got 20-22 Hz. After I removed calloc-s, it rus at more than 30 Hz. I didn't change saveAssembledAndDownsampled. \n> \n> It is well known that malloc() in glibc is not very fast with multi-threading. Using jemalloc instead might solve the problem. \n> \n> I will investigate this further once our beam time is over. \n> \n> — \n> Reply to this email directly or view it on GitHub<https://github.com/antonbarty/cheetah/issues/20#issuecomment-101710931>. \n> —\n> Reply to this email directly or view it on GitHub <https://github.com/antonbarty/cheetah/issues/20#issuecomment-101726543>.\n> \n\n"}}