{"action":"created","issue":{"url":"https://api.github.com/repos/GRIFFINCollaboration/GRSISort/issues/510","labels_url":"https://api.github.com/repos/GRIFFINCollaboration/GRSISort/issues/510/labels{/name}","comments_url":"https://api.github.com/repos/GRIFFINCollaboration/GRSISort/issues/510/comments","events_url":"https://api.github.com/repos/GRIFFINCollaboration/GRSISort/issues/510/events","html_url":"https://github.com/GRIFFINCollaboration/GRSISort/issues/510","id":108017575,"number":510,"title":"Sorting events without TTreeIndex","user":{"login":"VinzenzBildstein","id":7277216,"avatar_url":"https://avatars.githubusercontent.com/u/7277216?v=3","gravatar_id":"","url":"https://api.github.com/users/VinzenzBildstein","html_url":"https://github.com/VinzenzBildstein","followers_url":"https://api.github.com/users/VinzenzBildstein/followers","following_url":"https://api.github.com/users/VinzenzBildstein/following{/other_user}","gists_url":"https://api.github.com/users/VinzenzBildstein/gists{/gist_id}","starred_url":"https://api.github.com/users/VinzenzBildstein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VinzenzBildstein/subscriptions","organizations_url":"https://api.github.com/users/VinzenzBildstein/orgs","repos_url":"https://api.github.com/users/VinzenzBildstein/repos","events_url":"https://api.github.com/users/VinzenzBildstein/events{/privacy}","received_events_url":"https://api.github.com/users/VinzenzBildstein/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/GRIFFINCollaboration/GRSISort/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-09-23T22:09:36Z","updated_at":"2015-09-25T00:39:29Z","closed_at":null,"body":"As mentioned in #478, we need to come up with a way to sort our data as single hits, not fragments to correctly sort pile-up hits. This will only work once we also have good CFD information as that is what determines the timing of the hits within one fragment, but I thought I'd get started on getting our code one the way to handle this.\r\nThe way I did this is by adding a new TFragment constructor that copies a single hit from an existing TFragment and adds the total number of pile-ups, and the index of the hit as transient members. This new TFragment is then stored in an STL container that automatically sorts them. Due to the missing CFD information that sorting is still done by timestamp ATM, but this can be easily changed by changing the operator> and operator< of TFragment. Sorting on the timestamp also allowed an easier comparison of the performance of the new code vs. the old code.\r\nThe decision when to start taking fragments out of the STL container and putting them as \"built events (a vector of fragments)\" into the event queue was made based on there being at least TGRSIRunInfo::BufferSize() events in the container and that the time difference between the first and the last fragment is larger than TGRSIRunInfo::BuildWindow().\r\nTo start I sorted one single file (run4774 from the Aug2015 directory, a 60Co run) and check the griffin multiplicities that the old code produced and what the new code using a multiset produces for different buffer sizes:\r\nmult\told\t            multiset (100k)\tmultiset (1M)\t\tmultiset (10M)\t diff (1M - norm)\r\n1\t1858826     2700602\t\t1858831\t\t \t1858826\t\t +5\r\n2\t363882\t   84950 \t \t \t363881\t \t \t363882\t\t -1\r\n3\t84579\t   16512\t\t \t84578\t\t \t84579\t\t -1\r\n4\t17601\t   3452\t\t \t17601\t\t \t17601\t\t 0\r\n5\t3415\t   677\t\t \t3415\t\t \t 3415\t\t 0\r\nWith a buffer size of at least 1M all the higher multiplicities are the same for the old and new code. There is no difference at all between the two codes if the buffer size is larger than the number of entries in the tree (about 3M in this file). For a buffer size of 1M, there is one multiplicity 2 and one multiplicity 3 events missing, which isn't so bad.\r\n\r\nThere are three different STL containers I've tested, the multiset, the priority_queue based on a vector, and the priority_queue based on a deque. I've tested all three for a buffer size of 1M, first with the old makefile, and then with the -O3 flag set (only for the new code), here is the memory usage and the time it took run the code:\r\nold:                              5.3G - real\t1m40.478s user\t2m53.979s sys\t0m5.159s\r\nmultiset:                      4.8G - real\t1m15.952s user\t2m39.005s sys\t0m5.128s\r\npriority_queue/vector: 4.7G - real\t2m55.851s user\t4m24.032s sys\t0m3.787s\r\npriority_queue/deque: 4.5G - real\t3m22.177s user\t4m52.423s sys\t0m4.568s\r\nwith -O3:\r\nmultiset:                      5.2G - real\t0m44.113s user\t1m5.508s sys\t0m5.662s\r\npriority_queue/vector: 4.7G - real\t0m45.955s user\t1m22.365s sys\t0m5.660s\r\npriority_queue/deque: 4.8G - real\t0m44.882s user\t1m23.059s sys\t0m5.359s\r\n"},"comment":{"url":"https://api.github.com/repos/GRIFFINCollaboration/GRSISort/issues/comments/143090186","html_url":"https://github.com/GRIFFINCollaboration/GRSISort/issues/510#issuecomment-143090186","issue_url":"https://api.github.com/repos/GRIFFINCollaboration/GRSISort/issues/510","id":143090186,"user":{"login":"r3dunlop","id":5428160,"avatar_url":"https://avatars.githubusercontent.com/u/5428160?v=3","gravatar_id":"","url":"https://api.github.com/users/r3dunlop","html_url":"https://github.com/r3dunlop","followers_url":"https://api.github.com/users/r3dunlop/followers","following_url":"https://api.github.com/users/r3dunlop/following{/other_user}","gists_url":"https://api.github.com/users/r3dunlop/gists{/gist_id}","starred_url":"https://api.github.com/users/r3dunlop/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/r3dunlop/subscriptions","organizations_url":"https://api.github.com/users/r3dunlop/orgs","repos_url":"https://api.github.com/users/r3dunlop/repos","events_url":"https://api.github.com/users/r3dunlop/events{/privacy}","received_events_url":"https://api.github.com/users/r3dunlop/received_events","type":"User","site_admin":false},"created_at":"2015-09-25T00:39:29Z","updated_at":"2015-09-25T00:39:29Z","body":"Are the missing events due to clearing out the entire buffer before refilling? Is there any performance loss for shorter buffers?\r\n\r\n> On Sep 23, 2015, at 3:13 PM, Vinzenz Bildstein <notifications@github.com> wrote:\r\n> \r\n> There doesn't seem to be a big difference once we use the -O3 flag, keeping in mind that the memory usage was done by looking at the maximum value that appeared using \"top\". The multiset seems to have a slight advantage in the user time, but the real time is very similar for all three options.\r\n> \r\n> Does anyone have any strong preferences for one or the other, or other ideas how to do this better?\r\n> \r\n> —\r\n> Reply to this email directly or view it on GitHub.\r\n> \r\n"}}