{"push_id":682242360,"size":37,"distinct_size":0,"ref":"refs/heads/2.12","head":"11769caf60b41af4106a88447312454673a5764b","before":"e9011d93b6a5780fac49f56172f125aac78308a2","commits":[{"sha":"7d53884a07dd91e90fb2f543d8c755c850afefc1","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"pool: Fix timeout behavior of HSM requests\n\nTimeouts in the pool were implemented from the request creation time rather\nthan the request activation time. This is a regression compared to 2.6 and\nleads to problems when requests are queued: Since reqeusts are executed in FIFO\norder, the requests that are activated are always the oldest requests.  Since\nthose older than the deadline have been removed, the oldest requests left are\noften close to the deadline and will be cancelled shortly after activation.\nThis leads to a lot of churn with no real progress.\n\nThe intended implementation of timouts for HSM requests was never finished: The\nintention was the whoever requests an HSM operation (in particular stage) would\nspecify a lifetime. E.g. SRM allows this. This would be the deadline and an\nadvanced provider could use this to schedule requests. Specific providers could\nimplement their own timeouts for particular operations, e.g.  callouts to\nscripts. This was the reason for why the deadline was measured from the\ncreation time. Currently however only the latter form of timeout is implemented\nand not within the provider but in the framework. Eventually this needs to be\ncleaned up. For the time being this patch changes the deadline to be calculated\nfrom the activation time.  Requests in the queue never time out.\n\nThis implementation is to be considered temporary (although without a limit on\nwhen it will be changed).\n\nTarget: trunk\nRequire-notes: yes\nRequest: 2.12\nRequest: 2.11\nRequest: 2.10\nTicket: http://rt.dcache.org/Ticket/Display.html?id=8620\nAcked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>\nPatch: https://rb.dcache.org/r/8224/\n(cherry picked from commit 1aa2b291865a682d53782a2b6f9cde637782a06d)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/7d53884a07dd91e90fb2f543d8c755c850afefc1"},{"sha":"a1a4d6db2e034f25c6686c6d83afcd625c98dcc7","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"srm: Fix queue size reporting\n\nWe logged the lenght of the wrong queue in the SRM info output in the\n\"Waiting for CPU\" row.\n\nTarget: trunk\nRequest: 2.12\nRequest: 2.11\nRequest: 2.10\nRequire-notes: yes\nRequire-book: no\nAcked-by: Paul Millar <paul.millar@desy.de>\nPatch: https://rb.dcache.org/r/8226/\n(cherry picked from commit d2803118e20d089aeb71c4cc2573d5580753bc91)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/a1a4d6db2e034f25c6686c6d83afcd625c98dcc7"},{"sha":"d0188a451b40857885d3520e20481f8f2bed8863","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"pool: Add HSM options to hsm script remove callout\n\nWe forgot to add the options specified with the 'hsm set' commands when doing\nthe remove callout using the script driver.  This is a regression compared to\nthe old HSM interface.\n\nTarget: trunk\nRequest: 2.12\nRequest: 2.11\nRequest: 2.10\nRequest: 2.9\nRequire-notes: yes\nRequire-book: no\nAcked-by: Paul Millar <paul.millar@desy.de>\n(cherry picked from commit a2aab5f567565e45930c0a64a1f3f6faa6595a0e)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/d0188a451b40857885d3520e20481f8f2bed8863"},{"sha":"a22278bc59e8c86c98c7d3131e39b3c0a763d463","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"pinmanager: Fix listing by id\n\nThe pin manager allows pins to be listed by pin ID or PNFS ID. Since any\nnon-negative long is also a valid PNFS ID, the PnfsId#isvalid method is not\nparticularly useful to distinguish the two cases. Thus listing by pin ID\ndoesn't actually work. This patch resolves this issue.\n\nTarget: trunk\nRequest: 2.12\nRequest: 2.11\nRequest: 2.10\nRequest: 2.9\nRequest: 2.8\nRequest: 2.7\nRequest: 2.6\nRequire-notes: yes\nRequire-book: no\nAcked-by: Paul Millar <paul.millar@desy.de>\nPatch: https://rb.dcache.org/r/8228/\n(cherry picked from commit 7451599bd2e5a46a7e9f7a08bf1e6c5b0cb65fb2)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/a22278bc59e8c86c98c7d3131e39b3c0a763d463"},{"sha":"200036718917ddfcaf088b16162a2748f7bd0bb8","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"srm: Fix sync to async mode timeout\n\nThe logic for switching from synchroneous to asynchroenous mode needs to both\ndetect a timeout and completion of the request. For large bulk requests the\nlogic was flawed as continuous state changes of file level requests would keep\nthe SRM looping and miss the timeout.\n\nTarget: trunk\nRequire-notes: yes\nRequire-book: yes\nRequest: 2.12\nRequest: 2.11\nRequest: 2.10\nAcked-by: Paul Millar <paul.millar@desy.de>\nPatch: https://rb.dcache.org/r/8229/\n(cherry picked from commit 61c2b658df36f7072e78f2e064fdddb71f878b64)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/200036718917ddfcaf088b16162a2748f7bd0bb8"},{"sha":"227ea6cee060213a211b8d293faf9f05a7f3bb40","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"system-test: Enable MVCC and logging for HSQLDB\n\nWithout MVCC, HSQLDB resorts to table level locks. With pin manager, these\nlocks lead to aborted transactions as soon as more than one request is\nprocessed at a time. The patch enables MVCC for all databases. This also\nmakes the behaviour of HSQLDB closer to that of PostgreSQL.\n\nThe patch also enables SQL statement logging in HSQLDB. This makes it\neasier to debug database related problems.\n\nTarget: trunk\nRequest: 2.12\nRequest: 2.11\nRequest: 2.10\nRequire-notes: no\nRequire-book: no\nAcked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>\nPatch: https://rb.dcache.org/r/8234/\n(cherry picked from commit bd8dd02fac1fe84f03bb7c2b8ea6c244eb1ab3df)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/227ea6cee060213a211b8d293faf9f05a7f3bb40"},{"sha":"19251f36d1089633f11624edeb99bec732b33cb8","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"srm: Abort pinning when cancelling bring-online requests\n\nThe SRM failed to abort pin operations when aborting bring-online requests.\nThis patch fixes this problem.\n\nTarget: trunk\nRequest: 2.12\nRequest: 2.11\nRequest: 2.10\nRequest: 2.9\nRequest: 2.8\nRequest: 2.7\nRequest: 2.6\nRequire-notes: yes\nRequire-book: no\nAcked-by: Dmitry Litvintsev <litvinse@fnal.gov>\nPatch: https://rb.dcache.org/r/8239/\n(cherry picked from commit 041fb5049a09dc140f4c9a8dc5931213c7f7d1cf)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/19251f36d1089633f11624edeb99bec732b33cb8"},{"sha":"ecf29e14b134a38a9bf7012cfeae68ba477718c3","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"srm: Further optimize SRM scheduler\n\nIn case there are less requests in the queue than we are still allowed to\nstart, the need to apply any prioritisation is less. Since the prioritisation\nis very expensive, this patch skips it entirely until the queue becomes longer\nthan the number of empty task slots. This is mostly a workaround for the slow\nprioritization. A future patch for master should reimplement how requests\nare prioritised.\n\nTarget: trunk\nRequire-notes: yes\nRequire-book: no\nRequest: 2.12\nRequest: 2.11\nRequest: 2.10\nIssue: #1533\nAcked-by: Dmitry Litvintsev <litvinse@fnal.gov>\nPatch: https://rb.dcache.org/r/8241/\n(cherry picked from commit 3cbc590cf3c20c3aa403959681ad189c5d5bfde5)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/ecf29e14b134a38a9bf7012cfeae68ba477718c3"},{"sha":"edd5fd2bc9c27ecd06f3c7bef7191a5194a9e8e7","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"srm: Optimize scheduler performance\n\nThe SRM scheduler slows down drastically once several thousand requests are\nqueued. This is particularly devestating for bulk bring-online requests. This\npatch applies some simple improvements to the queue management. The effect is\nbig enough that on my system-test build, a bulk stage request of 7000 files\nwent from at most pinning one to two files at a time to killing pinmanager (the\nlatter is of course not good, but that's a different problem - and one I guess\nis related more to HSQLDB than to pinmanager itself).\n\nTarget: trunk\nRequest: 2.12\nRequest: 2.11\nRequest: 2.10\nRequire-notes: yes\nRequire-book: no\nIssue: #1533\nAcked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>\nPatch: https://rb.dcache.org/r/8232/\n(cherry picked from commit fcb66c73a37996c4343464f524092b1814a7a3b1)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/edd5fd2bc9c27ecd06f3c7bef7191a5194a9e8e7"},{"sha":"92380a79a064f066ee9b2947f7d3bc0b9ad74855","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"pnfsmanager: Use per-thread upload directory to reduce lock contention\n\nDirectory creation and deletion in Chimera updates the link count and time\nstamps on the inode of the parent directory. For the upload directory this\nmeans that concurrent operations all block on updating this inode, thus leading\nto lock contention. This patch avoids this by introducing a per-thread\nsub-directory under the base upload directory. Thus two threads will never\nblock on accessing the same base upload directory.\n\nNo configuration changes are needed, but sites will see a change in how TURLs\nare constructed.\n\nTarget: trunk\nRequest: 2.12\nRequest: 2.11\nRequest: 2.10\nRequire-notes: yes\nRequire-book: yes\nAcked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>\nPatch: https://rb.dcache.org/r/8254/\n(cherry picked from commit e1a6b336f82c8f9e27445eb3cc3dcc34b0379d3f)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/92380a79a064f066ee9b2947f7d3bc0b9ad74855"},{"sha":"ea1b3d5a68a8b34e3fa5b7e676510d687e46e486","author":{"email":"d922616d71ca96f778c63ea49015b141afa0a2ae@desy.de","name":"Tigran Mkrtchyan"},"message":"nfs-proxy: close RPC client if we fail to connect\n\nfixes file descriptor leak.\n\nTarget: master, 2.12\nAcked-by: Gerd Behrmann\nRequire-book: no\nRequire-notes: yes\n(cherry picked from commit 0b4a327f7de19008998737ce92d06e33ce6724c1)\nSigned-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/ea1b3d5a68a8b34e3fa5b7e676510d687e46e486"},{"sha":"c88f4cd45c9e1e2ab3f5a5c081a65bfb735b00f4","author":{"email":"d922616d71ca96f778c63ea49015b141afa0a2ae@desy.de","name":"Tigran Mkrtchyan"},"message":"nfs-proxy: introduce an exponential backoff when connecting to pools\n\nacked-by: Gerd Behrmann\nTarget: master, 2.12\nRequire-notes: yes\nRequire-book: no\n(cherry picked from commit c35288045a0af5b95fa22671080571d72d9d920c)\nSigned-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/c88f4cd45c9e1e2ab3f5a5c081a65bfb735b00f4"},{"sha":"ae842b26f3ee97ce10d7dac220133c7890428bf7","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"pnfsmanager: Discard upload path creation request on TTL expiration\n\nTarget: trunk\nRequest: 2.12\nRequest: 2.11\nRequest: 2.10\nRequire-notes: yes\nRequire-book: no\nAcked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>\nPatch: https://rb.dcache.org/r/8255/\n(cherry picked from commit e3ba1b69d4f265a6bc5ebc8ed62d741356214e1d)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/ae842b26f3ee97ce10d7dac220133c7890428bf7"},{"sha":"0dd82463f268fd76d66f6dbb32329856022dff0c","author":{"email":"6f9af0692a7d1f914f468ec8416a8e6aa93178b4@gmail.com","name":"Gerd Behrmann"},"message":"common: Fix rounding error in request gauge\n\nThe gauge used for several request counters in dCache measures the average and\nstandard deviation of request execution times. The gauge internally maintains\nthe average and the average squared execution time.\n\nSince these are kept as longs, they are subject to rounding errors. When\nupdating the average, the existing average is multiplied by the old sample\ncount and the rounding error is thus amplified.  I have seen cases where this\nproduces clearly wrong results (e.g. prepare-to-put calls that initially failed\nquickly caused the average to be represented as 1 ms; assume that we have 1000\nsuch failing calls; after dCache is fully started subsequent calls take 100 ms;\nthe average would then be updated as (1 * 1000 + 100) / 1001, which would then\nbe rounded to 1; thus no matter how many 100ms samples are added, the average\nstays a 1 ms).\n\nThe patch resolves this by mantaining the sum of the execution times and\nsquared execution times.\n\nTarget: trunk\nRequire-notes: yes\nRequire-book: no\nRequest: 2.12\nRequest: 2.11\nRequest: 2.10\nRequest: 2.9\nRequest: 2.8\nRequest: 2.7\nRequest: 2.6\nAcked-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>\nPatch: https://rb.dcache.org/r/8260/\n(cherry picked from commit 4e65ce2a64d5218eb32cc76d1882282ff170e494)","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/0dd82463f268fd76d66f6dbb32329856022dff0c"},{"sha":"054d1e52d4270f277540c956458e74b71fdd2196","author":{"email":"e260d3bce7ebdf2b872b611fd23ca302a2bc8a49@desy.de","name":"Paul Millar"},"message":"Merge pull request #1536 from gbehrmann/fix/2.12/rb8224\n\npool: Fix timeout behavior of HSM requests","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/054d1e52d4270f277540c956458e74b71fdd2196"},{"sha":"da6d0927438571fdc51a6a48e18b5724db50f1d1","author":{"email":"e260d3bce7ebdf2b872b611fd23ca302a2bc8a49@desy.de","name":"Paul Millar"},"message":"Merge pull request #1539 from gbehrmann/fix/2.12/rb8226\n\nsrm: Fix queue size reporting","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/da6d0927438571fdc51a6a48e18b5724db50f1d1"},{"sha":"1ac4da578140f6fd1b939c66121ff232d0b8d52d","author":{"email":"e260d3bce7ebdf2b872b611fd23ca302a2bc8a49@desy.de","name":"Paul Millar"},"message":"Merge pull request #1542 from gbehrmann/fix/2.12/rb8227\n\npool: Add HSM options to hsm script remove callout","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/1ac4da578140f6fd1b939c66121ff232d0b8d52d"},{"sha":"870877fa7b054212dcd940e89fe5df00a3cb0e25","author":{"email":"e260d3bce7ebdf2b872b611fd23ca302a2bc8a49@desy.de","name":"Paul Millar"},"message":"Merge pull request #1545 from gbehrmann/fix/2.12/rb8228\n\npinmanager: Fix listing by id","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/870877fa7b054212dcd940e89fe5df00a3cb0e25"},{"sha":"2fe9910c17a6442a3b191b6bcabe0b2e62204b41","author":{"email":"e260d3bce7ebdf2b872b611fd23ca302a2bc8a49@desy.de","name":"Paul Millar"},"message":"Merge pull request #1552 from gbehrmann/fix/2.12/rb8229\n\nsrm: Fix sync to async mode timeout","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/2fe9910c17a6442a3b191b6bcabe0b2e62204b41"},{"sha":"d6804193b2acfc3e73a46738defae572b47a0e9a","author":{"email":"e260d3bce7ebdf2b872b611fd23ca302a2bc8a49@desy.de","name":"Paul Millar"},"message":"Merge pull request #1555 from gbehrmann/fix/2.12/rb8234\n\nsystem-test: Enable MVCC and logging for HSQLDB","distinct":false,"url":"https://api.github.com/repos/gbehrmann/dcache/commits/d6804193b2acfc3e73a46738defae572b47a0e9a"}]}