{"action":"created","issue":{"url":"https://api.github.com/repos/dmwm/WMCore/issues/6288","labels_url":"https://api.github.com/repos/dmwm/WMCore/issues/6288/labels{/name}","comments_url":"https://api.github.com/repos/dmwm/WMCore/issues/6288/comments","events_url":"https://api.github.com/repos/dmwm/WMCore/issues/6288/events","html_url":"https://github.com/dmwm/WMCore/pull/6288","id":108502975,"number":6288,"title":"simplify/rewrite dbs3upload","user":{"login":"hufnagel","id":1798355,"avatar_url":"https://avatars.githubusercontent.com/u/1798355?v=3","gravatar_id":"","url":"https://api.github.com/users/hufnagel","html_url":"https://github.com/hufnagel","followers_url":"https://api.github.com/users/hufnagel/followers","following_url":"https://api.github.com/users/hufnagel/following{/other_user}","gists_url":"https://api.github.com/users/hufnagel/gists{/gist_id}","starred_url":"https://api.github.com/users/hufnagel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hufnagel/subscriptions","organizations_url":"https://api.github.com/users/hufnagel/orgs","repos_url":"https://api.github.com/users/hufnagel/repos","events_url":"https://api.github.com/users/hufnagel/events{/privacy}","received_events_url":"https://api.github.com/users/hufnagel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":12,"created_at":"2015-09-26T23:34:05Z","updated_at":"2015-09-28T00:25:55Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/dmwm/WMCore/pulls/6288","html_url":"https://github.com/dmwm/WMCore/pull/6288","diff_url":"https://github.com/dmwm/WMCore/pull/6288.diff","patch_url":"https://github.com/dmwm/WMCore/pull/6288.patch"},"body":"Current code was not compatible with the Tier0. It somehow worked so far, but never really correctly (mostly causing internal multiplication of work) and with the huge backlog we accumulated due to the few deadlocks and component restarts it completely broke it. I tried to keep the changes minimal. Didn't quite succeed, but this is about as minimal as I can get it.\r\n\r\nI needed to remove the workflow parameter from DBSBufferBlock as workflow is not a property of a block. Replaced it with a workflows set, which is filled when files are added to the DBSBufferBlock. Then for block closing I look at whether any of the workflows in a block is completed.\r\n\r\nDataset_algo is also not a property of a block, but it was used as a key for a block cache and to split blocks into smaller groups to process. Removed the whole cache based on dataset_algo as it wasn't needed and replaced the splitting to use datasetpath instead. That actually is a property of a block, but I had to add it to DBSBufferBlock as it wasn't stored.\r\n\r\nAs I had to touch some queries for these datasetpath and workflow changes anyways, optimized these queries when I saw something obvious. Saw plenty more to be optimized there, but constrained myself...\r\n\r\nReworked the transaction code in DBSBufferUtil, read methods don't need transactions, they just slow things down. Removed them. Left the transactions intact for writes.\r\n\r\nTested this code in a short Tier0 replay, both in live mode and in the mode where DBSUpload component is restarted and has to pick up from open blocks stored in the database. Worked.\r\n\r\nCurrently running a larger Tier0 replay with Express and Bulk processing and automatic transfers from T0_CH_CERN_Disk to T2_CH_CERN and automatic deletions from T0_CH_CERN_DIsk. If that works too I am going to pull the trigger and deploy this in the production Tier0."},"comment":{"url":"https://api.github.com/repos/dmwm/WMCore/issues/comments/143606804","html_url":"https://github.com/dmwm/WMCore/pull/6288#issuecomment-143606804","issue_url":"https://api.github.com/repos/dmwm/WMCore/issues/6288","id":143606804,"user":{"login":"DMWMBot","id":1124170,"avatar_url":"https://avatars.githubusercontent.com/u/1124170?v=3","gravatar_id":"","url":"https://api.github.com/users/DMWMBot","html_url":"https://github.com/DMWMBot","followers_url":"https://api.github.com/users/DMWMBot/followers","following_url":"https://api.github.com/users/DMWMBot/following{/other_user}","gists_url":"https://api.github.com/users/DMWMBot/gists{/gist_id}","starred_url":"https://api.github.com/users/DMWMBot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DMWMBot/subscriptions","organizations_url":"https://api.github.com/users/DMWMBot/orgs","repos_url":"https://api.github.com/users/DMWMBot/repos","events_url":"https://api.github.com/users/DMWMBot/events{/privacy}","received_events_url":"https://api.github.com/users/DMWMBot/received_events","type":"User","site_admin":false},"created_at":"2015-09-28T00:25:55Z","updated_at":"2015-09-28T00:25:55Z","body":"Beginning pylint and unit tests.\nSee https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/666/ for details"}}