{"action":"created","issue":{"url":"https://api.github.com/repos/ncw/rclone/issues/263","labels_url":"https://api.github.com/repos/ncw/rclone/issues/263/labels{/name}","comments_url":"https://api.github.com/repos/ncw/rclone/issues/263/comments","events_url":"https://api.github.com/repos/ncw/rclone/issues/263/events","html_url":"https://github.com/ncw/rclone/issues/263","id":123843857,"number":263,"title":"on ACD directories with many files cannot properly by sync'd.","user":{"login":"left1000","id":16432525,"avatar_url":"https://avatars.githubusercontent.com/u/16432525?v=3","gravatar_id":"","url":"https://api.github.com/users/left1000","html_url":"https://github.com/left1000","followers_url":"https://api.github.com/users/left1000/followers","following_url":"https://api.github.com/users/left1000/following{/other_user}","gists_url":"https://api.github.com/users/left1000/gists{/gist_id}","starred_url":"https://api.github.com/users/left1000/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/left1000/subscriptions","organizations_url":"https://api.github.com/users/left1000/orgs","repos_url":"https://api.github.com/users/left1000/repos","events_url":"https://api.github.com/users/left1000/events{/privacy}","received_events_url":"https://api.github.com/users/left1000/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2015-12-25T01:45:51Z","updated_at":"2015-12-27T16:49:43Z","closed_at":null,"body":"When uploading 100,000 or more files rclone will often get stuck trying to read the ls list of files from ACD.\r\nThis will result in a large number of errors that say \"file with this name already exists\" because it will not find the file then it will add it to the list of files to upload then when it's time to upload it'll find it already existed.\r\n\r\nEven worse this will often result in the checks portion of rclone getting stuck, literally forever, for some files. This means that when I want to sync 100,000-1,000,000 files or so to ACD I simply can't.\r\n\r\nI don't blame rclone, it's probably all ACD's fault, but I am wondering if anything could be done to help this out? perhaps rclone could save a ls from acd? allowing rclone some command to generate a full ls? allowing the user to run it a few times? to make sure they have a good ls? then start the sync using that locally stored ls? I don't know...\r\n\r\nI started just uploading large 7z files of my largest folders instead.\r\n\r\n\r\nanother sub issue here is thusly, if remote:E:\\ contains let's say 10subfolders and those 10subfolders contain 1,000,000 files... and I want to upload a single file directly to \"remote:E\" instead of just confirming that file isn't one of the ten things in the root directory it will demand to check all 1,000,000 files (or well maybe not \"check\" but the ls command will be for all subdirectories)... this has resulted in me wanting to upload to \"remote:incoming\" then use ACD's shitty website to move the file back to \"remote:E\"...\r\n\r\nideally the ls rclone runs from ACD would only check subdirectories it needs to? this would speed things up a lot especially because ACD's response to ls is very buggy."},"comment":{"url":"https://api.github.com/repos/ncw/rclone/issues/comments/167425760","html_url":"https://github.com/ncw/rclone/issues/263#issuecomment-167425760","issue_url":"https://api.github.com/repos/ncw/rclone/issues/263","id":167425760,"user":{"login":"zeshanb","id":6672739,"avatar_url":"https://avatars.githubusercontent.com/u/6672739?v=3","gravatar_id":"","url":"https://api.github.com/users/zeshanb","html_url":"https://github.com/zeshanb","followers_url":"https://api.github.com/users/zeshanb/followers","following_url":"https://api.github.com/users/zeshanb/following{/other_user}","gists_url":"https://api.github.com/users/zeshanb/gists{/gist_id}","starred_url":"https://api.github.com/users/zeshanb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zeshanb/subscriptions","organizations_url":"https://api.github.com/users/zeshanb/orgs","repos_url":"https://api.github.com/users/zeshanb/repos","events_url":"https://api.github.com/users/zeshanb/events{/privacy}","received_events_url":"https://api.github.com/users/zeshanb/received_events","type":"User","site_admin":false},"created_at":"2015-12-27T16:49:43Z","updated_at":"2015-12-27T16:49:43Z","body":"No sure. Looking at line 244 on swift.go there is a default 256 object list\nlimit for some reason. One would need to know the requirements of Auth API\nbeing used and configs made by data host Admin.\n\n@ncw Is rClone trying to host objects(Fs) in a object(fs) before\ntransporting? Is there a way to find out the data throttle requirements\nwhile listing and transferring?\n\n\n\n\nOn Sun, Dec 27, 2015 at 10:17 AM, left1000 <notifications@github.com> wrote:\n\n> had no trouble at all with a folder with 30,000 files in it this week, so\n> it might only come up when a folder on acd has 100,000-300,000 or more\n> files (roughly) like the folders I had been testing out the week before\n> that.\n>\n> —\n> Reply to this email directly or view it on GitHub\n> <https://github.com/ncw/rclone/issues/263#issuecomment-167419750>.\n>\n"}}