{"action":"created","issue":{"url":"https://api.github.com/repos/bit-team/backintime/issues/9","labels_url":"https://api.github.com/repos/bit-team/backintime/issues/9/labels{/name}","comments_url":"https://api.github.com/repos/bit-team/backintime/issues/9/comments","events_url":"https://api.github.com/repos/bit-team/backintime/issues/9/events","html_url":"https://github.com/bit-team/backintime/issues/9","id":110866225,"number":9,"title":"hardlinking doesnt work with schedule per included folder enabled","user":{"login":"Germar","id":2731999,"avatar_url":"https://avatars.githubusercontent.com/u/2731999?v=3","gravatar_id":"","url":"https://api.github.com/users/Germar","html_url":"https://github.com/Germar","followers_url":"https://api.github.com/users/Germar/followers","following_url":"https://api.github.com/users/Germar/following{/other_user}","gists_url":"https://api.github.com/users/Germar/gists{/gist_id}","starred_url":"https://api.github.com/users/Germar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Germar/subscriptions","organizations_url":"https://api.github.com/users/Germar/orgs","repos_url":"https://api.github.com/users/Germar/repos","events_url":"https://api.github.com/users/Germar/events{/privacy}","received_events_url":"https://api.github.com/users/Germar/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/bit-team/backintime/labels/Fix%20Released","name":"Fix Released","color":"ddffdd"},{"url":"https://api.github.com/repos/bit-team/backintime/labels/High","name":"High","color":"ffdddd"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2015-10-11T16:28:49Z","updated_at":"2015-10-11T17:25:18Z","closed_at":null,"body":"Schedule per included folder is BUGGY, do NOT use it unless you want to do some debugging or if you have plenty of space (and computing power)!\n\nThe snapshots are too big, also without changing anything the snapshot size is the size of the folder, so hardlinking is not working. For the tasks with higher frequency the hardlinking fails: consuming the full space of all the folders that are scheduled for backup (also the ones with lower frequencies). The time to make the snapshot is incredibly long (exceeding 10 times the time to create a normal complete snapshot, which is logical as it recreates every single bit)\nThe lowest frequency tasks and manual backups behave normally\n\n"},"comment":{"url":"https://api.github.com/repos/bit-team/backintime/issues/comments/147223116","html_url":"https://github.com/bit-team/backintime/issues/9#issuecomment-147223116","issue_url":"https://api.github.com/repos/bit-team/backintime/issues/9","id":147223116,"user":{"login":"Germar","id":2731999,"avatar_url":"https://avatars.githubusercontent.com/u/2731999?v=3","gravatar_id":"","url":"https://api.github.com/users/Germar","html_url":"https://github.com/Germar","followers_url":"https://api.github.com/users/Germar/followers","following_url":"https://api.github.com/users/Germar/following{/other_user}","gists_url":"https://api.github.com/users/Germar/gists{/gist_id}","starred_url":"https://api.github.com/users/Germar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Germar/subscriptions","organizations_url":"https://api.github.com/users/Germar/orgs","repos_url":"https://api.github.com/users/Germar/repos","events_url":"https://api.github.com/users/Germar/events{/privacy}","received_events_url":"https://api.github.com/users/Germar/received_events","type":"User","site_admin":false},"created_at":"2015-10-11T17:25:17Z","updated_at":"2015-10-11T17:25:17Z","body":"(by bratdaking)\nBy the way I changed the name and the description a bit\n\n2009/9/1 Bart <email address hidden>\n\n> Some strange things I notice: your includes and excludes are a bit weird\n> like:\n> --exclude=\"/home/peter/BiTtest\"\n> --include=\"/home/peter/BiTtest/subfolder/**\"\n> Should not mention the exclude... oh wait it should mention it if the\n> folder that should only be backupped once every year or something is a\n> subfolder of a folder that should be backupped every 10 minutes, it should\n> exclude it. And apparently it does not overwrite the includes -> it produces\n> the snapshot with the files anyway\n> So that makes sense\n>\n> What does not on the other hand is the cp -alb function for a folder it\n> already copied, anyway, actually what is the whole reason for this command,\n> ignored folders are copied anyway during the hardlinking?\n>\n> At the moment I am checking a quite difficult backup scheme to see if this\n> behaviour is reproducable here. Probably some commands for the schedule per\n> folder backup do the wrong thing.\n> Ok, it is now making a snapshot and calls indeed the cp -alb command, it\n> consumes loads of time. It even comes over the 10 minutes (the rsync took\n> not more than 2 minutes)  so it delays my next (hourly) backup. However it\n> does do the right thing if the directories are disjoint. I will try to make\n> an exclude that is joint...\n> The backup dir has grown incredibly in size, however I do not see the ~\n> behaviour yet. What I do see is time consumption, and size consumption. The\n> hardlinking definitely fails when it does the small job (I have two\n> schedules: 1 every 10 minutes, 1 every hour). The every hour job goes\n> perfectly fine and as normal, but the 10 minutes job is behaving not as it\n> should (actually right at this point it consumed all the space there was,\n> filling up my logs with warnings, while there used to be plenty of space two\n> hours ago when I started this test, it also did not rename or remove the\n> new-snapshot folder any longer, -> another bug...)\n> So far my main conclusion is that you should not use the schedule per\n> included folder option as it is still buggy!\n>\n> Cheers,\n> Bart\n>\n>\n> 2009/8/31 Borph <email address hidden>\n>\n> I reproduced now the issue with the *~ files!\n>>\n>> I made two folders with two files each:\n>> peter@borphtux:~$ ls /home/peter/BiTtest/\n>> file1.txt  file2.txt  subfolder\n>> peter@borphtux:~$ ls /home/peter/BiTtest/subfolder/\n>> file3.txt  file4.txt\n>>\n>> The schedule was:\n>> /home/peter/BiTtest:   every 10min\n>> /home/peter/BiTtest/subfolder:  every 5min\n>>\n>> The config file (part):\n>> snapshots.exclude_patterns=*.backup*:/media:/lost+found:**/.thumbnails/**\n>> snapshots.expert.per_directory_schedule=true\n>>\n>> snapshots.include_folders=/home/peter/BiTtest|4:/home/peter/BiTtest/subfolder|2\n>>\n>> I made a forced backup and got a \"20090831-203106\" folder with the\n>> expected files.\n>>\n>> I changed file1 and file3 and waited. 20090831-204000 got created with\n>> including both folders as expected.\n>>\n>> I changed again both files. 20090831-204500 got created with these files:\n>> peter@borphtux:~$ ls -l\n>> /media/BigBilly/System/BackInTime/backintime/20090831-204500/backup/home/peter/BiTtest/\n>> insgesamt 12\n>> -r--r--r-- 2 peter peter   11 2009-08-31 20:35 file1.txt\n>> -r--r--r-- 4 peter peter   10 2009-08-31 20:28 file2.txt\n>> drwxr-xr-x 2 peter peter 4096 2009-08-31 20:36 subfolder\n>> (this is as expected, unchanged, but was not due)\n>>\n>> peter@borphtux:~$ ls -l\n>> /media/BigBilly/System/BackInTime/backintime/20090831-204500/backup/home/peter/BiTtest/subfolder/\n>> insgesamt 16\n>> -r--r--r-- 2 peter peter 11 2009-08-31 20:36 file3.txt\n>> -r--r--r-- 1 peter peter 12 2009-08-31 20:41 file3.txt~\n>> -r--r--r-- 4 peter peter 10 2009-08-31 20:28 file4.txt\n>> -r--r--r-- 1 peter peter 10 2009-08-31 20:28 file4.txt~\n>>\n>> Hum, these *~ files got created! During the next 10-min backup they\n>> disappear, ok, but this means they come each time a subfolder is backed\n>> up and a folder above not!\n>>\n>> Lets see the logfile:\n>> INFO: Create hard-links\n>> INFO: Command \"cp -al\n>> \"/media/BigBilly/System/BackInTime/backintime/20090831-204000/backup/home/peter/BiTtest/subfolder\"*\n>> \"/media/BigBilly/System/BackInTime/backintime/new_snapshot/backup/home/peter/BiTtest/subfolder\"\"\n>> returns 0\n>> INFO: Call rsync to take the snapshot\n>> INFO: Command \"rsync -aEAX  -v --delete-excluded  --chmod=Fa-w,D+w\n>> --whole-file --delete  --exclude=\"/media/BigBilly/System/BackInTime\"\n>> --exclude=\"/root/.local/share/backintime\"\n>> --include=\"/home/peter/BiTtest/subfolder/\" --include=\"/home/peter/BiTtest/\"\n>> --include=\"/home/peter/\" --include=\"/home/\" --exclude=\"*.backup*\"\n>> --exclude=\"/media\" --exclude=\"/lost+found\" --exclude=\"**/.thumbnails/**\"\n>> --exclude=\"/home/peter/BiTtest\" --include=\"/home/peter/BiTtest/subfolder/**\"\n>> --exclude=\"*\" /\n>> \"/media/BigBilly/System/BackInTime/backintime/new_snapshot/backup/\"\" returns\n>> 0\n>> INFO: Save permissions\n>> INFO: Command \"cp -alb\n>> \"/media/BigBilly/System/BackInTime/backintime/20090831-204000/backup/home/peter/BiTtest/\"*\n>> \"/media/BigBilly/System/BackInTime/backintime/new_snapshot/backup/home/peter/BiTtest\"\"\n>> returns 0\n>>\n>> So I guess the second cp -alb copies the old version (the previous\n>> snapshot) again over the new_snapshot. This explains the file-dates!\n>>\n>> Of course the cp -alb after the rsync is meant to copy hard-linked the\n>> folders, which have not been synced. But if the folders are not\n>> disjoint, this command includes recursively already synced files,\n>> attempting to overwrite them.\n>>\n>> Last but not least:\n>> peter@borphtux:~$ backintime -v\n>> Back In Time\n>> Version: 0.9.26\n>>\n>> --\n>> hardlinking doesnt work from cronjob with schedule per included folder\n>> enabled\n>> https://bugs.launchpad.net/bugs/412470\n>> You received this bug notification because you are a direct subscriber\n>> of the bug.\n>>\n>\n>\n"}}