Rsync failures - need to be able to exclude directories

[bcbc]

Every now and then rsync will fail due to permissions errors or corruption. E.g. firefox profile corruption and /dev/.bootchart/ (also the recent addition of lightdm added a new .gvfs directory which cannot be copied.
It's not possible to anticipate every new directory so the best solution is to add a new option --exclude= that can be repeated and allow users to build a custom exclusion list.

[bcbc]

Built in a new --resume option so you can fix any corruption and resume the migration (without starting from scratch). Doesn't help for new .gvfs directories, but it's a start.

[andersk]

This is exactly what rsync --one-file-system (rsync -x) is for. If you use that, you shouldn’t need any exclude options.

[bcbc]

What about separate virtual disks?

[andersk]

Find them in /etc/fstab or /etc/mtab (whitelist by filesystem type, perhaps) and rsync -x them separately?

[bcbc]

Yes. That's definitely a solution. But I still like the idea of excluding directories because of this tendency for new gvfs folders to crop up with new releases (and that stops the thing dead). I put in a special exclude already for anything mounted so that's why I haven't bothered with the -x or copying virtual disks separately.

Thanks for the suggestions though. PS do you know of a way I could bypass failures on .gvfs directories automatically without a specific --exclude= ?

Edit: Actually, I still need to support dynamic excludes for e.g. things like /dev/.bootchart

[andersk]

Perhaps I wasn’t clear? That was the entire point of my suggestion. The reason that ~/.gvfs stops the thing dead is because it’s a separate filesystem (in this case, a FUSE filesystem). So is /dev/.bootchart/proc (procfs). So is any other problem folder that crops up with new releases, or with packages you didn’t anticipate (e.g. openafs-client’s networked filesystem /afs). rsync -x is guaranteed to avoid all of those problems by only copying things that are part of the real physical filesystem. Trying to copy anything that isn’t part of the real physical filesystem is wrong, both because it doesn’t usually work (as you’ve seen), and because it wouldn’t make sense even if it does work.

[bcbc]

Okay... got it. Thanks for explaining.

[bcbc]

Okay that works great on the .gvfs folders, but not on /home/*/.cache/gvfs. Do you know anything about that?

wubi-move.sh: Copying from /home rsync: readlink_stat("/home/bcbc/.cache/gvfs") failed: Permission denied (13) IO error encountered -- skipping file deletion rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]

Command is: rsync -a --delete --one-file-system "$root""$1" "$target"
i.e. rsync -a --delete --one-file-system /home /targetmountpoint

PS I can copy /home/bcbc/gvfs as user bcbc, but it fails with sudo

Attributes:
dr-x------ 2 bcbc bcbc 0 Feb 26 12:55 gvfs

[bcbc]

Had problems with .gvfs in testing. Added back exclusions for now.

Still remains to provide an easy way for an end user to exclude a directory without editing the script