Hi,
as we have a discussion ongoing about setting up a new PC, I thought I could jump in with my current issue : I have retired my laptop and bought a desktop PC. Gone through the fresh installation process (once every 12 years is OK to declutter a bit), and rsync'ed my home directory (parts of it, mostlyskipped ~/.local and~/.config where I found some 10-20 years old files from programs I used once but never since).
Now I still want to use my laptop when I'm on the road, which happens rather seldomly (once every 2 weeks), i.e. have my files, my bookmarks etc available on my laptop. Also not only have to sync my /home/$USER but also /root and some more mountpoints (source code, website stuff, audio, video, pictures), totaling around 1.5TB of data
I've been playing around with rsync and adding many many exclude rules to
a) limit the amount of files being transferred (eg exclude ~/.cache, browser's cache, thunderbird's imap cache etc) so I can keep an overview if something is getting overwritten that I don't want
b) keep certain files (especially XFCE config etc) distinct as eg I have 2 monitors on my PC vs only 1 on the laptop -> different menus, window settings etc
Now this is pretty tedious, initial rsync's dry-run showed ~700k files that would be sync'ed, which I brought down to 1k after several iterations and adjustment of exclude settings, and this is so far my first run - currently using my laptop and will sync back when I'm home again... let's see how many files will differ then !
So I'm looking for more easy-to-use alternatives. Most important thing : I want dry-run functionality to prevent unwanted data loss, which means automatic sync tools are not the way to go ! I think synthing is such a candidate, unless I missed the dry-run option somewhere. Have used unison many many years ago, but I think it was also not perfect and had issues handling thousands/millions of files. Heard of some new systemd homesync (?), but didn't have a look at it now, sounds also like it automatically syncs.
Any thoughts / recommendations / advices ?
Many thanks
Joel
El lun, 31 oct 2022 a las 16:06, Joel (j-archlinux@joel-hatsch.net) escribió:
Hi,
Hi!
as we have a [..] syncs.
Any thoughts / recommendations / advices ?
Synthing. Don't complicate yourself. It is not necessary to have the dry-run because Syncthing will keep the repositories (directories) that you tell it to keep completely synchronized in real time, you do not have to be launching the command every time you want to synchronize.
Personally I use it on large repositories (a few terabytes) and zero problems so far. You also have the option of configuring file versioning, so that when a file is modified you can keep backup copies of it, making it impossible to lose anything.
Greetings.
-- Óscar García Amor | ogarcia at moire.org | http://ogarcia.me
On 31/10/2022 15:24, Óscar García Amor wrote:
Syncthing
Plus one for Syncthing. It works well.
I use it across a set of around 10 devices, and, as an example of larger-scale use, Chaotic-AUR uses it for mirror synchronisation.
On Mon, 2022-10-31 at 16:24 +0100, Óscar García Amor wrote:
El lun, 31 oct 2022 a las 16:06, Joel (j-archlinux@joel-hatsch.net) escribió:
as we have a [..] syncs.
Any thoughts / recommendations / advices ?
Synthing. Don't complicate yourself. It is not necessary to have the dry-run because Syncthing will keep the repositories (directories) that you tell it to keep completely synchronized in real time, you do not have to be launching the command every time you want to synchronize.
-- Óscar García Amor | ogarcia at moire.org | http://ogarcia.me
I've been using rsync, unison and nowadays I am using syncthing. My latest setup, which I am super duper happy with: basically I have 3 devices running syncthing: my workstation, my laptop and a small raspberry. The main issue with unison was only: when shall I run it? Just before I leave, I am usually stressed. It's when I am on my way that I suddenly miss some of my latest files. That's what I have the raspberry for. When I leave, I turn off the workstation, which at that time will be in sync with the raspberry. When I arrive at the hotel, my laptop will sync with the raspberry. No missing files anymore. Now: the rsync comes in play. On the raspberry I have multiple hard disks connected. Once a day at night, I perform an incremental backup using rsync and hard links. In case there is ever some sync issue, I can go into the backup and pick the file from yesterday or any day before. For syncthing: do not sync the entire home folder; just sync each folder individually. It turned out to be of an advantage: I nowadays sync a few of the folders with my phone. And my setup grew with time. There are more devices and the sync structure got more complex. When you start to love syncthing as much as I do, you might add some more devices. Unison worked perfect for my very large home folder (1.6 TB), you should not worry about that part, as long as the number of files in folders with changes is not too high. I.e., folders with 1k files are fine as long as they are stable. A tiny change in such folder causes heavy work.
regards,
ente
On 11/2/22 14:23, ente wrote:
On Mon, 2022-10-31 at 16:24 +0100, Óscar García Amor wrote:
El lun, 31 oct 2022 a las 16:06, Joel (j-archlinux@joel-hatsch.net) escribió:
as we have a [..] syncs.
Any thoughts / recommendations / advices ?
Synthing. Don't complicate yourself. It is not necessary to have the dry-run because Syncthing will keep the repositories (directories) that you tell it to keep completely synchronized in real time, you do not have to be launching the command every time you want to synchronize.
-- Óscar García Amor | ogarcia at moire.org |http://ogarcia.me
I've been using rsync, unison and nowadays I am using syncthing. My latest setup, which I am super duper happy with: basically I have 3 devices running syncthing: my workstation, my laptop and a small raspberry. The main issue with unison was only: when shall I run it? Just before I leave, I am usually stressed. It's when I am on my way that I suddenly miss some of my latest files. That's what I have the raspberry for. When I leave, I turn off the workstation, which at that time will be in sync with the raspberry. When I arrive at the hotel, my laptop will sync with the raspberry. No missing files anymore. Now: the rsync comes in play. On the raspberry I have multiple hard disks connected. Once a day at night, I perform an incremental backup using rsync and hard links. In case there is ever some sync issue, I can go into the backup and pick the file from yesterday or any day before. For syncthing: do not sync the entire home folder; just sync each folder individually. It turned out to be of an advantage: I nowadays sync a few of the folders with my phone. And my setup grew with time. There are more devices and the sync structure got more complex. When you start to love syncthing as much as I do, you might add some more devices. [...]
Hi,
many thanks for all the replies, much appreciated.
I've given syncthing a try as this was the most recommended. Also the point of being able to sync after having left is a nice feature !
I've set up multiple sync targets for ~/Documents, ~/git_repos etc... and the sync works like a charm
Now I had a look at my home directory, and found many dot files (.zshrc, .emacs, etc) and dot directories I also wanted to sync. Ente's recommendation was to set up sync individually, but this will not really scale :-(
My idea was to set up a sync for my whole home directory, add an ignore pattern for *, and then include the single dot files and dot dirs individually using negated ignore patterns. Of course, with this kind of setup, I could also add the ~/Documents etc from above in the same unique sync target
Sounds a bit weird ... so am I on the right track or is there a more elegant way of doing ?
Thanks
Joel
On 11/6/22 17:34, Joel wrote:
On 11/2/22 14:23, ente wrote:
On Mon, 2022-10-31 at 16:24 +0100, Óscar García Amor wrote:
El lun, 31 oct 2022 a las 16:06, Joel (j-archlinux@joel-hatsch.net) escribió:
as we have a [..] syncs.
Any thoughts / recommendations / advices ?
Synthing. Don't complicate yourself. It is not necessary to have the dry-run because Syncthing will keep the repositories (directories) that you tell it to keep completely synchronized in real time, you do not have to be launching the command every time you want to synchronize.
-- Óscar García Amor | ogarcia at moire.org |http://ogarcia.me
I've been using rsync, unison and nowadays I am using syncthing. My latest setup, which I am super duper happy with: basically I have 3 devices running syncthing: my workstation, my laptop and a small raspberry. The main issue with unison was only: when shall I run it? Just before I leave, I am usually stressed. It's when I am on my way that I suddenly miss some of my latest files. That's what I have the raspberry for. When I leave, I turn off the workstation, which at that time will be in sync with the raspberry. When I arrive at the hotel, my laptop will sync with the raspberry. No missing files anymore. Now: the rsync comes in play. On the raspberry I have multiple hard disks connected. Once a day at night, I perform an incremental backup using rsync and hard links. In case there is ever some sync issue, I can go into the backup and pick the file from yesterday or any day before. For syncthing: do not sync the entire home folder; just sync each folder individually. It turned out to be of an advantage: I nowadays sync a few of the folders with my phone. And my setup grew with time. There are more devices and the sync structure got more complex. When you start to love syncthing as much as I do, you might add some more devices. [...]
Hi,
many thanks for all the replies, much appreciated.
I've given syncthing a try as this was the most recommended. Also the point of being able to sync after having left is a nice feature !
I've set up multiple sync targets for ~/Documents, ~/git_repos etc... and the sync works like a charm
Now I had a look at my home directory, and found many dot files (.zshrc, .emacs, etc) and dot directories I also wanted to sync. Ente's recommendation was to set up sync individually, but this will not really scale :-(
My idea was to set up a sync for my whole home directory, add an ignore pattern for *, and then include the single dot files and dot dirs individually using negated ignore patterns. Of course, with this kind of setup, I could also add the ~/Documents etc from above in the same unique sync target
Sounds a bit weird ... so am I on the right track or is there a more elegant way of doing ?
Thanks
Joel
Have you tried any of the ideas from the wiki's dotfiles page [1]? I personally use a ~/dotfiles directory and stow, and while I don't sync it, it could potentially work pretty well with syncthing.
AVB
El dom, 6 nov 2022 a las 23:35, Joel (j-archlinux@joel-hatsch.net) escribió:
On 11/2/22 14:23, ente wrote:
Hi,
Hi again!
many [..] of doing ?
My advice is that not all dotfiles are candidates for synchronization. Many of them are only interesting on a particular machine.
What I do is to have a .dot directory with the configurations I am interested in keeping on all machines and "sync" that directory in git. This allows you to have a good versioning of those configuration files and work only with those that can be ported from one machine to another.
You can take a look at it here: https://github.com/ogarcia/dot The instructions to deploy it are in the repository wiki: https://github.com/ogarcia/dot/wiki. Basically what it does is to put this script as post-commit (so that if you make changes in the repository it will be executed) and run it. What this script does is to create symbolic links from your home to the .dot files with the same name (for example the .dot bashrc file is linked from your home as ~/.bashrc). This way you can have everything you need in the .dot directory without it being scattered all over your home.
Greetings. -- Óscar García Amor | ogarcia at moire.org | http://ogarcia.me
On 11/7/22 09:06, Óscar García Amor wrote:
El dom, 6 nov 2022 a las 23:35, Joel (j-archlinux@joel-hatsch.net) escribió:
On 11/2/22 14:23, ente wrote:
Hi,
Hi again!
many [..] of doing ?
My advice is that not all dotfiles are candidates for synchronization. Many of them are only interesting on a particular machine.
What I do is to have a .dot directory with the configurations I am interested in keeping on all machines and "sync" that directory in git. This allows you to have a good versioning of those configuration files and work only with those that can be ported from one machine to another.
You can take a look at it here: https://github.com/ogarcia/dot The instructions to deploy it are in the repository wiki: https://github.com/ogarcia/dot/wiki.%C2%A0Basically what it does is to put this script as post-commit (so that if you make changes in the repository it will be executed) and run it. What this script does is to create symbolic links from your home to the .dot files with the same name (for example the .dot bashrc file is linked from your home as ~/.bashrc). This way you can have everything you need in the .dot directory without it being scattered all over your home.
Hi Oscar, Anthony,
that's a great idea !
At least for the dot*files* it should do the trick perfectly. For the dot*dirs* it could work for the ones with small amount of files and changes. And for the others with much more content, as Oscar said, not all are candidates for sync, so the amount could still be handled by setting up dedicated sync targets in syncthing (eg .ssh, .gnupg, .aqbanking etc)
Thanks again for the good hints
Joel
On Mon, 31 Oct 2022 16:06:16 +0100 Joel j-archlinux@joel-hatsch.net wrote:
Hi,
as we have a discussion ongoing about setting up a new PC, I thought I could jump in with my current issue : I have retired my laptop and bought a desktop PC. Gone through the fresh installation process (once every 12 years is OK to declutter a bit), and rsync'ed my home directory (parts of it, mostlyskipped ~/.local and~/.config where I found some 10-20 years old files from programs I used once but never since).
Now I still want to use my laptop when I'm on the road, which happens rather seldomly (once every 2 weeks), i.e. have my files, my bookmarks etc available on my laptop. Also not only have to sync my /home/$USER but also /root and some more mountpoints (source code, website stuff, audio, video, pictures), totaling around 1.5TB of data
I've been playing around with rsync and adding many many exclude rules to
a) limit the amount of files being transferred (eg exclude ~/.cache, browser's cache, thunderbird's imap cache etc) so I can keep an overview if something is getting overwritten that I don't want
b) keep certain files (especially XFCE config etc) distinct as eg I have 2 monitors on my PC vs only 1 on the laptop -> different menus, window settings etc
Now this is pretty tedious, initial rsync's dry-run showed ~700k files that would be sync'ed, which I brought down to 1k after several iterations and adjustment of exclude settings, and this is so far my first run - currently using my laptop and will sync back when I'm home again... let's see how many files will differ then !
So I'm looking for more easy-to-use alternatives. Most important thing : I want dry-run functionality to prevent unwanted data loss, which means automatic sync tools are not the way to go ! I think synthing is such a candidate, unless I missed the dry-run option somewhere. Have used unison many many years ago, but I think it was also not perfect and had issues handling thousands/millions of files. Heard of some new systemd homesync (?), but didn't have a look at it now, sounds also like it automatically syncs.
The systemd home stuff is all pretty new - I wouldn't use it yet myself.
Any thoughts / recommendations / advices ?
Now you've gone through the painful bit of setting up a suitable rsync command, why not just store that in a file as a bash script somewhere (maybe ~/bin ?) Then you can easily use it whenever you want and shouldn't need to spend any more time than typing its name :)
Many thanks
Joel
arch-general@lists.archlinux.org