random thoughts

Backup AWS S3 bucket to external location

· Rafał

Aside from S3 bucket replication feature you might want to backup your data outside of your current cloud provider (for a myriad of valid reasons).

If you’re in Europe you might consider Hetzner which once again offers the best bang for your buck; alternatively - if you’re based in the US - Backblaze might be a better fit. Either way, I recommend using rclone which integrates out-of-the-box with S3 and all the protocols you can think of.

Few tips based on my personal experience:

  • enable encryption right away, but be aware that you might run into limitations if you have long filenames, like over 143 characters

  • observe AWS costs, as S3 can skyrocket, depending on your buckets structure (and of course - bucket size); to make the backup job less API/transfer heavy a lot of folks experiment with fast-list, size-only, no-traverse and max-age; all this depends on how you structure your data and might depend on backup destination, just don’t be surprised about monthly AWS bill.

  • if you’re experimenting with rclone on a metal node using crontab you can stream the logs to syslog from your crontab entry for easier debugging (/usr/local/bin/rclone copy -v rest-of-your-command) 2>&1 | logger -t backup-rclone

  • technically you can mount even encrypted drive directly to your local system (even on MacOS) - it can be slightly slow if you try to open ridiculously big bucket/directory

For remote-to-local backups with versioning I also use restic which can be paired with rclone, but I recall running into some edge cases and abandoning the idea.