Google Drive (GDrive) and other cloud storage alternatives such as Dropbox and Microsoft Ondrive all have the serious drawback of keeping one's information in a third party cloud repository. Privacy and security are generally compromised this way, even when paying for storage (as opposed to having an advertising model, which is worse in many ways).
The challenge is to have an equally robust service that can effectively, and efficiently (regarding resource requirements) sychronize files across multiple devices. Remain on our own devices. And remain open source. In our case we have three different operating systems on four devices to support:
- ChromeOS (which runs Android apps)
- Debian Linux stable and testing
Options such as OwnCloud don't work because of the high overhead needed to get the services to run, in terms of memory and processing on a server.
- Syncthing for File Synchronization
- Install Syncthing on Debian stable
- Allow Ports on Debian
- Install Syncthing on Android
- Configure Syncthing to Turn on Automatically
- Some Issues with Synchronizing
- Synchronization vs. SFTP
- Syncthing Autostart
- Syncthing Version Control (very-lite)
- Value of Syncthing for the Enduser
- Real World Replacement of Dropbox
Syncthing for File Synchronization
File synchronization is not backup, though with versioning there is a sort of backup-lite going on. Syncthing versioning doesn't really work as expected by the name, so that is really backup-very-lite.
Syncthing is available from repositories and directly from Github. There are ports and other configuration issues to enable for routing. There is also an Android app, so that is what will be used on Android and ChromeOS.
Install Syncthing on Debian stable
Note: for Debian testing there is no need to add a repository.
Add PGP key
curl -s https://syncthing.net/release-key.txt | sudo apt-key add -
Add the "stable" channel to your APT sources:
echo "deb https://apt.syncthing.net/ syncthing stable" | sudo tee /etc/apt/sources.list.d/syncthing.list
sudo apt update
sudo apt install syncthing
Edit the .xml file
sudo nano ~/.config/syncthing/config.xml
127.0.0.1 address to
0.0.0.0 to enable access from anywhere
Allow Ports on Debian
Had to punch holes through lightsail networking and also add rules for UFW
- https://18.104.22.168:8384/ (server)
- http://127.0.0.1:8384/ (workstation)
Install Syncthing on Android
For Android and ChromeOS devices, install Syncthing for Android from the Play Store.
Note that the best approach is to share out something like
/syncthing-photos as a one-way share, and root that to the photos directory (either the local device or MicroSD card), and then also create an inbound two-way sync for a subset of critical documents.
For devices with more storage, additional inbound two-way shares could be added to provide greater access to one's documents.
For example on an Android device with a MicroSD card, the photos are shared under
/storage/CE6A-1402/DCIM/Camera. Note that the best approach is then a manual copy/paste of the images to a better archived directory structure that is then synced to a backup server, and a manual image deletion on the Android device, on a monthly basis.
For an inbound, two-way sync to the Android device for critical (editable) files, the directory is:
/storage/emulated/0/Download. Note that this is the same directory that is used in ChromeOS for inbound, two-way shares.
Configure Syncthing to Turn on Automatically
For Debian/LXDE the simple approach is to add an item to the autostart menu
- Menu > Preferences > Default Applications for LXSession > Autostart
See also Starting Syncthing Automatically
The Android syncthing app has a toggle to support auto-on functionality as needed.
Some Issues with Synchronizing
The main thing is to think out one's synchronization policies and plans. One-way synchronization, two-way sinchronization, master and slave device replication, etc. There are many options. Some files one will want to keep everywhere, with version control. Other files one will want only in one or two locations (large files/repositories).
The best approach is to partition into folders so that different folders contain different content that will be sychronized differently. Some examples:
- Images/Photos folder on a mobile device
- Should be synchronized but also allow for repository of more images on a backup location.
- Workflow: sync mobile folder to desktop. On desktop, move images to a second folder (removing them from mobile via synchronization), and then have the second folder synchronized to a server. That server folder can have SFTP for remote access and also provide two-way synchronization back to the desktop for things such as editing images that are on a web server.
It is important to have a manual workflow as well (or semi-automated) so that things are easier to manage.
Synchronization vs. SFTP
Synchronization is useful, but is not a replacement for SFTP which should be seen as on-demand push/pull. For example, a large repository can be synchronized between two larger-capacity devices (e.g, Debian server and Debian workstation), but also allow access via SFTP for smaller-capacity devices (ChromeOS/Android).
For a desktop, add to Startup Applications
/usr/bin/syncthing -no-browser -home="/home/user/.config/syncthing"
For a server, create a service, or something like that.
Syncthing Version Control (very-lite)
There are many different version control options for Syncthing, depending on what versioning scheme and how much storage will be set aside for it. Basically, however, this version of version control/backup is total crap since it only makes a backup on synching to another machine (e.g., if the source is overwritten by a downstream sync, then the source keeps a copy of the original before overwriting. This means if working on a single original source there is no version control/backup.
Basically, the most reasonable option for version control, in this situation of a severe lack, is probably an automated git versioning.
Value of Syncthing for the Enduser
Ultimately, Syncthing lets the enduser take full control over their data on their devices in terms of files that are synchronized with other devices. Along with SFTP on a server, and possibly something like AWS S3 and Glacier, it appears to provide a useful protocol, gui admin console, and applications that can do everything that GDrive/Dropbox/OneDrive offer in terms of synchronization. Since disk space is already something that can be managed at the level of S3/Glacier and local devices, it provide a key element in a resource-efficient, open source package.
Real World Replacement of Dropbox
I've got syncthing running on four devices with the main desktop (Debian testing) as the primary repository. Nearly all of the reposed data is syncronized to a Debian stable server running on AWS lightsail. This is a web server with 80gb of storage ($20/month), and my data consists of approximately 90,000 files, 10,000 directories, and 40gb of storage. The third device is my Android phone which syncs out photos, and inbound syncs a lightweight directory of editable files, which allow for reading/editing those files on-the-go. The fourth device is a Chromebook convertible which has the least storage of all devices, but nevertheless is able to handily inbound sync about 5gb of data and allows for most of the working directories on active projects.