How Syncthing provides secure file syncing without sharing your files with a third party

(Read this article on the blog) Syncthing is an open-source project that implements secure file synchronization between machines. It’s one of many such services, but the more I read about how it works the more I’m amazed how well it solves this quite hard problem without relying on the trust of anything “cloud”. In this […]

(Read this article on the blog)

Syncthing is an open-source project that implements secure file synchronization between machines. It’s one of many such
services, but the more I read about how it works the more I’m amazed how well it solves this quite hard problem without relying on the trust of anything “cloud”.

In this article, you’ll learn about how Syncthing works and how it brings back control over your data to your hand. Hopefully, by the end of this post, you’ll be
as amazed by Syncthing as I am.

File synchronization

Synchronizing folders between computers became mainstream with Dropbox. This is when you put an image into a shared directory and it automatically appears on
the other machine in a few seconds. It’s like magic and it’s extremely convenient when you get the feeling of it.

Dropbox uses a central server to store your files and the clients connect to it. When one client detects a change it uploads it to the Dropbox server. Then if
another client comes online it downloads all changed files.

LaptopPhonephoto.jpgphoto.jpgDropboxsyncsync

This is a convenient setup as it relies on a central server that is globally reachable. The clients don’t need a direct way to communicate which is usually a pain
due to firewalls, NATs, and dynamic IP addresses. The central server solves this problem.

The downside is that this is a machine that Dropbox controls and has access to all your files. Moreover, Dropbox provides the client too, so even if
it implements zero-trust encryption Dropbox can push an update to change that. So you need to trust Dropbox. Maybe it does not look into your files and takes good
care of security. And maybe not, but you can do little beyond trusting it.

This is how most cloud services work. You need to trust the provider, and it is the way of business in many services. But when there are ways to eliminate this
trust they are usually more secure alternatives. I never liked the idea that I’m sending files to a provider and only “company policy” stands in the way of it
abusing them.

Of course, you can use an encrypted filesystem (like eCryptfs) and share only that with Dropbox, but it’s a pain to configure and run. I’ve done that, but it’s
not a solution I’d recommend to anybody.

Encrypted synchronization

The next step I took was Seafile. This works similarly to Dropbox, but there are 3 important differences:

  • First, it supports client-side encryption. This way the server can not look into the files.
  • Equally important, the client is open source so that I can be reasonably sure that the encryption is implemented well.
  • And third, the server is also available as open-source so I can install it on a machine I control

These solve the trust problem. I can run my own server, use the client from third-party repositories (such as the main Ubuntu
repo
) and set a password. This setup doesn’t need
any trust other than the packages used.

LaptopPhonephoto.jpgphoto.jpgSeafileencrypted syncencrypted sync

But running and maintaining my own server is a pain. There are hosted solutions, but using one changes the financial situation. It’s no longer a decision
between trust and no trust but between a free solution and a paid one.

Peer-to-peer synchronization

Syncthing provides p2p synchronization. There is no central server that provides a common connection point for the clients. Why it can work this way is due to
how it implements communication between the peers.

LaptopPhonephoto.jpgphoto.jpgsync

Syncthing implements the TLS protocol, the same that the browser uses when you connect to websites via HTTPS.
This solves the two problems with encrypted communication:

  • It provides encryption with up-to-date algorithms
  • It provides authentication so that the client can be sure that it communicates with the other client and no adversary can listen in

But when you configure HTTPS for your website, you need to prove access to a domain to get a valid certificate. For example, Let’s Encrypt requires that you can
write a file at a specific location, or the Amazon Certificate Manager needs that you put a CNAME record to the domain. How Syncthing can use TLS but without
all these complications?

The browser trusts only a few root certificates but not the ones provided by the website. There is a chain of trust here where a certificate is signed by a
trusted root, so it becomes trusted too. Then another certificate is signed by this trusted one, and so on. If the website can provide a chain of certificates
that can reach a trusted root then the browser will trust this connection. You need to verify domain ownership to get a valid certificate from the issuer. The
web works this way as it’s not realistic to add certificates of websites manually to the trusted set.

In the case of Syncthing, it generates a public-private key pair when you install it. The Device ID which you need to connect two peers is generated from the
public key. Syncthing doesn’t need to verify ownership to a trusted third party because the trust of the other end is established with this exchange of Device IDs.

LaptopPhonePrivate keyPublic keyPrivate keyPublic keyTrustsTrustsSyncthing trust

This solves the authentication part of the encryption. As a result, it does not matter how the two peers communicate they will be the only ones who can access
the files. This is end-to-end encryption done right.

Relaying

An authenticated and encrypted connection between the peers opens the way to use relays, even ones that are not trusted. Syncthing maintains a list of public
relays
that anybody can use, effectively providing a middleman.

LaptopPhonephoto.jpgPrivate keyPublic keyphoto.jpgPrivate keyPublic keyRelayTrustsTrusts

Relaying solves two hard problems. First, since the connection is authenticated it does not matter how the peers find each other. This allows a discovery
service
where both ends register and get the current IP address of the other. With this, you don’t need to provide a static IP address or domain name. When you
share a folder with a device, they will be able to find each other.

And second, a relay allows connections between peers even when they are not accessible to each other, such as when both of them are behind a NAT. They
can both connect to the same relay and use that to forward the communication.

In practice, after you paired the devices using each others’ Device ID, synchronization just happens. You don’t need to worry about connectivity issues or
static addresses. By default, they will use the discovery service to find each other and a relay if they can’t connect directly. This setup is as
easy-to-use as using a central server but without trusting your files with the provider.

Versioning

Syncthing also supports file versioning that defines how one end handles changes to existing files coming
from the other end. You can use simple versioning so that you can recover overwritten (and deleted) files up to a fixed number of changes.

Or you can use the so-called staggered versioning that automatically deletes
versions as they age, leaving fewer and fewer the longer they are stored. This tries to strike a balance between storage cost and recovery.

Keeping modified files is important. Ransomware overwrites files and a file synchronization solution would happily propagate these changes to the other ends.
Without a way to store old versions, you’d lose access to your files even on other devices.

While versioning works for most cases, keep in mind that Syncthing is not a backup solution. It’s a good idea to take a snapshot of the synced folders from
time to time.

Client-side encryption at rest

A newer feature is untrusted devices. This means one end encrypts the files with a password so that even the
other end can’t access them. This feature is still experimental but it opens the way of using any device as a backup.

Just use a good password.

Conclusion

With Syncthing, you get all the benefits of a cloud-based file synchronization service but without the need to trust a third party with your data. This opens
the way to bring a lot of things “back from the cloud” without losing the convenience.

Gmail provides a convenient way to reach your emails from any device. I remember when I used Thunderbird (it’s still
around, btw) to download my emails using POP3. This was before the smartphone era and it wasn’t a priority to get emails to multiple devices and it also meant
that without manually backing them up, my emails were lost when I moved from one computer to another. Now, when I get a new phone it configures itself from the old
one automatically using cloud backups.

File synchronization brings the cloud experience to the local data. I can download my emails on one device, and access them on another because everything is
stored in files that can be made identical over a network.

All that, without privacy concerns. That’s why I’m excited about Syncthing.

Source: Advanced Web Machinery