New repository format for faster and smaller updates

We just released version 1.16 of the official F-Droid client app for
Android which includes many radical changes under the hood as well as many
bug fixes for long standing issue. Read on for more details.

Growing repository size

As more and more apps make their way into F-Droid, the official repository
index that includes all apps and their metadata also keeps growing.
Currently, the size of the compressed index is 8 MB which is 33 MB
uncompressed. When updating the repository index, each F-Droid client app
has to download and process those 8MB again and again. This problematic
trend became apparent a long time
ago.

Updating only what has changed

Our repository index is in JSON format and we decided to make use of RFC
7396 JSON Merge Patch to
create much smaller JSON files that only contain the changes since your
F-Droid app last updated. Technically, this works by downloading a new
entry.json file instead of the full
index. This entry file points to the full index in case the app had never
updated before and needs all app metadata anyway. But the entry also
contains pointers to various smaller difference files. The app
automatically picks the right diff and only downloads that much smaller file
instead of the full index. This not only saves bandwidth, it also makes
updating the index much faster as only the minimum amount of data needs to
be downloaded, processed and stored. As of writing, the latest diff is 80
KB compressed which is 241 KB uncompressed which is only 1% of the full
index. Version 1.16 of the official F-Droid client app for Android
supports this new repository format.

Other improvements

The new version has a large number of radical changes under the hood. For
example, the entire database had to be replaced to support the new
difference based repository format. We also used the opportunity to improve
various bit and pieces along the way:

• improved mirror support: all files (e.g. images) now get loaded from
  mirrors reducing the load on the main server
• better support for low RAM devices, because the index now gets streamed
  into the DB instead of loading all of it into memory
• hash verification: The SHA256 hash of all files is now part of the
  repository metadata and gets verified while downloading
• stronger digest algorithm for repository signing: We now use SHA256
  instead of SHA1 for the index signature
• support for downloading repository files via IPFS
• many bug fixes that came out of modernizing ancient code

The new version went through a series of alpha releases with
extended testing to make sure no severe issues make it into the stable
release. It is now considered ready for general use.

Information for repository and client maintainers

Owners of third-party F-Droid repositories can enable the new format by
upgrading to the latest version of
fdroidserver which will
automatically publish the new format in addition to the old ones which we
keep around to support older client apps.

Developers of third-party F-Droid clients are encouraged to adopt the new
format either with their own implementations or by making use of the new
libraries
we are publishing to make using this technology as easy as possible and to
share as much code as possible between different F-Droid implementations.

This work was funded by the FFDW-DVD grant