Behind the scenes of F-Droid is a giant pile of automation to manage the
process of building thousands of apps from source. This means checking out
thousands of source repos, checking them all for updates, building and new
releases, and securely signing them en masse. All builds are run in a fresh
virtual machine guest instance known as the
buildserver.
All Gradle binaries and Android SDK packages are verified against our
public
logs of observed
SHA-256 checksums. The transparency log processes also verify against
upstream’s public checksums.
Our setup runs on Debian almost exclusively. Debian is a leader in free
software, rock solid servers, and reproducible builds. That makes it a
natural home for F-Droid. We also work to ensure we maintain the packages
we use, and build our processes on top of Debian packages. That means we
share the maintenance with anything that uses Debian. It may seem like more
work to give back, but our experience is that it pays off in the long run.
The F-Droid community is able to maintain many things with a small team.
Another example of this is this website itself: it is built using Jekyll
packages that are all in Debian.
If you have an app on f-droid.org, you might have noticed that all builds
happen on a 5 year old Debian release: stretch. We are in the midst of a
big effort to upgrade to
the latest bullseye release right now. This is not just a simple apt-get, we are also taking this opportunity to overhaul the build process
upgrade
so that app builds work with a relatively plain Debian install as the base
OS. We have to provide a platform to build thousands of apps, so we cannot
just upgrade the base image as often as we like. Some apps need the latest,
greatest. Other apps need the ancient, stable base OS. This change means
that the metadata
contains as much of the build logic as possible, so that the app maintainer
has control over all the steps. To achieve this, as much as possible is
being stripped out of the from the buildserver base image.
We have considered offering a selection of base images. This is a possible
solution, but it is not easy as just using any available Docker image. Only
base images that are guaranteed to be free software are appropriate. Just
pointing to any Docker image would open up the possibility of proprietary
build dependencies, since it is not possible to automatically check whether
any Docker image is 100% free software. Using a selection of pre-approved
base boxes could solve that. Keep in mind, this is more complex than with
GNU/Linux distros since Android apps are cross-compiled. GNU/Linux distros
build their packages on their own OS. During builds, Debian does not even
allow network access since all dependencies need to come from Debian
packages. That level of verification is a goal of F-Droid, and
Maven’s
work towards a reproducible Maven
Central ecosystem helps a lot.
Since CalyxOS builds in F-Droid by default, Calyx
Institute also want to ensure that F-Droid is
running smoothly, and that app developers are happy. I would like to thank
Calyx Institute for sponsoring 42 hours a month of my time to work on making
our build infrastructure run smoothly. Additionally, I will be working on
improving the automation of the signing process. Our signing process is
currently 100% offline. While this is nice for security, it does slow down
the release process. With modern hardware security modules and server
setups, it is possible to have good security without being 100% offline.
Having signing automation then opens up possibilities for parallelizing the
whole process, including running multiple app builds, and also, running the
main steps of building, index generating, and signing all in parallel. This
work will be incrementally deployed as each bit is finished. So be patient,
and you will notice releases happening faster and faster!
