Automation is one key factor that lets the small F-Droid team provide apps
to millions. The @checkupdates-bot
is a new reworking of an old piece of automation in F-Droid:
checkupdates. This goes through all of the existing apps, checks if they
are set up for automatic updates,
and if so, runs the process to automatically generate a new build entry for
that release. That gets added to
fdroiddata, which is what the
production buildserver processes in order to build and ship apps.
Since the best security is provided by systems that follow the principal of
least
privilege, we
recently reorganized our setup and workflow around that principal so that
the @checkupdates-bot has the least privileges needed to do its job. The
checkupdates process now runs in its own isolated project, separate from
fdroiddata and any other F-Droid project on gitlab.com. It now only
pushes commits to its own dedicated
project, then makes
a merge
request
per app to fdroiddata. Both our automated CI processes and our trusted
human reviewers now handle all updates using the same process as for new
apps.
At the same time, we removed a key bit of cruft in our code:
stats/known_apks.txt. This file was the place that stored the dates when
each app was added to the collection. This file was updated on the
buildserver and maintained in fdroiddata. That information is also in
the index file, so we
switched to
fetching it from there instead. That meant we could remove the last deploy
key in use in fdroiddata. Our operations no longer require any deploy
keys in fdroiddata.
While we were at it, we added some additional checks via the merge
requests. For example, now any time an image file is added or modified, a
CI job checks
whether the image contains any EXIF metadata, which can be used as an
exploit vector. We also added some additional enforcement to make sure
changes to key files go through human review via merge requests.
Security issue as inspiration
About a month ago, @SomberNight reported a
security issue to us
in a confidential issue. We appreciate this detailed report, and also want
to highlight their diligent follow up. In specific situations, the old
setup was leaking the private deploy key which granted access to directly
push commits to fdroiddata. We immediately revoked that key, then removed
all privileges from the @fdroidci user that
was associated with that private key. We also investigated all the leads we
could follow to see if someone had used this key to insert something into
F-Droid. We searched the activity of the @fdroidci user and found no
evidence that unauthorized commits were added.
To be sure, we did some additional investigations. Since checkupdates had
been running as part of the fdroiddata project on gitlab.com, a malicious
app build recipe could have also read the CHECKUPDATES_SSH_DEPLOY_KEY
variable which contained the private key. We checked fdroiddata’s history
for signs of exfiltration and found nothing. We require that apps are built
from source code, and that source code is in a source code management system
like Git. That ensures a local copy with history is retained on our
buildserver. We searched our local copy source code and found no evidence
that any app build processes where trying to exfiltrate the checkupdates
private key.
Do you have more ideas for things to search? Please dig in and let us know
if you find anything suspicious. Working in public means everyone is free
to investigate and come to their own conclusions, and contribute to a more
secure free software ecosystem on Android.