Trust is good, control is better
Reproducible builds at F-Droid
Author: Nico Alt (CC BY-NC-SA 4.0)
También hay una traducción a Español. Es gibt auch eine deutschsprachige Version.
Free Software is a great thing, because it allows users to check that apps are doing what they pretend to do. But almost nobody compiles apps themselves. Instead, binaries are used which in case of Google Play or Apple’s App Store are built by the apps’ developers or in case of F-Droid come from the project itself. With all stores, users have to trust that those compiled apps only contain what’s published in source code. However, it’s easy to modify apps to include malware before they are published on those platforms. This is where reproducible builds come into play. If an app is built reproducibly, everybody can check whether an app has been modified during its build process. Users then no longer have to trust developers or F-Droid when they install apps from one of these stores. Instead, everybody keen can try to reproduce the app on their own.
The goal of reproducible builds is to “create an independently-verifiable path from source to binary code”. Latter means the code that is installed and executed on users’ devices. Unlike to what you might expect, the same source code does not always lead to the same binary code. Instead, most of the time binaries are slightly modified with each build process in a way that doesn’t change the program’s behavior but makes verifying the binary code impossible. If you’re lucky, it’s enough to just document the exact versions of build tools to make reproducing an app possible. However, sometimes problems can occur that aren’t that easy to solve.
In the past, reproducible builds weren’t well known among developers. Only in recent years, the concept has become more and more popular, especially around free and open source software because it offers a lot of great opportunities. One example is that with reproducible builds projects no longer have to fully trust their build infrastructure and instead can more freely accept hardware donations. F-Droid is a good example: in contrast to the big, commercial app stores, at F-Droid only apps that are free and open source software are offered. There are situations where the original developers of an app make changes to a published app that aren’t documented in its publicly available source code. In order to prevent these, apps at F-Droid get build from source. This is both its greatest strength as its greatest weakness: truly every single app is built from source and therefore F-Droid’s build infrastructure is highly secured, with only Ciaran Gultnieks, who founded the project almost nine years ago, having access to it. That’s why this important and central point within F-Droid’s infrastructure therefore is also its biggest bottleneck which occasionally leads to a significant delay of updates.
This excerpt of a diffoscope comparison of the app OsmAnd~ shows how a hash got appended in former versions of F-Droid’s server which made reproducing an app impossible. The information which got added on the right and which contains the versions of used build tools is important to make an app reproducible.
Plans to improve this situation exist and all of them include reproducible builds as their key feature. One idea which could be done immediately and which promises faster updates for single apps got suggested by the author of this text to the alternative YouTube app NewPipe. They want to implement this idea within the next months. To recap it shortly: if an app is built reproducibly, F-Droid can offer the original .apk file which got signed by its developers. If updates at F-Droid get delayed again for whatever reason, developers can show users a notification within in the app that allows users to upgrade to new versions more quickly. This is possible because the user installed the app that got signed by its developers, not by F-Droid, and therefore doesn’t have to wait until F-Droid has built, signed and included the update in its store. Another idea which should be seen as a long-term plan is to fully decentralize F-Droid’s build infrastructure. With more apps being built in a reproducible manner, more F-Droid contributors could use more computational power in order to intent reproducing apps and later state publicly whether they were successful or not. Once enough official servers agreed upon an app being built reproducibly from its official source code, those apps could be included straight away in the F-Droid store, without having to wait for the current single build server. With those progresses around F-Droid, an app could get into its store that many F-Droid users use nowadays but that hasn’t made it yet: Signal. Despite many attempts to include it, the popular instant messenger is not available in F-Droid since it got renamed from its predecessor TextSecure. The reason is that the organization behind Signal forbids distributing other binaries than the ones signed by them. Because of this, users of Signal have no other option than to trust that the organization has not modified the app’s source code before publishing the official binaries. If Signal would be built reproducibly, everyone including F-Droid could check whether the app has been built straight from its source code and could then include it in F-Droid’s store. The infrastructure to do this at F-Droid is ready. Now it’s down to Signal to make the app build reproducibly.
With Checkey users can see details of the signatures of installed apps. On the left it’s shown that Transportr got signed by F-Droid (CN=FDroid) while Briar got signed by its original developers (O=briarproject.org).
As a user it’s not quite easy yet to see whether an app has been built reproducibly or not. The topic of reproducible builds has become popular only in recent years and because of that there doesn’t yet exist a common opinion on how to display the state of reproducibility to users. At the time of writing, only a handful of apps in F-Droid are being built reproducibly. If a warning of apps not being built reproducibly is shown to users, many users would have to dismiss it in order to keep using the store. To not “burn” the topic, at least for the beginning F-Droid has decided not to show any status indication at all. F-Droid will come back to that once reproducible builds have become more popular and UX designers have come up with “best practices”. For users still interested in finding out more, there are two possible ways to pursue. One is to use the app Checkey which shows information about signatures of installed apps. At the moment, most apps in F-Droid are signed with certificates by F-Droid itself. Apps like these show up in Checkey with the values “CN=FDroid […]” in the column “Issuer”, while apps like Briar which got included with binaries signed by their developers show up with different values, namely “O=briarproject.org”. Another way to find out more about the state of reproducibility of an app is to visit the webpage at verification.f-droid.org. All apps of F-Droid are regularly being re-built by this server which later publishes information on whether an app has been reproduced or not. Unfortunately, it doesn’t mean that being able to reproduce an app leads to immediate inclusion of binaries signed by its original developers. Instead, developers have to get active in order to publish their binaries on F-Droid.
Reproducible builds are also an important topic outside of free software. Just like developers of free software companies that create proprietary software can become targets of attacks ([1], [2], [3]). In the past it happened quite often that computer systems of companies got infected with malware that in turn injected malware into software the company produced while it was being built. To detect attacks like these and to lower the risk to get targeted at all, companies have to spend time to make their builds become reproducible. Once this is done, multiple independent build server can compile a software and compare their results at the end. If all of them produced exactly the same binary, it’s possible to ensure with a high probability that the binary doesn’t include malware. Of course, for this to be true the source code must not contain malware neither. The practice of reproducible builds therefore doesn’t replace but rather extends existing practices like pentesting and code reviews. A totally different improvement takes place in the area of delta updates. With this special form of updates users don’t download the complete file but only the difference to former versions, resulting in dramatically lowered bandwidth requirements. Delta updates have been used in Android for quite some time already and Microsoft uses them as well. Since Windows XP they are known as “express installation files”. If reproducing a program isn’t possible, tiny changes get applied to the binary with every build, resulting in unnecessarily larger updates. Among others this is one of the reason why Google is also interested in reproducible builds and therefore fixed a central bug that made reproducing apps on Android impossible.
It wasn’t much of a problem to reproduce apps on Android until version 3.0.1 of the gradle plugin. With later version reproducing apps failed on different devices because of small changes that got applied to the file resources.arsc. Briar developer Torsten Grote reported this issue to Google in June 2018 when he was trying to get the darknet messenger as the first reproducibly built app into F-Droid. He wasn’t successful since the popular public transit application Öffi found its way first into F-Droid by using an older version of the gradle plugin and thus keeping reproducibility possible. Three months later, Briar also made it into F-Droid because Grote found a workaround by using a special file system called disorderfs. Finally, in February of this year Google developers announced that the bug was fixed and that it’ll get included in next versions of the gradle plugin 3.4 and 3.5. Hopefully, more apps will find their way with those new versions reproducibly built into F-Droid.
Reproducible builds is a topic many different projects work on these days. To only name a few: the cryptocurrency Bitcoin and the Tor project worked on making their binaries build reproducibly in the past and continue to do that to this date. At Bitcoin, multiple independent developers build new versions of the software and compare the results with those of other developers. If they match, the binary gets signed by everybody. Anyone who’s interested in reproducing the programs can find instructions on the projects’ website.
Sad news came from Debian: theoretically the newest version of the Linux distribution, called Buster, has a high amount of reproducibly built software with more than 90%. However, Debian developer Holger Levsen reported in a mail to the project’s mailing list that Buster’s official binaries will only be reproducible to about 54%. The reason for this dramatic drop is that a large part of Debian packages got built before significant improvements in terms of reproducibility of the project’s build tools got implemented. They are currently thinking about doing mass rebuilds of all packages to increase this amount.
Reproducing an app is not always simple and in some cases it can require a lot of time to make it build reproducibly. But by all means, it’s worth it. Until most of the apps in F-Droid are built reproducibly and binaries signed by their developers are included in F-Droid, some more time will pass by. But the groundwork for this has been laid and the tasks that need to be done now mainly consist of fixing tinier bugs and talking with upstream developers to include their binaries in F-Droid.
Many thanks to Bennet Weiß who has massively helped to improve this translation.
Links
F-Droid documentation about the verification server
Interesting for everybody who wants to reproduce apps on their own
F-Droid documentation about reproducible builds
Gives more in-depth information about the topic
Trust, Privacy, and Free Software - F-Droid.org
Related article by eighthave that also talks about reproducible builds