Update: Version 3.5.1 had its issues. First, the version was not tagged, when that was fixed, the build was not reproducible and then, when we ran the same script on the same file a third time … maybe the dice fell lucky? In any case we got the exact same result as what we got from Google Play.

We ran our test script. again which delivered these results:

Results:
appId:          it.airgap.vault
signer:         486381324d8669c80ca9b8c79d383dc972ec284227d65ebfe9e31cad5fd3f342
apkVersionName: 3.5.1
apkVersionCode: 23940
apkHash:        f46de03b62975b57350b9c30975d7fb85e4c9a88e46ca15bc2125fea24a56823

Diff:
Files /tmp/fromPlay_it.airgap.vault_23940/apktool.yml and /tmp/fromBuild_it.airgap.vault_23940/apktool.yml differ
Files /tmp/fromPlay_it.airgap.vault_23940/original/META-INF/MANIFEST.MF and /tmp/fromBuild_it.airgap.vault_23940/original/META-INF/MANIFEST.MF differ
Only in /tmp/fromPlay_it.airgap.vault_23940/original/META-INF: PAPERS.RSA
Only in /tmp/fromPlay_it.airgap.vault_23940/original/META-INF: PAPERS.SF

Revision, tag (and its signature):
object 32c980cd295c3976b5a4350cec30d8b10e00e650
type commit
tag v3.5.1
tagger Mike Godenzi <m.godenzi@papers.ch> 1605788380 +0100

version 3.5.1

Which means the build is reproducible.

Prior script run on the same file

Results:
appId:          it.airgap.vault
signer:         486381324d8669c80ca9b8c79d383dc972ec284227d65ebfe9e31cad5fd3f342
apkVersionName: 3.5.1
apkVersionCode: 23940
apkHash:        f46de03b62975b57350b9c30975d7fb85e4c9a88e46ca15bc2125fea24a56823

Diff:
Files /tmp/fromPlay_it.airgap.vault_23940/apktool.yml and /tmp/fromBuild_it.airgap.vault_23940/apktool.yml differ
Files /tmp/fromPlay_it.airgap.vault_23940/assets/public/index.html and /tmp/fromBuild_it.airgap.vault_23940/assets/public/index.html differ
Only in /tmp/fromBuild_it.airgap.vault_23940/assets/public: main.48c0b1291dc2c9240100.js
Only in /tmp/fromPlay_it.airgap.vault_23940/assets/public: main.cd4034adbb37067a0b90.js
Files /tmp/fromPlay_it.airgap.vault_23940/original/META-INF/MANIFEST.MF and /tmp/fromBuild_it.airgap.vault_23940/original/META-INF/MANIFEST.MF differ
Only in /tmp/fromPlay_it.airgap.vault_23940/original/META-INF: PAPERS.RSA
Only in /tmp/fromPlay_it.airgap.vault_23940/original/META-INF: PAPERS.SF

Revision, tag (and its signature):
object 32c980cd295c3976b5a4350cec30d8b10e00e650
type commit
tag v3.5.1
tagger Mike Godenzi <m.godenzi@papers.ch> 1605788380 +0100

version 3.5.1

which means the build is not verifiable.

Digging deeper we looked at what’s going on and found:

• index.html differs in its reference to the main.*.js
• meld didn’t like comparing those main.*.js consisting of one line with 5,206,447 characters each

After unfolding this obfuscated JS code with js-beautify, the diff became more manageable. Turns out the obfuscator “invented” a different name for one function name replacement and listed the functions in a different order:

The diff became a long list of essentially this:

...
82387c82387
<             var jt = Ht("N+aw"),
---
>             var jt = Ht("AQYT"),
87698c87698
<             var jt = Ht("N+aw"),
---
>             var jt = Ht("AQYT"),
88540c88540
<                 var jt = Ht("N+aw"),
---
>                 var jt = Ht("AQYT"),
90380c90380
<             var Jt = Ht("N+aw"),
---
>             var Jt = Ht("AQYT"),
...

and these function definitions in different lines:

55407,56453d56453
<         "N+aw": function(Qt, Ft, Ht) {
<             (function(Qt) {
<                 ! function(Qt, Ft) {

vs.

21489a21490,22536
>         AQYT: function(Qt, Ft, Ht) {
>             (function(Qt) {
>                 ! function(Qt, Ft) {

We could replace one string with the other in one of the files, move a code block and come to the same result. This does not count as reproducible as nobody can be burdened with these steps but neither did we find a smoking gun. If the code on GitHub is fine then so is the app we got from Google Play.

Lastly we recently introduced a new category for obfuscated apps and will have to move this app there if the problem cannot be resolved with the next release.

Obfuscation/Minification is not a problem as long as the app is reproducible but a diff in obfuscated code makes analysis significantly harder as we had to experience today.

In theory, if only one of 100 attempts of reproducing an app succeeds, succeeding once is proof that the binary is derived from the source, so failing to reproduce one and succeeding once should be good enough. We just won’t re-run many times in the future. The provider has to fix the setup.

Verdict Explained

Reproducible when tested
At the time of this analysis, the app on Google Play was reproducible from the code provided by the developers!

The app can be independently verified. If the provider puts your funds at risk on purpose or by accident, security researchers can see this if they care to look. It also means that inside the company engineers can verify that the release manager is releasing the app based on code known to all engineers on the team. A scammer would have to work under the potential eyes of security researchers. He would have to take more effort in hiding any exploit.

"Reproducible" does not mean "verified". There is good reason to believe that security researchers as of today would not detect very blatant backdoors in the public source code before it gets exploited, much less if the attacker takes moderate efforts to hide it.

To understand why some lines of difference are ok and others not one has to consider how app signing works. Android supports currently 3 signing schemes and in version 1 signing the signature is put inside the application file. As the tester must not have the release signing key, those files necessarily are missing or differ from the version on Google Play. The file "apktool.yml" was never part of the app and is generated by the analysis tool "apktool".