This is just the first step to getting full Docker support for everywhere
the buildserver is used. This will at least let us replace the hacks that
are separately maintained in fdroid/ci-images-base and
fdroid/ci-images-client.
In order to support Docker, this should be able to operate without ssh,
e.g. using vagrant-communicator-docker. This removes the buildserverid
hack and makes it a provisioner shell script.
There has been a whitespace change in the accessibility.properties
configuration file as part of the openjdk-8-jre-headless
8u302-b08-1~deb9u1 version. As we modified the file, this broke
makebuildserver, asking for confirmation.
allowlist and blocklist are much clearer terms with no cultural baggage.
This changes all "whitelist" references to "allowlist", and all "blacklist"
references to "blocklist".
'ndk_paths' will be automatically filled out from well known sources
like $ANDROID_HOME/ndk-bundle and $ANDROID_HOME/ndk/*. If a required
version is missing in the buildserver VM, it will be automatically
downloaded and installed into the standard $ANDROID_HOME/ndk/
directory. Manually setting it here will override the auto-detected
values. The keys can either be the "release" (e.g. r21e) or the
"revision" (e.g. 21.4.7075529).
https://developer.android.com/studio/projects/configure-agp-ndk#agp_version_41
* sdkmanager installs "ndk;12.3.4567890" into $ANDROID_SDK_ROOT/ndk/
* sdkmanager installs "ndk-bundle" into $ANDROID_SDK_ROOT/ndk-bundle/
This keeps the Long Term Support release and the latest release installed.
r10e was kept in because it needs a special extraction method, since it is
a .bin file, not a .zip. r12b is kept in because it is the old default.
Here is a survey of the NDK versions used in the most recent Builds entry
in each app that uses the NDK:
{'r10e': 6,
'r12b': 93,
'r13b': 4,
'r14b': 5,
'r15c': 7,
'r16b': 14,
'r17b': 4,
'r17c': 7,
'r18b': 9,
'r19c': 17,
'r20': 1,
'r20b': 22,
'r21': 3,
'r21d': 56,
'r21e': 65,
'r22': 9,
'r22b': 15,
'r9b': 1}
#517
import glob
import os
import yaml
try:
from yaml import CSafeLoader as SafeLoader
except ImportError:
from yaml import SafeLoader
ndks = dict()
for f in glob.glob('metadata/*.yml'):
with open(f) as fp:
app = yaml.load(fp, Loader=SafeLoader)
if app.get('Disable'):
continue
build = app.get('Builds', [])[-1]
if build.get('disabled'):
continue
ndk = build.get('ndk')
if ndk and ndk[1] == '9':
print(f, build)
elif ndk and int(ndk[2:3]) < 18:
print(f, build)
if ndk:
print(f, ndk)
if ndk not in ndks:
ndks[ndk] = 0
ndks[ndk] += 1
import pprint
pprint.pprint(ndks)
!822
The gpg dependencies are complicated. The previous setup was always
failing with:
The following information may help to resolve the situation:
The following packages have unmet dependencies:
gpg : Depends: gpgconf (= 2.2.20-1~bpo9+1)
Depends: libassuan0 (>= 2.5.0) but 2.4.3-2 is to be installed
Depends: libgpg-error0 (>= 1.35) but 1.26-2 is to be installed
Thanks to @izzysoft to spotting this.
Up until now, the buildserver has not included androguard. Since a
good version of androguard (v3.3.3+) is included in stretch-backports
and the buildserver is already setup to use stretch-backports, this
sets up the buildserver with androguard.
closes#627
https://gitlab.com/eighthave/fdroiddata/-/jobs/611438997
Otherwise it fails to install:
```
gpg : Depends: gpgconf (= 2.2.20-1~bpo9+1)
Depends: libassuan0 (>= 2.5.0) but 2.4.3-2 is to be installed
Depends: libgpg-error0 (>= 1.35) but 1.26-2 is to be installed
```
* 7e8c244473
* !769
The basebox currently provides sudo, but that may not always be the case.
This makes the sudo dependency explicit, so that this provisioning script
can also be used in other settings, like GitLab CI.
Right now, the only thing that Briar needs in sudo: in order to be built
reproducibly is `apt-get install disorderfs`. This includes it in the
standard image to make it easy to spread that technique to many builds.
@grote
`pip install` is a relatively common thing for app builds to use in sudo=,
so it should be built into the buildserver. This then builds in the trust
relationship to pypi, since it comes from Debian rather than some random
pip download/install process.
This expands the gradle wrapper shell script used by the buildserver for
usage outside the buildserver environment. It also allows downloading
whitelisted versions of gradle if they are not yet deployed to the
buildserver by simply upsating the copy of fdroidserver (in contrast to
having to reprovision the whole buildserver).
We first move the buildserver/gradle shell script to the repo root
as gradlew-fdroid, as it's an fdroid specific gradle wrapper.
We also now sync it inside the build VM before each build.
We then add a list of whitelisted gradle distributions taken from the
makebuildserver script.
The script additionally now reads two env vars which tell it where to
expect installed versions of gradle and where it might store downloaded
gradle .zip files. Both of those are configurable from config.py. As the
first should normally just be a subdir of the second it's not exposed in
the example config.py but only used by the buildserver config.py.
Default config now uses this internal gradle wrapper but a path to a
custom wrapper or specific gradle distribution can still be set from
config.py.
Closesfdroid/fdroidserver#98
Ref: fdroid/fdroidserver#370
When a new minor version of an NDK is released, it replaces an older one,
e.g. r16 with r16b (see commit 6f295cb). But old NDK package remains in
the cache and provisioning script unpacks it too as it matches the mask.
Fix NDK provisioning to unzip only while-listed versions.
We should be replacing all our custom git shell commands with python3-git,
since it is a common library for doing that. It will receive a lot more
attention and maintenance than our code for doing it. For example, we
should not ever use shell=True, since that opens up a lot of security
risks.
The currently included Qt has known security issues and is outdated. This
can now be replaced by downloading and installing the Qt installer using
the sudo= build field. @relan's provisioner system will also replace this
once that's done. There are only two apps that currently use the Qt stuff:
* csd.qtproject.minesweeper
* org.openorienteering.mapper
