1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-05 06:50:10 +01:00
Commit Graph

448 Commits

Author SHA1 Message Date
Hans-Christoph Steiner
7976ecc12a buildserver: mark fdroidserver deps as manual
This lets build steps use `apt-get autoremove` and still have a working
fdroidserver.
2022-10-25 19:48:27 +02:00
Hans-Christoph Steiner
817a156ea5 buildserver: strip extraneous packages installed in Vagrant base box
The official Debian Vagrant box seems to be a "batteries included" kind of
thing.  The buildserver should be as small as possible, so this removes all
the extraneous packages I could find in the Vagrant basebox.
2022-10-25 19:47:31 +02:00
Hans-Christoph Steiner
1dd480405e
buildserver: let vagrant package handle insecure private key
This script was overwriting the provisioning that `vagrant package` does
already, and breaking `vagrant ssh`.  It should have been removed in !1099

closes #990
    fdroid-bootstrap-buildserver#12
    !1012
    !1099
c6f5956537
2022-10-20 17:42:53 +02:00
Hans-Christoph Steiner
9089750277 buildserver: show informative message if Vagrant is not configed 2022-10-13 13:56:56 +00:00
Hans-Christoph Steiner
bdc10caa57 add missing build-tools and platforms 2022-10-12 20:57:55 +02:00
Hans-Christoph Steiner
daa5ad152e buildserver: install default SDK packages using fdroid/sdkmanager
fdroid/sdkmanager provides a root of trust to verify all the packages it
downloads, so it fully replaces what makebuildserver was doing.

closes #927
2022-10-12 20:57:55 +02:00
Jochen Sprickerhof
300278e7c2 remove transitional apt-transport-https package 2022-10-12 18:57:23 +00:00
Hans-Christoph Steiner
fd9b911a42 Dockerfile: apt-mark manual fdroidserver dependencies
Then apt-get does not warn about unneeded deps that could be autoremoved.
2022-10-12 18:57:23 +00:00
Hans-Christoph Steiner
759fa3b40a
buildserver: fix apksigner install
Adding /bullseye-backports confuses things apparently.  Without it, apt
will look in bullseye-backports automatically, including for deps.  But it
will not look for deps in bullseye-backports if /bullseye-backports is used.

!1205
2022-10-11 16:23:45 +02:00
linsui
47cbc47346 Pre-install git-svn and mercurial 2022-10-11 13:31:51 +00:00
Hans-Christoph Steiner
b58eaa224b buildserver: include apksigner for Binaries: verification
Verification of Binaries: happens in the buildserver VM first, so it needs
a matching version of apksigner as the rest of the process.  This fixes:
https://gitlab.com/fdroid/fdroidserver/-/jobs/3154772374
2022-10-11 12:03:55 +00:00
Hans-Christoph Steiner
e192eb4a32
buildserver: prevent fdroidserver deps from being purged in Docker 2022-10-11 10:52:54 +02:00
Hans-Christoph Steiner
a0ea27632e buildserver: use HTTPS for security.debian.org
It is now officially supported:
https://guardianproject.info/2021/12/08/debian-over-https/
2022-10-10 21:22:59 +02:00
Hans-Christoph Steiner
c6f5956537 install Vagrant insecure_private_key into buildserver box
Vagrant uses the "insecure private key" to establish the first SSH
connection to a new VM based on a clean public box.  In theory, the
`vagrant package` command should do that automatically.  This process
is still using custom code instead of `vagrant package`, hence this
script.

This public key can be generated on any Vagrant install using:
  ssh-keygen -y -f  ~/.vagrant.d/insecure_private_key

https://www.vagrantup.com/docs/vagrantfile/ssh_settings#config-ssh-private_key_path
fdroid-bootstrap-buildserver#12
!1012
!1099
2022-10-10 21:22:59 +02:00
Jochen Sprickerhof
046d3c8dcf Use ssh_config.d 2022-10-10 21:22:59 +02:00
Jochen Sprickerhof
c208582e75 Drop old java_paths 2022-10-10 21:22:59 +02:00
Hans-Christoph Steiner
dbaa3f6ec2 Stop installing ConstraintLayout
sdkmanager does not work anymore with Java 11 and these can be installed in
the app's build metadata.
2022-10-10 21:22:59 +02:00
Jochen Sprickerhof
f03e812a13 Drop obsolete apt-transport-https 2022-10-10 21:22:59 +02:00
Jochen Sprickerhof
d762f423ce Install patch and rsync (required by fdroidserver) 2022-10-10 21:22:59 +02:00
Jochen Sprickerhof
cb2c3a34bb remove hack to disable GNOME.Accessibility for minimal deps
This is commented out by default on all OpenJDK packages since Java 8.
https://gitlab.com/fdroid/fdroidserver/-/merge_requests/1012?diff_id=263085985#note_726457435

root@sid:/# grep Accessibility /etc/java-*-openjdk/accessibility.properties
/etc/java-11-openjdk/accessibility.properties:#assistive_technologies=org.GNOME.Accessibility.AtkWrapper
/etc/java-17-openjdk/accessibility.properties:#assistive_technologies=org.GNOME.Accessibility.AtkWrapper
/etc/java-18-openjdk/accessibility.properties:#assistive_technologies=org.GNOME.Accessibility.AtkWrapper
/etc/java-8-openjdk/accessibility.properties:assistive_technologies=org.GNOME.Accessibility.AtkWrapper
2022-10-10 21:22:59 +02:00
FestplattenSchnitzel
be5aefd3ee Install required packages for fdroidserver automatically 2022-10-10 21:22:59 +02:00
FestplattenSchnitzel
cc641b5470 Stop adding i386 architecture to dpkg 2022-10-10 21:22:59 +02:00
FestplattenSchnitzel
df46eb86c3 Upgrade Buildserver VM
Use Vagrant boxes built with cloud-team/debian-vagrant-images instead of fdroid/basebox,
Use Debian Bullseye (11) instead of Debian Stretch (9)
2022-10-10 21:22:59 +02:00
Hans-Christoph Steiner
53a507d7c4 buildserver: default to HTTPS for apt sources
Debian Docker images will soon default to HTTPS for apt sources, so force
it now:
https://github.com/debuerreotype/docker-debian-artifacts/issues/15
2022-09-08 18:55:17 +02:00
linsui
5472cf88b8 Update preinstalled NDK to r23c 2022-06-14 11:03:42 +00:00
Hans-Christoph Steiner
6ef60f0d6b
scanner: include dexdump in buildserver for APK analysis
This scanner feature is not yet ready for the production buildserver but it
is already useful in CI.
2022-05-19 15:45:26 +02:00
Licaon_Kter
e678df14ce
buildserver: remove old LTS NDK and only pre-install current LTS 2022-04-26 10:25:31 +02:00
Hans-Christoph Steiner
c6fddebecd buildserver: disable flutter dev/build analytics globally
The Flutter dev tools includes opt-out analytics, with a config option to
opt out.  We can at least try to disable them in the buildserver. That
config option is unfortunately not even guaranteed to work:
https://github.com/flutter/flutter/issues/19304
https://github.com/flutter/flutter/issues/45369

issuebot#21
issuebot!49
https://github.com/flutter/flutter/issues/73657
2022-04-21 15:38:48 +02:00
linsui
3f62e0ebde update preinstalled ndk to r23b 2022-02-10 16:47:03 +08:00
Hans-Christoph Steiner
5d09e70330
buildserver: give names to all provisioners 2022-01-13 21:22:23 +01:00
Hans-Christoph Steiner
97553c516f
provision-android-ndk: let script work when no NDKs will be installed
The other form of find gave:

# find $NDK_BASE -type f -executable -print0 | xargs -0 chmod a+x
chmod: missing operand after ‘a+x’
Try 'chmod --help' for more information.
2022-01-13 21:22:21 +01:00
Hans-Christoph Steiner
5514000376
buildserver: include packages in Dockerfile that Vagrant boxes have
The goal is to have the Docker base image and the Vagrant base box have the
same setup.
2022-01-13 21:22:20 +01:00
Hans-Christoph Steiner
7535d75b40
buildserver: run provision scripts in Dockerfile
fdroidserver#119
2022-01-13 21:22:15 +01:00
Hans-Christoph Steiner
f20d3d249a
add Dockerfile to build a docker buildserver image
This is just the first step to getting full Docker support for everywhere
the buildserver is used.  This will at least let us replace the hacks that
are separately maintained in fdroid/ci-images-base and
fdroid/ci-images-client.
2022-01-13 21:21:17 +01:00
Hans-Christoph Steiner
4e310e3a59
buildserver: setup-env-vars: add var for ~vagrant for CI jobs
The CI jobs need to do a little setup based on the 'vagrant' user's $HOME.
2022-01-11 11:23:44 +01:00
Hans-Christoph Steiner
57447f18e1
buildserver: move buildserverid to provisioner
In order to support Docker, this should be able to operate without ssh,
e.g. using vagrant-communicator-docker.  This removes the buildserverid
hack and makes it a provisioner shell script.
2022-01-11 11:23:36 +01:00
Hans-Christoph Steiner
319bf47a6b enable intel-android-extra-license on buildserver, it is DFSG-free
It is a BSD-3-clause:
38f9564932/android/repository/addon.xml (L568)
2021-12-07 10:24:27 +01:00
Jochen Sprickerhof
ae9a1df1aa Add newline in ssh_config
Otherwise both options end on the same line and are not used.
2021-11-04 09:10:57 +00:00
Felix C. Stegerman
779a592ea0
[buildserver] run update again after upgrade 2021-08-25 14:51:20 +02:00
Jochen Sprickerhof
e1a67c9d97 [makebuildserver] run dpkg with --force-confdef
There has been a whitespace change in the accessibility.properties
configuration file as part of the openjdk-8-jre-headless
8u302-b08-1~deb9u1 version. As we modified the file, this broke
makebuildserver, asking for confirmation.
2021-08-23 22:48:14 +02:00
Hans-Christoph Steiner
78d37bb13b switch to allow/block list terminology throughout code base
allowlist and blocklist are much clearer terms with no cultural baggage.
This changes all "whitelist" references to "allowlist", and all "blacklist"
references to "blocklist".
2021-06-18 18:26:50 +00:00
Hans-Christoph Steiner
28a4f9481a
provision-android-ndk: do all configuration via args
This makes it so it is easy to reuse this script in other contexts, like
in Docker, GitLab CI, etc.
2021-06-01 14:33:41 +02:00
Hans-Christoph Steiner
adafd4560c
provision-android-sdk: ensure ANDROID_HOME parent dir is accessible by all
https://gitlab.com/fdroid/fdroidserver/-/merge_requests/926#note_586778883
2021-05-28 10:24:55 +02:00
Hans-Christoph Steiner
7a1d236c8d
only support zipballs in NDK provisioning
Since I discovered there is an r10e zipball, this can now get all NDKs
in zipball form.
fdroid/android-sdk-transparency-log@447fea86e7

closes #902
2021-05-28 09:13:36 +02:00
Hans-Christoph Steiner
9f77044d0d
auto-detect NDKs installed in standard paths
'ndk_paths' will be automatically filled out from well known sources
like $ANDROID_HOME/ndk-bundle and $ANDROID_HOME/ndk/*.  If a required
version is missing in the buildserver VM, it will be automatically
downloaded and installed into the standard $ANDROID_HOME/ndk/
directory.  Manually setting it here will override the auto-detected
values.  The keys can either be the "release" (e.g. r21e) or the
"revision" (e.g. 21.4.7075529).

https://developer.android.com/studio/projects/configure-agp-ndk#agp_version_41
* sdkmanager installs "ndk;12.3.4567890" into $ANDROID_SDK_ROOT/ndk/
* sdkmanager installs "ndk-bundle" into $ANDROID_SDK_ROOT/ndk-bundle/
2021-05-28 09:13:34 +02:00
Hans-Christoph Steiner
93145a43fb buildserver: use standardized paths for the NDKs
This should help with reproducibility since some tools like Python still
include the build paths in the binaries.
https://gitlab.com/fdroid/fdroidserver/-/merge_requests/919#note_578180986

The default ANDROID_SDK_ROOT base dir of /opt/android-sdk is hard-coded in
buildserver/Vagrantfile.  The $ANDROID_HOME/ndk subdir is where Android
Studio will install the NDK into versioned subdirs.
https://developer.android.com/studio/projects/configure-agp-ndk#agp_version_41
2021-05-26 09:21:52 +02:00
Hans-Christoph Steiner
bb77d7a6d2 buildserver: standardize SDK install location /opt/android-sdk
_/opt/android-sdk_ was chosen for a number of reasons:
* _/opt_ is [standardized][1] for packages like the Android SDK, which has its own directory layout.
* _android-sdk_ is used rather than the upstream directory name from the ZIP (e.g. _android-sdk-linux_) so that the path is the same on all platforms.
* On platforms without official _/opt_ support ([macOS][2], [Windows][3], [FreeBSD][4], etc.), it does not conflict with any existing system directory.

[1]: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s13.html
[2]: https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html#//apple_ref/doc/uid/TP40010672-CH2-SW7
[3]: https://en.wikipedia.org/wiki/Directory_structure#Windows_10
[4]: https://www.freebsd.org/cgi/man.cgi?query=hier&sektion=7&format=html
2021-05-25 17:30:32 +02:00
Hans-Christoph Steiner
07da062a34 buildserver: fix ssh BatchMode config 2021-05-25 17:21:26 +02:00
Hans-Christoph Steiner
ec2cace222
buildserver: trim pre-installed NDK list down to the bare minimum
This keeps the Long Term Support release and the latest release installed.
r10e was kept in because it needs a special extraction method, since it is
a .bin file, not a .zip.  r12b is kept in because it is the old default.
Here is a survey of the NDK versions used in the most recent Builds entry
in each app that uses the NDK:

{'r10e': 6,
 'r12b': 93,
 'r13b': 4,
 'r14b': 5,
 'r15c': 7,
 'r16b': 14,
 'r17b': 4,
 'r17c': 7,
 'r18b': 9,
 'r19c': 17,
 'r20': 1,
 'r20b': 22,
 'r21': 3,
 'r21d': 56,
 'r21e': 65,
 'r22': 9,
 'r22b': 15,
 'r9b': 1}

#517


import glob
import os
import yaml

try:
    from yaml import CSafeLoader as SafeLoader
except ImportError:
    from yaml import SafeLoader

ndks = dict()
for f in glob.glob('metadata/*.yml'):
    with open(f) as fp:
        app = yaml.load(fp, Loader=SafeLoader)
        if app.get('Disable'):
            continue
        build = app.get('Builds', [])[-1]
        if build.get('disabled'):
            continue
        ndk = build.get('ndk')
        if ndk and ndk[1] == '9':
            print(f, build)
        elif ndk and int(ndk[2:3]) < 18:
            print(f, build)
        if ndk:
            print(f, ndk)
            if ndk not in ndks:
                ndks[ndk] = 0
            ndks[ndk] += 1

import pprint

pprint.pprint(ndks)
2021-05-25 17:06:28 +02:00
relan
6d9410597b Upgrade NDK r22 to r22b 2021-04-07 12:12:04 +03:00