1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-05 06:50:10 +01:00
Commit Graph

423 Commits

Author SHA1 Message Date
Hans-Christoph Steiner
6ef60f0d6b
scanner: include dexdump in buildserver for APK analysis
This scanner feature is not yet ready for the production buildserver but it
is already useful in CI.
2022-05-19 15:45:26 +02:00
Licaon_Kter
e678df14ce
buildserver: remove old LTS NDK and only pre-install current LTS 2022-04-26 10:25:31 +02:00
Hans-Christoph Steiner
c6fddebecd buildserver: disable flutter dev/build analytics globally
The Flutter dev tools includes opt-out analytics, with a config option to
opt out.  We can at least try to disable them in the buildserver. That
config option is unfortunately not even guaranteed to work:
https://github.com/flutter/flutter/issues/19304
https://github.com/flutter/flutter/issues/45369

issuebot#21
issuebot!49
https://github.com/flutter/flutter/issues/73657
2022-04-21 15:38:48 +02:00
linsui
3f62e0ebde update preinstalled ndk to r23b 2022-02-10 16:47:03 +08:00
Hans-Christoph Steiner
5d09e70330
buildserver: give names to all provisioners 2022-01-13 21:22:23 +01:00
Hans-Christoph Steiner
97553c516f
provision-android-ndk: let script work when no NDKs will be installed
The other form of find gave:

# find $NDK_BASE -type f -executable -print0 | xargs -0 chmod a+x
chmod: missing operand after ‘a+x’
Try 'chmod --help' for more information.
2022-01-13 21:22:21 +01:00
Hans-Christoph Steiner
5514000376
buildserver: include packages in Dockerfile that Vagrant boxes have
The goal is to have the Docker base image and the Vagrant base box have the
same setup.
2022-01-13 21:22:20 +01:00
Hans-Christoph Steiner
7535d75b40
buildserver: run provision scripts in Dockerfile
fdroidserver#119
2022-01-13 21:22:15 +01:00
Hans-Christoph Steiner
f20d3d249a
add Dockerfile to build a docker buildserver image
This is just the first step to getting full Docker support for everywhere
the buildserver is used.  This will at least let us replace the hacks that
are separately maintained in fdroid/ci-images-base and
fdroid/ci-images-client.
2022-01-13 21:21:17 +01:00
Hans-Christoph Steiner
4e310e3a59
buildserver: setup-env-vars: add var for ~vagrant for CI jobs
The CI jobs need to do a little setup based on the 'vagrant' user's $HOME.
2022-01-11 11:23:44 +01:00
Hans-Christoph Steiner
57447f18e1
buildserver: move buildserverid to provisioner
In order to support Docker, this should be able to operate without ssh,
e.g. using vagrant-communicator-docker.  This removes the buildserverid
hack and makes it a provisioner shell script.
2022-01-11 11:23:36 +01:00
Hans-Christoph Steiner
319bf47a6b enable intel-android-extra-license on buildserver, it is DFSG-free
It is a BSD-3-clause:
38f9564932/android/repository/addon.xml (L568)
2021-12-07 10:24:27 +01:00
Jochen Sprickerhof
ae9a1df1aa Add newline in ssh_config
Otherwise both options end on the same line and are not used.
2021-11-04 09:10:57 +00:00
Felix C. Stegerman
779a592ea0
[buildserver] run update again after upgrade 2021-08-25 14:51:20 +02:00
Jochen Sprickerhof
e1a67c9d97 [makebuildserver] run dpkg with --force-confdef
There has been a whitespace change in the accessibility.properties
configuration file as part of the openjdk-8-jre-headless
8u302-b08-1~deb9u1 version. As we modified the file, this broke
makebuildserver, asking for confirmation.
2021-08-23 22:48:14 +02:00
Hans-Christoph Steiner
78d37bb13b switch to allow/block list terminology throughout code base
allowlist and blocklist are much clearer terms with no cultural baggage.
This changes all "whitelist" references to "allowlist", and all "blacklist"
references to "blocklist".
2021-06-18 18:26:50 +00:00
Hans-Christoph Steiner
28a4f9481a
provision-android-ndk: do all configuration via args
This makes it so it is easy to reuse this script in other contexts, like
in Docker, GitLab CI, etc.
2021-06-01 14:33:41 +02:00
Hans-Christoph Steiner
adafd4560c
provision-android-sdk: ensure ANDROID_HOME parent dir is accessible by all
https://gitlab.com/fdroid/fdroidserver/-/merge_requests/926#note_586778883
2021-05-28 10:24:55 +02:00
Hans-Christoph Steiner
7a1d236c8d
only support zipballs in NDK provisioning
Since I discovered there is an r10e zipball, this can now get all NDKs
in zipball form.
fdroid/android-sdk-transparency-log@447fea86e7

closes #902
2021-05-28 09:13:36 +02:00
Hans-Christoph Steiner
9f77044d0d
auto-detect NDKs installed in standard paths
'ndk_paths' will be automatically filled out from well known sources
like $ANDROID_HOME/ndk-bundle and $ANDROID_HOME/ndk/*.  If a required
version is missing in the buildserver VM, it will be automatically
downloaded and installed into the standard $ANDROID_HOME/ndk/
directory.  Manually setting it here will override the auto-detected
values.  The keys can either be the "release" (e.g. r21e) or the
"revision" (e.g. 21.4.7075529).

https://developer.android.com/studio/projects/configure-agp-ndk#agp_version_41
* sdkmanager installs "ndk;12.3.4567890" into $ANDROID_SDK_ROOT/ndk/
* sdkmanager installs "ndk-bundle" into $ANDROID_SDK_ROOT/ndk-bundle/
2021-05-28 09:13:34 +02:00
Hans-Christoph Steiner
93145a43fb buildserver: use standardized paths for the NDKs
This should help with reproducibility since some tools like Python still
include the build paths in the binaries.
https://gitlab.com/fdroid/fdroidserver/-/merge_requests/919#note_578180986

The default ANDROID_SDK_ROOT base dir of /opt/android-sdk is hard-coded in
buildserver/Vagrantfile.  The $ANDROID_HOME/ndk subdir is where Android
Studio will install the NDK into versioned subdirs.
https://developer.android.com/studio/projects/configure-agp-ndk#agp_version_41
2021-05-26 09:21:52 +02:00
Hans-Christoph Steiner
bb77d7a6d2 buildserver: standardize SDK install location /opt/android-sdk
_/opt/android-sdk_ was chosen for a number of reasons:
* _/opt_ is [standardized][1] for packages like the Android SDK, which has its own directory layout.
* _android-sdk_ is used rather than the upstream directory name from the ZIP (e.g. _android-sdk-linux_) so that the path is the same on all platforms.
* On platforms without official _/opt_ support ([macOS][2], [Windows][3], [FreeBSD][4], etc.), it does not conflict with any existing system directory.

[1]: https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch03s13.html
[2]: https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html#//apple_ref/doc/uid/TP40010672-CH2-SW7
[3]: https://en.wikipedia.org/wiki/Directory_structure#Windows_10
[4]: https://www.freebsd.org/cgi/man.cgi?query=hier&sektion=7&format=html
2021-05-25 17:30:32 +02:00
Hans-Christoph Steiner
07da062a34 buildserver: fix ssh BatchMode config 2021-05-25 17:21:26 +02:00
Hans-Christoph Steiner
ec2cace222
buildserver: trim pre-installed NDK list down to the bare minimum
This keeps the Long Term Support release and the latest release installed.
r10e was kept in because it needs a special extraction method, since it is
a .bin file, not a .zip.  r12b is kept in because it is the old default.
Here is a survey of the NDK versions used in the most recent Builds entry
in each app that uses the NDK:

{'r10e': 6,
 'r12b': 93,
 'r13b': 4,
 'r14b': 5,
 'r15c': 7,
 'r16b': 14,
 'r17b': 4,
 'r17c': 7,
 'r18b': 9,
 'r19c': 17,
 'r20': 1,
 'r20b': 22,
 'r21': 3,
 'r21d': 56,
 'r21e': 65,
 'r22': 9,
 'r22b': 15,
 'r9b': 1}

#517


import glob
import os
import yaml

try:
    from yaml import CSafeLoader as SafeLoader
except ImportError:
    from yaml import SafeLoader

ndks = dict()
for f in glob.glob('metadata/*.yml'):
    with open(f) as fp:
        app = yaml.load(fp, Loader=SafeLoader)
        if app.get('Disable'):
            continue
        build = app.get('Builds', [])[-1]
        if build.get('disabled'):
            continue
        ndk = build.get('ndk')
        if ndk and ndk[1] == '9':
            print(f, build)
        elif ndk and int(ndk[2:3]) < 18:
            print(f, build)
        if ndk:
            print(f, ndk)
            if ndk not in ndks:
                ndks[ndk] = 0
            ndks[ndk] += 1

import pprint

pprint.pprint(ndks)
2021-05-25 17:06:28 +02:00
relan
6d9410597b Upgrade NDK r22 to r22b 2021-04-07 12:12:04 +03:00
Hans-Christoph Steiner
ee84eb762c buildserver: SharedFoldersEnableSymlinksCreate: false 2021-03-23 11:01:40 +01:00
Hans-Christoph Steiner
4b05854ffc buildserver: only install the latest gradle version
gradle checksums are now handled by gradlew-fdroid

refs #370 #517 #737
2021-03-11 08:40:34 +01:00
Licaon_Kter
d683a40f96 Add NDK 21e 2021-03-10 16:36:14 +02:00
Licaon_Kter
5396b79da7 Fix after https://gitlab.com/fdroid/fdroidserver/-/merge_requests/853 2021-02-12 09:14:27 +00:00
TacoTheDank
c88a3f56a1 Update NDK to r22 and add buildtools 30.0.3 2020-12-26 19:51:22 -05:00
Hans-Christoph Steiner
e927ed02a8
provision-apt-get-install: fix gpg from stretch-backports-sloppy
!822

The gpg dependencies are complicated.  The previous setup was always
failing with:
The following information may help to resolve the situation:
The following packages have unmet dependencies:
 gpg : Depends: gpgconf (= 2.2.20-1~bpo9+1)
       Depends: libassuan0 (>= 2.5.0) but 2.4.3-2 is to be installed
       Depends: libgpg-error0 (>= 1.35) but 1.26-2 is to be installed

Thanks to @izzysoft to spotting this.
2020-11-09 14:37:55 +01:00
Michael Pöhn
196dc75481 Merge branch 'yaml-config' into 'master'
move config.py to config.yml

See merge request fdroid/fdroidserver!571
2020-11-05 23:00:32 +00:00
Michael Pöhn
51a099cb3d Merge branch 'gpgaptfix' into 'master'
Use newer gpg from sloppy

See merge request fdroid/fdroidserver!822
2020-11-04 15:22:39 +00:00
licaon-kter
2fc0178972 Use newer gpg from sloppy 2020-11-01 14:02:48 +02:00
Hans-Christoph Steiner
a9fdb5b401 init: switch to config.yml as the default format 2020-10-23 14:20:49 +02:00
Hans-Christoph Steiner
fd41b70e27 purge common.use_androguard, it is now used by default
Up until now, the buildserver has not included androguard.  Since a
good version of androguard (v3.3.3+) is included in stretch-backports
and the buildserver is already setup to use stretch-backports, this
sets up the buildserver with androguard.

closes #627
2020-10-22 11:58:55 +02:00
Marcus Hoffmann
ba28d631b7 add now required packages to buildserver installation
Dependencies introduced in ffdroid/fdroidserver!798
2020-09-27 12:47:03 +02:00
Hans-Christoph Steiner
e659a5353d
provision-apt-get-install: gpg needs to come from stretch-backports
https://gitlab.com/eighthave/fdroiddata/-/jobs/611438997
Otherwise it fails to install:

```
  gpg : Depends: gpgconf (= 2.2.20-1~bpo9+1)
        Depends: libassuan0 (>= 2.5.0) but 2.4.3-2 is to be installed
        Depends: libgpg-error0 (>= 1.35) but 1.26-2 is to be installed
```

* 7e8c244473
* !769
2020-06-25 18:08:31 +02:00
Hans-Christoph Steiner
7e8c244473 provision-apt-get-install: ensure gpg is installed
The basebox installs gpg, but when this script is used to provision
a GitLab CI Runner, gpg was missing.  It is used in some builds.
2020-06-24 21:43:58 +02:00
TacoTheDank
0216513da6 Upgrade NDK r21c to r21d 2020-06-10 15:55:53 -04:00
Hans-Christoph Steiner
b1eb086259 Merge branch 'patch-2' into 'master'
Add NDK r21c

See merge request fdroid/fdroidserver!755
2020-06-02 19:59:38 +00:00
TacoTheDank
edd41b3c58 Upgrade NDK r21b to r21c 2020-05-29 16:23:20 -04:00
Hans-Christoph Steiner
1ed4ed61c7 buildserver: explicitly include sudo as a dependency
The basebox currently provides sudo, but that may not always be the case.
This makes the sudo dependency explicit, so that this provisioning script
can also be used in other settings, like GitLab CI.
2020-05-27 22:09:52 +02:00
relan
13ac46af07 makebuildserver: upgrade NDK r21 to r21b 2020-05-14 08:43:33 +03:00
Licaon_Kter
9e8aef6771 Add stretch-backports-sloppy repo too 2020-03-30 07:10:51 -04:00
relan
1b001cf1b5 provision-gradle: remove fake proxy configuration
The prohibition of HTTP causes connection errors on some builds:

    IOException: https://dl.google.com/android/repository/addons_list-3.xml
    java.net.ConnectException: Connection refused (Connection refused)
    IOException: https://dl.google.com/android/repository/addons_list-2.xml
    java.net.ConnectException: Connection refused (Connection refused)
    IOException: https://dl.google.com/android/repository/addons_list-1.xml
    java.net.ConnectException: Connection refused (Connection refused)
    Failed to download any source lists!

Partly revert a746917022.
2020-03-15 17:41:29 +03:00
Hans-Christoph Steiner
d9722f4453 buildserver: use long timeouts for gradle downloads
default is 30 seconds, this uses 10 minutes to avoid things like:

* What went wrong:
A problem occurred configuring root project 'org.fdroid.fdroid'.
> Could not resolve all files for configuration ':classpath'.
   > Could not download auto-value.jar (com.google.auto.value:auto-value:1.5.2)
      > Could not get resource 'https://repo.maven.apache.org/maven2/com/google/auto/value/auto-value/1.5.2/auto-value-1.5.2.jar'.
         > Read timed out

* https://stackoverflow.com/a/49646993
* https://github.com/gradle/gradle/issues/4629#issuecomment-393182135
* https://github.com/gradle/gradle/pull/3371/files
2020-02-13 22:32:51 +01:00
Hans-Christoph Steiner
bfe8f05de6
fix syntax error from ae86dc3d38
fdroid/fdroidserver!713

[skip ci]
2020-02-11 12:48:42 +01:00
Hans-Christoph Steiner
a746917022 provision-gradle: disable daemon and block HTTP repos
closes #624
#712
2020-02-05 20:36:25 +01:00
Hans-Christoph Steiner
ae86dc3d38 buildserver: set quiet options to stop apt-get spamming build logs
https://gitlab.com/fdroid/fdroidserver/issues/636#note_266483988
2020-02-05 20:36:25 +01:00