1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-20 13:50:12 +01:00
Commit Graph

582 Commits

Author SHA1 Message Date
Hans-Christoph Steiner
09fa49a7a3
make get_android_tools_versions() search ndk_paths from config 2021-05-28 09:13:38 +02:00
Hans-Christoph Steiner
7a1d236c8d
only support zipballs in NDK provisioning
Since I discovered there is an r10e zipball, this can now get all NDKs
in zipball form.
fdroid/android-sdk-transparency-log@447fea86e7

closes #902
2021-05-28 09:13:36 +02:00
Hans-Christoph Steiner
9f77044d0d
auto-detect NDKs installed in standard paths
'ndk_paths' will be automatically filled out from well known sources
like $ANDROID_HOME/ndk-bundle and $ANDROID_HOME/ndk/*.  If a required
version is missing in the buildserver VM, it will be automatically
downloaded and installed into the standard $ANDROID_HOME/ndk/
directory.  Manually setting it here will override the auto-detected
values.  The keys can either be the "release" (e.g. r21e) or the
"revision" (e.g. 21.4.7075529).

https://developer.android.com/studio/projects/configure-agp-ndk#agp_version_41
* sdkmanager installs "ndk;12.3.4567890" into $ANDROID_SDK_ROOT/ndk/
* sdkmanager installs "ndk-bundle" into $ANDROID_SDK_ROOT/ndk-bundle/
2021-05-28 09:13:34 +02:00
Hans-Christoph Steiner
4686c06f62 metadata: allow ndk: to be str or list of release or revision
There are two version numbers used for NDKs: the "release" and the
"revision".  The "release" is used in the download URL and zipball and the
"revision" is used in the source.properties and the gradle ndkVersion field.

Also, there are some builds which need multiple NDKs installed, so this
makes it possible to have a list of release/revision entries in build.ndk.
This does not yet add full support since _fdroidserver/build.py_ will also
need changes.
2021-05-28 09:13:27 +02:00
Hans-Christoph Steiner
9d44fa7919
gitlab-ci: auto-generate merge request when NDK release found
Following the pattern of the gradle bot, this will check the transparency
log for any new NDK release.  If there are any, it will make a merge
request from @fdroid-bot.
2021-05-25 17:06:30 +02:00
Hans-Christoph Steiner
69fcd6a024
build: auto-download missing NDKS if they're known and can be verified
refs #517 #717
2021-05-25 17:06:24 +02:00
Hans-Christoph Steiner
0b0bc803b7 build: --test now keeps unsigned APKs in tmp/ that fail to reproduce
Before, whenever an unsigned APK failed to reproduce, it was just deleted.
That makes debugging hard.  This makes it keep the unsigned APK, which is
written in tmp/ when using --test.

@jspricke this is related to !864
2021-04-16 09:46:23 +02:00
Hans-Christoph Steiner
3d69e767d8
common: test abs and rel paths in get_all_gradle_and_manifests() 2021-04-16 09:40:19 +02:00
Jochen Sprickerhof
f01a3caf77 Bump METADATA_VERSION for apksigner transition
We switched to apksigner in 50f0534d but old apks where still verified
with jarsigner (or an old apksigner version). Bumping the
METADATA_VERSION to force a rebuild of apkcache.
Hopefully this resolves de.chagemann.regexcrossword getting the
KnownVuln, DisabledAlgorithm tags.
2021-04-14 22:08:26 +00:00
Hans-Christoph Steiner
2946c90dd4 publish: rename vars to match naming in JAR Signature docs
https://docs.oracle.com/javase/tutorial/deployment/jar/intro.html

closes #892
2021-04-14 23:01:47 +02:00
Hans-Christoph Steiner
44d481768f publish: add test for reproduble builds with signatures 2021-04-14 23:01:47 +02:00
Felix C. Stegerman
202fd8b25a
vendor & use apksigcopier v0.4.0-12-g93d8e14 2021-04-14 21:06:20 +02:00
Felix C. Stegerman
67a0f3ae5b
use subclass hack for better ZIP cloning
See https://bugs.python.org/issue43547 for more info on the details.

thanks to @obfusk for the technique
2021-04-14 15:17:56 +02:00
Hans-Christoph Steiner
cff575f402 common: add test case for read_pkg_args() 2021-04-12 09:11:58 +02:00
Hans-Christoph Steiner
b5485ded64
skip test_find_apksigner_system_package_android_home if no apksigner 2021-03-23 23:29:50 +01:00
Hans-Christoph Steiner
b69b95103e add complete tests for finding apksigner; fix minor detection bug
find_apksigner() was preferring the oldest valid version rather than the
newest.
2021-03-23 21:01:53 +01:00
Hans-Christoph Steiner
67e65ce2ea purge complete-ci-tests, those are better covered in .gitlab-ci.yml 2021-03-22 23:36:04 +01:00
Hans-Christoph Steiner
93ca13fa16 run-tests: remove broken test that never ran due to a syntax error 2021-03-22 22:31:29 +01:00
Hans-Christoph Steiner
7ec2a8ebb3 add apksigner, jarsigner, and keytool to relevant status JSON 2021-03-22 22:31:29 +01:00
Hans-Christoph Steiner
903a7396b1 switch to preferring apksigner, requiring for fdroid publish 2021-03-22 22:30:49 +01:00
Jochen Sprickerhof
ea9f917ddc Support versionCode and versionName in brackets
Example: versionCode(29)
https://github.com/florisboard/florisboard/blob/v0.3.9/app/build.gradle.kts#L24
2021-03-19 13:58:02 +01:00
Hans-Christoph Steiner
dbf80ad771
metadata: remove git repo handling from metadata parsing
This is a vestige of implementing builds from a .fdroid.yml file directly
in the app's source repo.  It was never fully complete and seems to not be
used in any apps in fdroiddata.  This makes `fdroid build --all` runs much
faster since it does not need to do any git handling for apps that do not
have any new builds to run.

4e8e29794
948689281
a4e431080
e37be9b06
e775d330c
2021-03-17 13:44:24 +01:00
Hans-Christoph Steiner
cc3b88ab30 gradlew-fdroid: use gradle zipballs from local cachedir, if present 2021-03-11 08:40:34 +01:00
Hans-Christoph Steiner
65836d9e9b run-tests: fix local_copy_dir test, it broke when /tmp/fdroid existed 2021-03-09 17:55:10 +01:00
Hans-Christoph Steiner
649347dcd4 common: properly parse version from NDK's source.properties 2021-03-09 15:55:45 +01:00
Hans-Christoph Steiner
6a875a1821 update: stop extracting and storing XML icons, they're useless
APKs that only have XML icons in them should include an icon.png in the
metadata or fastlane/triple-t

closes #344
2021-03-09 15:40:30 +01:00
Hans-Christoph Steiner
544a45c16a index: raise error rather than crash on bad repo file
If a non-APK is added with the appid/packageName that matches some APKs, it
should through an error.

Traceback (most recent call last):
  File "/home/hans/code/fdroid/server/fdroid", line 22, in <module>
    fdroidserver.__main__.main()
  File "/home/hans/code/fdroid/server/fdroidserver/__main__.py", line 211, in main
    mod.main()
  File "/home/hans/code/fdroid/server/fdroidserver/update.py", line 2343, in main
    index.make(apps, sortedids, apks, repodirs[0], False)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 142, in make
    fdroid_signing_key_fingerprints)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 166, in make_v1
    v1_sort_packages(packages, fdroid_signing_key_fingerprints)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 292, in v1_sort_packages
    packages.sort(key=v1_sort_keys)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 288, in v1_sort_keys
    .format(apkfilename=package['apkName']))
fdroidserver.exception.FDroidException: at.roteskreuz.stopcorona_8.jobf does not have a valid signature!
2021-03-09 15:40:30 +01:00
Jochen Sprickerhof
7f2cbd1876 Fix serverwebroot test 2021-02-12 10:20:30 +00:00
Jochen Sprickerhof
33bff6b298 Publish unsigned builds
This should make debugging reproducible builds easier.
2021-02-12 10:20:30 +00:00
Hans-Christoph Steiner
525dcb8f98 update: handle large, corrupt, or inaccessible fastlane/triple-t files
```
Traceback (most recent call last):
  File "../fdroid", line 22, in <module>
    fdroidserver.__main__.main()
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 227, in main
    raise e
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 208, in main
    mod.main()
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2340, in main
    repoapps = prepare_apps(apps, apks, repodirs[0])
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2176, in prepare_apps
    copy_triple_t_store_metadata(apps_with_packages)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 1076, in copy_triple_t_store_metadata
    _set_author_entry(app, 'authorWebSite', os.path.join(root, f))
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 784, in _set_author_entry
    with open(f, errors='replace') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'build/player.efis.cfd/pfd/src/main/play/contact-website.txt'
```
2021-02-11 17:42:13 +01:00
Benedikt Brückmann
9c609aa361 Fix copy-paste error 2021-02-08 12:25:02 +01:00
Benedikt Brückmann
23931e6818 Install [test] dependencies when running complete-ci-tests 2021-02-08 12:25:02 +01:00
Benedikt Brückmann
b1bde9e407 * Skip tidy check if html5print is not available
* install test dependencies on arch
2021-02-08 12:25:02 +01:00
Benedikt Brückmann
022d73b3b6 Add HTML/CSS tidy test
and tidy up the content
2021-02-08 12:25:02 +01:00
Hans-Christoph Steiner
5cd9a8e396
tests: make init apksigner test use existing test keystore
This test does not need to generate a keystore, so using the existing one
reduces the run time from 4 seconds to 0.040 seconds.  Also, if makes this
mystery error on macOS go away:

```
Traceback (most recent call last):
  File "/Users/travis/build/fdroidtravis/fdroidserver/tests/init.TestCase", line 66, in test_main_in_empty_dir
    fdroidserver.init.main()
  File "/Users/travis/build/fdroidtravis/fdroidserver/fdroidserver/init.py", line 231, in main
    common.genkeystore(c)
  File "/Users/travis/build/fdroidtravis/fdroidserver/fdroidserver/common.py", line 3434, in genkeystore
    cmd = [config['keytool'], '-genkey',
KeyError: 'keytool'

```
2021-01-29 09:06:21 +01:00
Hans-Christoph Steiner
32a0c61010 init: enable apksigner by default if it is found 2021-01-28 22:25:33 +01:00
Hans-Christoph Steiner
0af157244a
gradle-release-checksums: runs after commits push to fdroid/fdroidserver 2021-01-25 11:05:20 +01:00
Hans-Christoph Steiner
669ce10d9a
enable complete lint/CI checking on gradle-release-checksums.py 2021-01-25 11:05:16 +01:00
Hans-Christoph Steiner
d133b38f22 fdroid-bot creates merge request when new gradle versions are found 2021-01-25 00:26:14 +01:00
Hans-Christoph Steiner
eec0d3a336 gitlab-ci: gradle job generates required changes for new versions 2021-01-24 16:58:38 +01:00
Hans-Christoph Steiner
f527b17715 sync repo examples with defaults and comment out example repo settings
!359
2021-01-24 09:26:11 +01:00
Hans-Christoph Steiner
aebf70a763 tests: run pip tests using fdroid that pip installed 2021-01-23 20:13:17 +01:00
Hans-Christoph Steiner
5ae2703477 init: no error if examples/ dir cannot be found; overhaul repo_icon
There are so many possible installation paths for Python modules, it has
been very hard to even find and test them all.  This adds a fallback option
if the examples dir cannot be found.  A repo can work without an icon or
the example config.py.

This removes the fake assumption that the icon can be a full path in the
config.py.  While the path was being properly passed through to the index
file, the file was never copied properly into place nor rsynced to the web
server.
2021-01-18 14:07:20 +01:00
Hans-Christoph Steiner
94cb19f571 set METADATA_VERSION to 20000 to leave room for 1.1.x changes 2021-01-15 14:33:21 +01:00
Hans-Christoph Steiner
fff59e5197 only use AutoName: in checkupdates
AutoName: is only needed for the commit messages generated by checkupdates,
and it makes the logic for localized names confusing.

closes #654
refs #304
2021-01-14 20:36:09 +01:00
Hans-Christoph Steiner
0f6b638986 common.get_app_display_name() for finding app names 2021-01-14 20:34:15 +01:00
Hans-Christoph Steiner
81b55e0d8f
tests: use wc -m to support BSD and macOS 2021-01-07 17:09:44 +01:00
Hans-Christoph Steiner
bf25b4ca03 eliminate app.builds everywhere, it should be app['Builds']
The .txt format was the last place where the lowercase "builds" was used,
this converts references everywhere to be "Builds".  This makes it possible
to load metadata YAML files with any YAML parser, then have it possible to
use fdroidserver methods on that data, like metadata.write_metadata().

The test files in tests/metadata/dump/*.yaml were manually edited by cutting
the builds: block and putting it the sort order for Builds: so the contents
should be unchanged.

```
sed -i \
 -e 's/app\.builds/app.get('Builds', \[\])/g' \
 -e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
 -e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
 -e "s/app\.get(Builds, \[\])/app.get('Builds', \[\])/g" \
 -e "s/app\.get('Builds', \[\])\.append/app\['Builds'\].append/g" \
 -e "s/app\['builds'\]/app.get('Builds', [])/g" \
 */*.*
```
2020-12-15 08:55:05 +01:00
Hans-Christoph Steiner
fb628c2cb2 include modified and untracked files in status JSON
Ideally, an fdroid repo should be running from a clean git repo, so that
all changes are tracked in git.  This change is useful in seeing which
changes and/or files are not in git.  If there are modified files, the
dirty flag will be set, so this info can help debugging that.
2020-12-15 08:30:48 +01:00
Hans-Christoph Steiner
384922118f index: sanitize fingerprint arg, extract_pubkey() returns with spaces
The key fingerprint should be only hex digits, everything else can be
discarded.  That makes it easy to use this function various fingerprint
formats, including the common, human-readable forms spaces between pairs or
quartets.
2020-12-15 08:30:48 +01:00