1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-19 05:10:10 +01:00
Commit Graph

147 Commits

Author SHA1 Message Date
Marcus Hoffmann
768a91370c publish: use apksigner for signing apks with targetSDK>=30
This makes apksigner a hard requirement of the signing procedure.
We'll first try to find a globally installed version from PATH and if
that's not available fall back to using a version from build-tools.

Future TODO: always sign with apksigner, blocked on signature transplant
support for apksigv2/v3

Closes fdroid/fdroidserver#634
Closes fdroid/fdroidserver#827
2020-09-09 18:09:28 +02:00
Marcus Hoffmann
ba28b44ae7 common: use androguard to figure out minSDK version
Closes: #816
2020-08-19 19:49:39 +02:00
Hans-Christoph Steiner
d0f426e076
replace $$srclib$$ with an absolute path
closes #725
2020-06-24 22:40:26 +02:00
Hans-Christoph Steiner
95c3ab2454 skip yamllint test if yamllint is not installed
!721
2020-05-27 22:07:17 +02:00
Michael Pöhn
b076e8cba7 add srclib spec parser 2020-04-25 17:22:48 +02:00
Michael Pöhn
d24484a950 simple testcase for common.run_yamllint 2020-04-24 15:47:47 +02:00
Michael Pöhn
fa7885063b scrlib: add test for getsrclib 2020-04-16 11:31:59 +02:00
Michael Pöhn
286220fe11 srclib: add test for getsrclibvcs 2020-04-16 11:31:59 +02:00
Hans-Christoph Steiner
ab2291475b import: mv reusable functions to common.py to avoid import_proxy.py
import is a strict keyword in Python, so it is not possible to import a
module called 'import', even with things like:

* import fdroidserver.import
* from fdroidserver import import
2020-03-11 13:41:13 +01:00
Hans-Christoph Steiner
202291d66c
integration test for creating and deploying status JSON files
----------------------------
2020-02-19 14:50:37 +01:00
Hans-Christoph Steiner
271b74af7d fix remove_signing_keys() for Kotlin gradles files (*.gradle.kts) 2020-02-13 22:32:51 +01:00
Hans-Christoph Steiner
83ffeb855f prefer build.gradle with Android Plugin as source of package/version/code
These days, the location that overrides all the others is in the android{}
block of the build.gradle file that loads the com.android.application
plugin.  So this should be the preferred place to read these values.

test files GPL licensed: https://github.com/Integreight/1Sheeld-Android-App
2020-02-13 22:32:51 +01:00
Hans-Christoph Steiner
bfe587979d import: make it work most of the time with git repos
This includes real tests too.
2020-02-13 13:51:52 +01:00
Hans-Christoph Steiner
e76a0c9d6a
git_mirror_size_limit config option to set max git mirror size
GitHub and GitLab have some kinds of limits on how big a git repo can be,
this makes that option configurable.  This also is very useful for tests.
2020-01-14 11:36:08 +01:00
Hans-Christoph Steiner
7d40e89341
checkupdates: split out vercode parsing into testable function 2019-12-03 23:51:48 +01:00
Jochen Sprickerhof
bbee2cf707 Add unit test for string_is_integer() 2019-12-03 21:49:44 +01:00
Hans-Christoph Steiner
0a8af2c55f Merge branch 'fix-readline-in-srclibs' into 'master'
build: fix bad regexs when removing signingConfig from srclibs

See merge request fdroid/fdroidserver!686
2019-10-25 08:18:08 +00:00
Hans-Christoph Steiner
afaa24f2fd
build: fix bad regexs when removing signingConfig from srclibs
I went through the source of all apps in fdroiddata for examples, and found
some that use readLine() for things totally unrelated to signingConfigs.

https://gitlab.com/fdroid/fdroiddata/merge_requests/4775#note_234132902
2019-10-23 12:44:47 +02:00
Michael Pöhn
7fa3c34e5b update tests for fixed log deployment+changelog 2019-10-15 15:19:18 +02:00
Michael Pöhn
d0368d0ad8 common add parse_androidmanifests_ignore test 2019-07-23 22:44:45 +02:00
Michael Pöhn
2c87b5e6f9 deploy build logs: no timestamps 2019-07-06 16:48:56 +02:00
Michael Pöhn
f30983368c build: rsync buildlogs to <webroot>/repo 2019-07-06 16:48:56 +02:00
Hans-Christoph Steiner
a0f5ee661e tests: common.test_sign_apk requires aapt to run 2019-07-03 09:07:36 +02:00
Hans-Christoph Steiner
57b9d1e316 tests: handle when apksigner considers MD5 signatures valid 2019-07-02 22:17:06 +02:00
Taco
457cf22361 Added newer ndks, gradles, latest sdk-license, and update java 1.8 version 2019-03-30 17:10:21 -04:00
Hans-Christoph Steiner
d96f5ff660 support APK Signature V2 when apksigner is installed
This was done with much help from @uniqx.  This is the first level of
supporting APK Signatures v1, v2, and v3.  This is enough to include
APKs with any combo of v1/v2/v3 signatures.  For this to work at all,
apksigner and androguard 3.3.3+ must be installed.

closes #399
2019-02-01 09:17:56 +01:00
Hans-Christoph Steiner
57556aceee remove redundant open() arg: encoding='utf8'
By default, open() returns a str:
https://docs.python.org/3/library/functions.html#open

By default, str is UTF-8:
https://docs.python.org/3/library/stdtypes.html#str

This used to matter on Python 2.x, but this code is 3.x only now.
2018-10-19 15:01:34 +02:00
Hans-Christoph Steiner
6b57cb6b7c fix strict Application ID checks
* upper case letters are allowed at all positions
* there must be a "." separator
2018-09-24 17:07:27 +02:00
Hans-Christoph Steiner
11d46072ab use androguard primitives to speed up finding debuggable flag
androguard parses the whole APK before handing the instance back, this uses
the primitives to just find the <application android:debuggable=""> value,
then stop parsing.

#557
2018-09-21 14:56:46 +02:00
Hans-Christoph Steiner
a3cecc16a3 use partial androguard binary XML parsing to speed up APK ID lookup
Normally, androguard parses the entire APK before it is possible to get any
values from it.  This uses androguard primitives to only attempt to parse
the AndroidManifest.xml, then to quit as soon as it gets what it needs.
This greatly speeds up the parsing (1 minute vs 60 minutes).

fdroid/fdroidserver#557
2018-09-21 14:56:46 +02:00
Hans-Christoph Steiner
807bf3d26b build: reuse common methods for getting metadata from APKs
This splits out the code that gets the list of native ABIs supported, then
uses the standard methods for the rest.
2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner
487c4d02f3 handle package: line output from aapt v28
fdroid/fdroiddata!3484
fdroid/fdroiddata!3562
fdroid/fdroidserver!548
2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner
9d12b1dc61 add strict, tested validation of Android/F-Droid package names
Android has stricter rules than Java for Package Names, but anything the
Python regex thinks is valid must be valid according to Java's rules too.

https://developer.android.com/studio/build/application-id
2018-09-07 14:17:39 +02:00
Hans-Christoph Steiner
5d161cc9fd validate appid when reading metadata files
The metadata file must be named after the Application ID of the app it is
describing, and Android Application IDs must be valid Java Package Names.
2018-09-03 22:56:08 +02:00
Hans-Christoph Steiner
4d13a904f3 use defusedxml to avoid DoS attacks while loading XML 2018-08-29 17:44:54 +02:00
Hans-Christoph Steiner
4503e7a92a replace unneeded eval() call and support negative versionCodes 2018-08-29 17:24:04 +02:00
Hans-Christoph Steiner
1b37b521c8 common: fix tests when running without any Android SDK installed
fdroid/fdroidserver!552
fdroid/fdroidserver#541
2018-08-07 14:51:42 +02:00
Michael Pöhn
5d5617bf40 fix: minor bugs in testcases for: common, metadata, update 2018-08-06 23:53:49 +02:00
Michael Pöhn
752f6a4ba2 applied review suggestions for deploying (build) logs 2018-06-19 15:18:13 +02:00
Michael Pöhn
88e64df3ef deploying build logs to server after each individual build run 2018-06-19 12:24:52 +02:00
Michael Pöhn
4c53c71fcf added helper function for uploading build logs with rsync 2018-06-19 12:24:52 +02:00
Michael Pöhn
429bc21da7 make common.TestCase run independant of cwd 2018-06-19 12:24:52 +02:00
Hans-Christoph Steiner
ff90c0246e fix PEP8 W605 invalid escape sequence
Python 3.7 will get a lot stricter with escape sequences.  They must be
valid.

* https://lintlyci.github.io/Flake8Rules/rules/W605.html
* https://docs.python.org/3/whatsnew/3.6.html#deprecated-python-behavior
2018-05-29 13:51:47 +02:00
Hans-Christoph Steiner
91068d8e00 tests: rename unicode test file to fix "file name too long" errors
pristine-tar and mkdir barf on this file.
2018-05-25 17:27:58 +02:00
Michael Pöhn
6b1f242d25 added tests for common.calculate_math_string 2018-05-22 13:24:37 +02:00
J-Jamet
d6f1de2649 Fix applicationIdSuffix / versionNameSuffix #455 2018-05-04 12:24:16 +02:00
Hans-Christoph Steiner
27a5cce832 implement common.get_apk_id() using androguard 2018-05-03 13:46:42 +02:00
Hans-Christoph Steiner
98a2f70e38 fix intermittent test failure
For some reason, the parser stopped working intermittently, even
though the format has been the same since aapt 23 or earlier.  Then
also, some of the test cases pointed to symlinks that were no longer
generated, and one test app now has a blank versionName.

Strange that this wasn't caught in the gitlab-ci runs.  !484

FAIL: test_get_api_id_aapt (__main__.CommonTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./common.TestCase", line 578, in testA_get_api_id_aapt
    self.assertEqual(versionName, vn)
AssertionError: '0.1' != "0.1' platformBuildVersionName='4.3.1-1425645"
- 0.1
+ 0.1' platformBuildVersionName='4.3.1-1425645
2018-05-03 13:46:42 +02:00
Hans-Christoph Steiner
e75bf70be6 signatures: future-proof fetching app ID info from APK
We're not using platformBuildVersionName and it might go away just like it
appeared: with no good reason or announcement.
2018-02-22 15:08:55 +01:00
Hans-Christoph Steiner
52b3436ff6 make is_apk_and_debuggable() default to using androguard before aapt 2018-02-22 15:08:53 +01:00
Hans-Christoph Steiner
e451ec0079 common: fix bug in new SHA-256 signatures for >= android-18
Luckily, this is only used in `fdroid nightly` so far.
2017-12-28 23:07:26 +01:00
mimi89999
455dcc34d8
Add Nextcloud and DavDroid test case 2017-12-27 11:51:57 +01:00
mimi89999
90c7dd29df
gradle file: use flavour specific versionCode/versionName, fall back to parsing line by line 2017-12-23 17:12:54 +01:00
mimi89999
918bd15c45
Revert: gradle file: use flavour specific versionCode/versionName, fall back to parsing line by line 2017-12-23 12:57:34 +01:00
Hans-Christoph Steiner
61aac0503a Merge branch 'fixFlavor' into 'master'
Regex only for flavor blocks: flavor { ... }

See merge request fdroid/fdroidserver!407
2017-12-14 16:56:01 +01:00
Hans-Christoph Steiner
bec4f7d547 add Conversations as gradle flavor test case 2017-12-14 16:52:02 +01:00
Hans-Christoph Steiner
6228162cbd handle jarsigner/apksigner output cleanly for rational logging
These were both spamming the output with lots of confusing messages, even
when --verbose was not used.  Jarsigner especially has confusing messages,
since it has warnings that do not pertain to APK signatures at all, like
the ones about timestamps and missing Certificate Authority.

closes #405
2017-12-07 17:32:14 +01:00
Marcus Hoffmann
bfe2c00834 common.testCase: fix find_sdk_tools when aapt is installed in /usr/bin
The testlogic was broken when having both aapt in /usr/bin and also as
part of the android sdk.
2017-12-06 12:30:47 +01:00
Hans-Christoph Steiner
1c3a4479ab add common.sign_apk() for nighly as test for using in publish
Since the MD5 migration was quite a bit of work, it makes sense to start
on moving away from SHA1 as much as possible while it is easy to do. SHA256
will only work in APK signatures on android-18 (4.3) or newer.  So if an
APK has a minSdkVersion of 18 or newer, then sign with SHA256.

https://issuetracker.google.com/issues/36956587
https://android-review.googlesource.com/c/platform/libcore/+/44491
2017-12-04 22:52:41 +01:00
Hans-Christoph Steiner
5b22ff7dc6 tests: use standard dir setup so all tests start in same dir 2017-11-30 17:32:53 +01:00
Hans-Christoph Steiner
3ff4b656c6 tests: name temp test dir after test function that used it 2017-11-30 17:32:53 +01:00
Hans-Christoph Steiner
c7c40cb59f PEP8 fixes 2017-11-30 13:42:37 +01:00
tobiasKaminsky
33aee96ed9
added test case 2017-11-30 11:12:18 +01:00
Hans-Christoph Steiner
2ee32f3524 fix tests to work when only Debian Android SDK is installed
jenkins.debian.net only had `apt install android-sdk`, it does not have any
of Google's packages installed.
2017-11-08 14:05:17 +01:00
Hans-Christoph Steiner
17efa13183 tests: pre-set failfast as reminder of a handy time saver 2017-10-25 23:01:25 +02:00
Hans-Christoph Steiner
e0df6d2479 choose the most recent available version of Java
This came about testing on OSX, where there are often multiple versions of
the JDK installed.  This was choosing the oldest version.  It should
choose the most recent version.
2017-10-25 23:01:25 +02:00
Michael Pöhn
788c8f97fd better error message in publish when repo_key is not set 2017-10-17 22:10:39 +02:00
Michael Pöhn
5a524d4d0c added some developer-signing key metadata to tests 2017-09-26 14:11:09 +02:00
Michael Pöhn
7c4b8bcac4 added parse release filename function 2017-09-26 14:11:09 +02:00
Michael Pöhn
04daa7a03a test for common.get_app_id_aapt 2017-09-26 14:11:09 +02:00
Michael Pöhn
efb0bf6ee5 add common functions for dealing with apk signatures 2017-09-26 14:11:09 +02:00
Michael Pöhn
c196f7dd7a common function for fetching sha256 signing-key fingerprint 2017-09-26 14:11:09 +02:00
Hans-Christoph Steiner
eaca20675c tests: test setting up app git repo like fdroid build does
I wrote this to try to find #379 but this test didn't trigger that bug.
2017-09-25 16:35:18 +02:00
Hans-Christoph Steiner
cbf7ba0414 tests: add setUp() method to common to handle standard stuff 2017-09-25 16:35:18 +02:00
Hans-Christoph Steiner
9471bf2731 regexs for getting packageName and versionCode from filenames
This is useful for parsing APK files, which can include packageName,
versionCode, and optionally 7 char signing key ID (i.e. <sig>).
This also can set the packageName and versionCoe for non APK files, so
that it is easy to assign them to metadata files, and to allow for
upgrades by setting the versionCode in the filename.
2017-06-01 16:01:05 +02:00
Hans-Christoph Steiner
17cd07f1a7 fix pylint unused-argument 2017-05-23 22:34:16 +02:00
Michael Pöhn
8b51e40d63 avoid duplicate value assignments when updating config files 2017-04-02 12:08:01 +02:00
Torsten Grote
9f765ed6f7
Move index signing methods into signindex.py 2017-03-29 12:11:40 -03:00
Hans-Christoph Steiner
223c793201 prefer apksigner if installed, jarsigner sucks
Google has their own utility for verifying APK signatures on a desktop
machine since Java's jarsigner is bad for the task.  For example, it
acts as if an unsigned APK validates.  And to check whether an APK is
unsigned using jarsigner is difficult.

apksigner also does the v2 signatures, so it will have to be used
eventually anyway.  It is already in Debian/stretch and can be
available in jessie-backports if need be.

https://android.googlesource.com/platform/tools/apksig
https://packages.debian.org/apksigner
2017-03-22 10:51:12 +01:00
Hans-Christoph Steiner
998b6245e9 verify: ensure only a single signature is in compared APK
The ZIP format allows multiple entries with the exact same filename, and on
top of that, it does not allow deleting or updating entries.  To make the
`fdroid verify` procedure failsafe, it needs to create a new temporary APK
that is made up on the contents of the "unsigned APK" and the signature
from the "signed APK".  Since it would be possible to give a signed APK as
in the unsigned one's position, `fdroid verify` was not able to update the
signature since it was just adding the new signature to the end of the ZIP
file.  When reading a ZIP, the first entry is used.
2017-03-22 10:51:12 +01:00
Hans-Christoph Steiner
fa657ce720 move update.signjar() to common so it can also be used in signindex 2017-03-17 13:55:40 +01:00
Hans-Christoph Steiner
e0f39a7e7b rename Build fields: version -> versionName, vercode -> versionCode
Since the YAML/JSON/etc. field names are now exactly the same as the field
names used in the internal dict in the Build class, this is a global rename

This keeps with the standard names used in Android:
https://developer.android.com/guide/topics/manifest/manifest-element.html
2017-02-24 11:01:01 +01:00
Daniel Martí
82b1d7ad14 all: make newer pycodestyle happy
Apparently the "two empty lines" rule is now stricter.
2016-11-15 20:55:06 +00:00
Hans-Christoph Steiner
84e09cd2a2 allow arbitrary build products, not only APKs
This makes it so that the final build product can be specified in output=
and it'll work no matter if its an APK or not.  This was developed around
the case of building the OTA update.zip for the Privileged Extension. It
should work for any build process in theory but it has not yet been tested.

https://gitlab.com/fdroid/privileged-extension/issues/9
2016-11-07 14:53:01 +01:00
Hans-Christoph Steiner
3768d7a4d6 refactor env handling for FDroidPopen to support .fdroid.* metadata
The start up sequence of processes that are based on the .fdroid.* metadata
is a bit different, so this ensures that the environment variables get
properly initialized in all cases.

This also creates a single function where the environment is set.  Before
it was being set in multiple places across multiple files.
2016-03-23 17:16:27 +01:00
Daniel Martí
24ad0418e3 tests: switch to python3 2016-03-11 13:27:00 +00:00
Daniel Martí
ee9a296b64 Make pre-commit hook pass after python3 switch 2016-03-10 16:43:37 +00:00
Dmitriy Bogdanov
5d54b761af Add FDroidPopen usage test 2016-02-18 00:45:06 +04:00
Daniel Martí
bf8518ee8f Rework build into a class
This simplifies usage, goes from

    build['flag']
to
    build.flag

Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.

As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.

While at it, unify "build", "thisbuild", "info", "thisinfo", etc into
just "build".
2015-12-02 20:03:11 +01:00
Daniel Martí
ab614ab442 Rework app into a class
This simplifies usage, goes from

	app['Foo']
to
	app.Foo

Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.

As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.
2015-11-28 17:11:05 +01:00
Daniel Martí
00bd75aa8c tests: close files 2015-10-08 13:20:35 +02:00
Hans-Christoph Steiner
9e5dd19fc8 add test files to pre-commit hook and fix pep8 errors 2015-09-01 11:39:50 +02:00
Hans-Christoph Steiner
d53a5af715 build: improve regsub pattern for setting buildToolsVersion
This addresses the discussion in !64
https://gitlab.com/fdroid/fdroidserver/merge_requests/64

Sometimes, buildToolsVersion is a kind of gradle macro call, and other
times it is a variable assignment.  This regsub pattern now handles both of
those cases.
2015-08-19 17:55:17 +02:00
Hans-Christoph Steiner
64a9c93ce7 test new common.regsub_file() method that replaces sed calls 2015-08-01 00:38:41 +02:00
Ciaran Gultnieks
384f97e998 Ensure package names are valid 2015-01-26 18:30:01 +00:00
Hans-Christoph Steiner
9244256461 find cmds from SDK build-tools in a more flexible way, on the fly
This is a more flexible approach than testing for the complete SDK and
build-tools up front.  This will only test for the commands that are
actually being run, so that if you only have `aapt` installed, you can do
`fdroid update` without errors, but other commands will still give
appropriate errors.

This also makes the build_tools item in config.py optional, it is only
needed if you want to force a specific version of the build-tools.
2014-12-14 13:25:20 +01:00
Hans-Christoph Steiner
298a88a498 added test case for common.isApkDebuggable()
Just getting into the habit of adding tests to everything that I change...
Also, it should be useful to have an unsigned APK in the test collection,
since `fdroid update` should handle it gracefully and give a warning of
some kind.
2014-12-14 13:25:20 +01:00