kamp/fdroidserver
1
0
Fork 0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-19 21:30:10 +01:00
Code Activity
5,985 Commits 1 Branch 59 Tags 44 MiB
1902bfb6fc
Commit Graph

472 Commits

Author SHA1 Message Date
Hans-Christoph Steiner
4bb590b6e5
publish: write status JSON 2020-02-19 14:50:24 +01:00
Hans-Christoph Steiner
271b74af7d fix remove_signing_keys() for Kotlin gradles files (*.gradle.kts) 2020-02-13 22:32:51 +01:00
Hans-Christoph Steiner
3de2d0f56f add basic test suite for gradlew-fdroid 
!707
fdroiddata#6216

The se.manyver app is licensed MPL, the files came from:
81d247a6cd
 2020-02-13 22:32:51 +01:00
Hans-Christoph Steiner
83ffeb855f prefer build.gradle with Android Plugin as source of package/version/code 
These days, the location that overrides all the others is in the android{}
block of the build.gradle file that loads the com.android.application
plugin.  So this should be the preferred place to read these values.

test files GPL licensed: https://github.com/Integreight/1Sheeld-Android-App
 2020-02-13 22:32:51 +01:00
Hans-Christoph Steiner
1153ac24fd import: overhaul URL validation to use urllib.parse 
Python provides us a lovely URL parser with some level of validation built
in.  The parsed URL is then much easier to validate.
 2020-02-13 13:51:52 +01:00
Hans-Christoph Steiner
e9a6c84efd import: split URL parsing from code cloning 
This makes things testable and easier to follow.
 2020-02-13 13:51:52 +01:00
Hans-Christoph Steiner
bfe587979d import: make it work most of the time with git repos 
This includes real tests too.
 2020-02-13 13:51:52 +01:00
Hans-Christoph Steiner
5fb368916f
index: fix no JAR test case that fails on example.org 
http://example.org/index-v1.jar now returns the HTTP header
"Content-Encoding: gzip" but then the reply is plain HTML. That
triggers a ContentDecodingError instead of an HTTPError, so this
changes the test to success on any RequestsException.
 2020-02-05 20:33:40 +01:00
Hans-Christoph Steiner
427856b5fd
tests: use same mirrors order in all tests 
This makes it so running `../fdroid update --nosign --pretty` in tests/ no
longer creates a diff in the tests/*/index* files.  It matches the order
set in tests/run-tests.
 2020-01-31 15:38:10 +01:00
Hans-Christoph Steiner
6ce22bff56
update: strip newlines from name/summary/video entries 
These entries are hardcoded as a single line in all the app stores, so
newlines should be stripped to get the data simple to use.  This is in
contrast with the on-disk format for Fastlane and Triple-T, which includes
a newline in the title.txt and short_description.txt files.  I think all
files in those systems are normalized to end in a newline.
 2020-01-31 15:37:59 +01:00
Hans-Christoph Steiner
fdede914cd
tests: properly name CheckupdatesTest class 2020-01-31 15:37:32 +01:00
Hans-Christoph Steiner
3df276cc3c
fix all bandit B310 urllib_urlopen 
"Audit url open for permitted schemes. Allowing use of ‘file:’’ or custom
schemes is often unexpected."

https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b310-urllib-urlopen
 2020-01-31 15:37:30 +01:00
Marcus Hoffmann
8961485125 remove dscanner subcommand 
It's unused and unmaintained. It could  potentially be revived as a
plugin at a later point.
 2020-01-31 14:17:12 +01:00
Michael Pöhn
f2a80ffa3d fix tests on old python version 2020-01-28 11:43:09 +01:00
Michael Pöhn
b25eeb66a1 fix code style 2020-01-28 11:43:09 +01:00
Michael Pöhn
f21481ca81 add some simple tests for main 2020-01-28 11:43:09 +01:00
Hans-Christoph Steiner
e76a0c9d6a
git_mirror_size_limit config option to set max git mirror size 
GitHub and GitLab have some kinds of limits on how big a git repo can be,
this makes that option configurable.  This also is very useful for tests.
 2020-01-14 11:36:08 +01:00
Gerhard Olsson
cf9bff1d56 AutoUpdateMode: Do not include + in suffix 
Add suffix to version only
 2020-01-02 13:27:12 +00:00
Marcus Hoffmann
83526e09a3 update: fix unarchiving of allowed disabled algorithm 
6d0b1bbe6fae0909683f2c6a154515bc4bfcb674 didn't handle the
allow_disabled_algorithm case at all, so we add it back.

This additionally fixes a (previously existing) bug where setting
allow_disabled_algorithms to True didn't move apks back from archive to
repo. Introduce a new test for this.

The disabled_algorithm archiving logic is still all over the place so
ideally that needs a future refactor.
 2020-01-01 15:25:43 +01:00
Marcus Hoffmann
7f6efa74f5 tests: add test for correctly handling CVC when archiving 2019-12-24 17:35:54 +01:00
Hans-Christoph Steiner
7d40e89341
checkupdates: split out vercode parsing into testable function 2019-12-03 23:51:48 +01:00
Jochen Sprickerhof
bbee2cf707 Add unit test for string_is_integer() 2019-12-03 21:49:44 +01:00
Mike Hardy
3648ef1f2e Update scanner.TestCase to match new regex regime 
scanning for analytics is now handled by firebase/play-services regexes
 2019-11-02 18:54:54 +00:00
Hans-Christoph Steiner
0a8af2c55f Merge branch 'fix-readline-in-srclibs' into 'master' 
build: fix bad regexs when removing signingConfig from srclibs

See merge request fdroid/fdroidserver!686
 2019-10-25 08:18:08 +00:00
Hans-Christoph Steiner
afaa24f2fd
build: fix bad regexs when removing signingConfig from srclibs 
I went through the source of all apps in fdroiddata for examples, and found
some that use readLine() for things totally unrelated to signingConfigs.

https://gitlab.com/fdroid/fdroiddata/merge_requests/4775#note_234132902
 2019-10-23 12:44:47 +02:00
Michael Pöhn
7fa3c34e5b update tests for fixed log deployment+changelog 2019-10-15 15:19:18 +02:00
Hans-Christoph Steiner
5904aef5a7
update: parse Triple-T v2.x layout 
Thanks to the very nice example from Piwigo, included here under the GPLv3+
https://github.com/Piwigo/Piwigo-Android

https://gitlab.com/fdroid/fdroiddata/merge_requests/5579#note_225834868
 2019-10-04 14:11:40 +02:00
Hans-Christoph Steiner
3801db064a lint: improve HTTPS check 
It was missing some domains, so I added another rule.  @IzzySoft pointed
out it was redudnant, so this removes the redudant rule and fixes the
original.

https://gitlab.com/fdroid/fdroidserver/merge_requests/681/diffs#note_225263464
 2019-10-04 11:12:48 +02:00
Hans-Christoph Steiner
af4e231f7d lint: enforce HTTPS on GitHub and GitLab pages 
The always provide HTTPS, so let's enforce it.
 2019-10-04 11:12:48 +02:00
Hans-Christoph Steiner
17320c23f4 lint: don't trip up on projects with 'master' in the name 
https://gitlab.com/fdroid/fdroiddata/merge_requests/5557#note_223283359
 2019-10-04 11:12:48 +02:00
Michael Pöhn
d5ab303d83 lint: license override config option + require FSF/OSI approved licenses by default 2019-10-02 22:14:09 +02:00
Hans-Christoph Steiner
c7048f2c39 update: log errors on bad graphics, and then ignore the file 
Python PIL is not so tolerant, so bad EXIF causes crashes:

  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2088, in main
    insert_localized_app_metadata(apps)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 978, in insert_localized_app_metadata
    _strip_and_copy_image(os.path.join(root, f), destdir)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 754, in _strip_and_copy_image
    in_image = Image.open(fp)
  File "/usr/lib/python3/dist-packages/PIL/Image.py", line 2687, in open
    % (filename if filename else fp))
OSError: cannot identify image file <_io.BufferedReader name='build/org.sw24softwares.starkeverben/fastlane/metadata/android/en-US/images/featureGraphic.png'>
 2019-09-26 22:06:21 +02:00
Hans-Christoph Steiner
cf2c1d0270
run-tests needs a source of APKs to function 
The test runs on jenkins.debian.net were failing because they
specified a dir that did not exist.
 2019-09-25 12:25:39 +02:00
Hans-Christoph Steiner
25548023e0
gitlab-ci: check gradle checksums against official list 2019-09-12 14:18:07 +02:00
relan
079754c56c scanner: add a test for the local Debian Maven repo 2019-08-29 21:33:12 +03:00
Hans-Christoph Steiner
70e7e720b9
update: use graphics filename with hash in index to support caching 
Using a filename based on the hash of the contents means that the caching
algorithms for fdroidclient and browsers can safely cache the file forever
using the filename, since this guarantees that the contents will never
change for a given filename.

This does not cover screenshots, only icon.png, featureGraphic.png,
tvBanner.png, and promoGraphic.png.

fdroidserver#689
fdroid-website!453
 2019-08-29 08:23:51 +02:00
Hans-Christoph Steiner
508af00e84
update: only copy graphics and screenshots if mtime/size has changed 
Instead of copying every time, trust the filesystem to tell us when the
file has changed.
 2019-08-29 08:23:47 +02:00
Hans-Christoph Steiner
f13c41e3ba
update: remove Provides: from template.yml test case 
!654
dcf3837bcb
 2019-08-28 15:57:40 +02:00
Hans-Christoph Steiner
7090ce63dd Merge branch 'remove-provides-in-yml' into 'master' 
Remove `Provides:` in yaml metadata

Closes #676

See merge request fdroid/fdroidserver!654
 2019-08-28 13:39:59 +00:00
Michael Pöhn
8e5232076f do not delete yml metadata when raumel not installed 2019-08-27 15:38:40 +02:00
Michael Pöhn
0885303672 add additional rewritemeta test for yml 2019-08-27 15:38:40 +02:00
Michael Pöhn
3951d93196 add rewritemeta.TestCase 2019-08-27 15:38:40 +02:00
Michael Pöhn
39c589fb85 Merge branch 'update-create-template' into 'master' 
make metadata template behave well for empty values

Closes #681

See merge request fdroid/fdroidserver!657
 2019-07-24 11:02:38 +00:00
Michael Pöhn
d0368d0ad8 common add parse_androidmanifests_ignore test 2019-07-23 22:44:45 +02:00
Michael Pöhn
bad888856a checkupdates: add check_http ignore test 2019-07-23 22:01:18 +02:00
Michael Pöhn
23280b6029 checkupdates add check_http test 2019-07-23 22:01:18 +02:00
Michael Pöhn
1c7af1dc2c add test for checkupdates_app() 2019-07-23 22:01:18 +02:00
Michael Pöhn
e2fed09af1 make metadata template behaves well for empty values 2019-07-22 01:34:55 +02:00
Michael Pöhn
dcf3837bcb parse yaml: ignore (and warn) deprecated field: Provides 2019-07-15 15:45:02 +02:00
Michael Pöhn
6e48663230 test that write yaml does not write provides 2019-07-11 03:35:23 +02:00
Michael Pöhn
723bd110a6 test that yaml parsing does not accept provides 2019-07-11 03:35:23 +02:00
Hans-Christoph Steiner
051596dd0d
tests: conditionally disable tests that can't work with apksigner 
apksigner treats MD5 signatures as valid, fdroid does not.
 2019-07-10 14:35:03 +02:00
Hans-Christoph Steiner
26af94974a Merge branch 'rsync-buildslogs-to-webroot-repo' into 'master' 
build: rsync buildlogs to <webroot>/repo

See merge request fdroid/fdroidserver!651
 2019-07-10 08:55:04 +00:00
Michael Pöhn
2c87b5e6f9 deploy build logs: no timestamps 2019-07-06 16:48:56 +02:00
Michael Pöhn
f30983368c build: rsync buildlogs to <webroot>/repo 2019-07-06 16:48:56 +02:00
Hans-Christoph Steiner
a248a69692 tests: skip disabled_algorithms test when apksigner is present 
apksigner doesn't treat MD5 signatures as deprecated, so that portion of
the tests would always fail.
 2019-07-04 16:45:50 +02:00
Hans-Christoph Steiner
a9aa8788e0 tests: only run source tarball test if running from git clone 2019-07-03 22:33:15 +02:00
Hans-Christoph Steiner
aa1e958360 tests: only run hooks/pre-commit if its present (not in source tarball) 2019-07-03 20:46:30 +02:00
Hans-Christoph Steiner
a0f5ee661e tests: common.test_sign_apk requires aapt to run 2019-07-03 09:07:36 +02:00
Hans-Christoph Steiner
57b9d1e316 tests: handle when apksigner considers MD5 signatures valid 2019-07-02 22:17:06 +02:00
Michael Pöhn
66105de833 improve litecoin validation + tests 2019-05-28 11:04:43 +02:00
Michael Pöhn
06cec2041d improve bitcoin validation regex + testcases 2019-05-07 22:43:05 +02:00
Hans-Christoph Steiner
dd2f9d60f8
publish: fix stupid error in repro-signing and add integration test 
stoopid mistake in ea84014f9b reported
by @CiaranG
 2019-04-11 14:06:51 +02:00
Marcus
7272689ced Merge branch 'master' into 'master' 
Added newer ndks, gradles, latest sdk-license, and update java 1.8 version

See merge request fdroid/fdroidserver!637
 2019-04-11 11:42:18 +00:00
Michael Pöhn
67731470cc Revert "Merge branch 'write-yaml-overhaul' into 'master'" 
This reverts merge request !630
 2019-04-01 10:24:00 +00:00
Taco
457cf22361 Added newer ndks, gradles, latest sdk-license, and update java 1.8 version 2019-03-30 17:10:21 -04:00
Michael Pöhn
2683b37044 yml metadata write: do not use local functions 2019-03-19 01:01:18 +01:00
Michael Pöhn
881a79fa84 test writing all yaml fields 2019-03-18 22:45:35 +01:00
Michael Pöhn
63afc0acb5 use pyyaml for writing metadata instead of ruamel 2019-03-18 22:45:35 +01:00
Hans-Christoph Steiner
d03b121152
update: allow tests to pass when apksigner is not installed 
This is only for the v2/v3 signatures.

fdroid/fdroidserver#627
 2019-02-12 12:43:21 +01:00
Hans-Christoph Steiner
50ca3967cc update: fix tests when running without apksigner 2019-02-03 15:50:39 +01:00
Hans-Christoph Steiner
17dc231dc9 update: fix running without androguard 
Soon, we can rip out all the aapt parsing stuff, but not yet!
 2019-02-03 15:50:39 +01:00
Michael Pöhn
dd695c650e update: treat target and min sdk version as int 2019-02-01 09:56:21 +01:00
Hans-Christoph Steiner
d96f5ff660 support APK Signature V2 when apksigner is installed 
This was done with much help from @uniqx.  This is the first level of
supporting APK Signatures v1, v2, and v3.  This is enough to include
APKs with any combo of v1/v2/v3 signatures.  For this to work at all,
apksigner and androguard 3.3.3+ must be installed.

closes #399
 2019-02-01 09:17:56 +01:00
Francesco Cervigni
984d276c1b Added test aapt output files for build-tools 28.0.3 2019-01-13 19:27:02 +01:00
Michael Pöhn
31ca2092a1 yaml parsing: script build flags can now be lists 2018-11-29 21:53:37 +01:00
Michael Pöhn
c15a7508e7 write yaml script metadata as lists 2018-11-29 21:53:37 +01:00
Michael Pöhn
a21635ae2e fix metadata test case: write yaml prebuild 2018-11-29 21:53:37 +01:00
Michael Pöhn
723815a25b fix metadata.Testcase:test_rewrite_yaml_special_build_params 2018-11-29 21:53:37 +01:00
Michael Pöhn
d0a129c216 add test for parsing build field prebuild as string 2018-11-29 21:53:37 +01:00
Michael Pöhn
942de28fa5 yaml metadata: split prebuild build field to list 2018-11-29 21:53:37 +01:00
Hans-Christoph Steiner
57556aceee remove redundant open() arg: encoding='utf8' 
By default, open() returns a str:
https://docs.python.org/3/library/functions.html#open

By default, str is UTF-8:
https://docs.python.org/3/library/stdtypes.html#str

This used to matter on Python 2.x, but this code is 3.x only now.
 2018-10-19 15:01:34 +02:00
Hans-Christoph Steiner
bfdf581201 import: use valid placeholder values for versionCode/versionName 
fdroid/fdroidserver!559
closes fdroid/fdroidserver#548
 2018-10-10 16:02:34 +02:00
Michael Pöhn
67e46694d3 add test for allowing to write placeholder values to yaml metadata files 2018-10-10 16:02:34 +02:00
Hans-Christoph Steiner
3bea689f74 add another lint field type test case 2018-10-10 16:02:34 +02:00
Hans-Christoph Steiner
e33c1c74b1 tests: include info.zwanenburg.caffeinetile in index tests 2018-10-10 16:02:34 +02:00
Hans-Christoph Steiner
8b251da79f update: do not set targetSdkVersion if the APK is missing it 
699b3e4c69 got it wrong for targetSdkVersion.
Also, one confusing thing is that aapt outputs "sdkVersion: '3'" for
com.politedroid_3.apk but no "sdkVersion:" for no.min.target.sdk_987.apk.
F-Droid never really supported running on android-1 or android-2, so it
seems pointless to debug support for them.
 2018-10-10 15:29:02 +02:00
Hans-Christoph Steiner
85993eb2f8 lint: check fields for proper type, e.g. list vs. string 
fdroid/fdroidserver#578
 2018-10-09 23:31:24 +02:00
Hans-Christoph Steiner
6b57cb6b7c fix strict Application ID checks 
* upper case letters are allowed at all positions
* there must be a "." separator
 2018-09-24 17:07:27 +02:00
Hans-Christoph Steiner
11d46072ab use androguard primitives to speed up finding debuggable flag 
androguard parses the whole APK before handing the instance back, this uses
the primitives to just find the <application android:debuggable=""> value,
then stop parsing.

#557
 2018-09-21 14:56:46 +02:00
Hans-Christoph Steiner
a3cecc16a3 use partial androguard binary XML parsing to speed up APK ID lookup 
Normally, androguard parses the entire APK before it is possible to get any
values from it.  This uses androguard primitives to only attempt to parse
the AndroidManifest.xml, then to quit as soon as it gets what it needs.
This greatly speeds up the parsing (1 minute vs 60 minutes).

fdroid/fdroidserver#557
 2018-09-21 14:56:46 +02:00
Hans-Christoph Steiner
4c4da3d176 update: remove 'SET' debug antifeature, closes #565 
oops, this slipped in in 3011953d0e

Bump the metadata version to purge the cache
 2018-09-18 10:57:10 +02:00
Hans-Christoph Steiner
92c4f048de update: test scan_apk() with both aapt and androguard 
#568
 2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner
807bf3d26b build: reuse common methods for getting metadata from APKs 
This splits out the code that gets the list of native ABIs supported, then
uses the standard methods for the rest.
 2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner
487c4d02f3 handle package: line output from aapt v28 
fdroid/fdroiddata!3484
fdroid/fdroiddata!3562
fdroid/fdroidserver!548
 2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner
d1acef0405 tests: generate aapt output for every version to make tests easy 2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner
9d12b1dc61 add strict, tested validation of Android/F-Droid package names 
Android has stricter rules than Java for Package Names, but anything the
Python regex thinks is valid must be valid according to Java's rules too.

https://developer.android.com/studio/build/application-id
 2018-09-07 14:17:39 +02:00
Hans-Christoph Steiner
5d161cc9fd validate appid when reading metadata files 
The metadata file must be named after the Application ID of the app it is
describing, and Android Application IDs must be valid Java Package Names.
 2018-09-03 22:56:08 +02:00
Hans-Christoph Steiner
3011953d0e convert apkcache from pickle to JSON 
pickle can serialize executable code, while JSON is only ever pure data.
The APK cache is only ever pure data, so no need for the security risks of
pickle.  For example, if some malicious thing gets write access on the
`fdroid update` machine, it can write out a custom tmp/apkcache which would
then be executed.  That is not possible with JSON.

This does just ignore any existing cache and rebuilds from scratch. That is
so we don't need to maintain pickle anywhere, and to ensure there are no
glitches from a conversion from pickle to JSON.

closes #163
 2018-09-03 18:07:40 +02:00
Hans-Christoph Steiner
4d13a904f3 use defusedxml to avoid DoS attacks while loading XML 2018-08-29 17:44:54 +02:00