1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-16 11:50:10 +01:00
Commit Graph

747 Commits

Author SHA1 Message Date
Hans-Christoph Steiner
b63e9e68c5
move all test metadata files to .yml format
for f in metadata/*.txt; do ../fdroid rewritemeta --to yml $(basename $f | sed 's,\.txt,,'); done
2020-06-10 09:56:31 +02:00
Hans-Christoph Steiner
13d9a122bf metadata: validate STRING and INT build field types
This converts float/int to string for things like commit: or versionName:.
For versionCode, which must be an integer, it throws an exception if the
data is any other type.
2020-06-02 22:06:01 +02:00
Hans-Christoph Steiner
7a84679b0d gradlew-fdroid: fix parsing when files have Windows linefeeds
The new test file should have Windows linefeeds, if I got the git config
correct.
2020-05-29 21:55:10 +02:00
Hans-Christoph Steiner
fc885c9b5c scanner: test for get_gradle_compile_commands() 2020-05-27 22:09:52 +02:00
Hans-Christoph Steiner
95c3ab2454 skip yamllint test if yamllint is not installed
!721
2020-05-27 22:07:17 +02:00
Hans-Christoph Steiner
67332d83a5 scanner: add --json option for outputting machine readable results
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot

This is a second attempt with tests for how `fdroid build` calls the
scanner functions. closes #771.  It was previously merged in !748 then
reverted in 68c072c72e
2020-05-27 08:35:14 +02:00
Hans-Christoph Steiner
5b9944fcde add test for things fdroid build cleans
This needs a lot of mocking because build.build_local() is a
gianormous single function.
2020-05-26 09:48:55 +02:00
Marcus Hoffmann
68c072c72e Revert "scanner: add --json option for outputting machine readable results"
This reverts commit cdaf62e5d9.

See: https://gitlab.com/fdroid/fdroidserver/-/merge_requests/748#note_347769371
2020-05-24 10:24:40 +02:00
Hans-Christoph Steiner
cdaf62e5d9 scanner: add --json option for outputting machine readable results
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot
2020-05-14 21:58:48 +02:00
Hans-Christoph Steiner
a0e3b01e94 metadata: parsed srclibs must always return a dict as the container 2020-05-14 16:28:13 +02:00
Michael Pöhn
b076e8cba7 add srclib spec parser 2020-04-25 17:22:48 +02:00
Michael Pöhn
d24484a950 simple testcase for common.run_yamllint 2020-04-24 15:47:47 +02:00
Michael Pöhn
4e69ff582f run yamllint when parsing failed and also on fdroid lint runs 2020-04-24 15:47:31 +02:00
Michael Pöhn
5741e6930b yml srclibs: support Subdir as list 2020-04-16 11:31:59 +02:00
Michael Pöhn
1ac7d612b1 yml srclibs: support multiline Prepare 2020-04-16 11:31:59 +02:00
Michael Pöhn
58776da694 get tests working 2020-04-16 11:31:59 +02:00
Michael Pöhn
ee3d8d2f18 srclib: yml parsing + tests for yml and txt 2020-04-16 11:31:59 +02:00
Michael Pöhn
fa7885063b scrlib: add test for getsrclib 2020-04-16 11:31:59 +02:00
Michael Pöhn
286220fe11 srclib: add test for getsrclibvcs 2020-04-16 11:31:59 +02:00
Michael Pöhn
2c86d38028 srclib: no spaces in dict key 2020-04-16 11:31:59 +02:00
Michael Pöhn
c6a97939f1 rename parse_srclib to parese_txt_srclib + test case 2020-04-16 11:31:59 +02:00
Hans-Christoph Steiner
b7901952a1 deploy: make androidobservatory and virustotal functions reusable
This should not change the logic at all, just make the loop runs into
standalone functions.
2020-03-11 22:13:39 +01:00
Hans-Christoph Steiner
ab2291475b import: mv reusable functions to common.py to avoid import_proxy.py
import is a strict keyword in Python, so it is not possible to import a
module called 'import', even with things like:

* import fdroidserver.import
* from fdroidserver import import
2020-03-11 13:41:13 +01:00
Marcus
9d24f2e4a7 add opencollective metadata and index field 2020-03-10 14:56:03 +00:00
Hans-Christoph Steiner
202291d66c
integration test for creating and deploying status JSON files
----------------------------
2020-02-19 14:50:37 +01:00
Hans-Christoph Steiner
4bb590b6e5
publish: write status JSON 2020-02-19 14:50:24 +01:00
Hans-Christoph Steiner
271b74af7d fix remove_signing_keys() for Kotlin gradles files (*.gradle.kts) 2020-02-13 22:32:51 +01:00
Hans-Christoph Steiner
3de2d0f56f add basic test suite for gradlew-fdroid
!707
fdroiddata#6216

The se.manyver app is licensed MPL, the files came from:
81d247a6cd
2020-02-13 22:32:51 +01:00
Hans-Christoph Steiner
83ffeb855f prefer build.gradle with Android Plugin as source of package/version/code
These days, the location that overrides all the others is in the android{}
block of the build.gradle file that loads the com.android.application
plugin.  So this should be the preferred place to read these values.

test files GPL licensed: https://github.com/Integreight/1Sheeld-Android-App
2020-02-13 22:32:51 +01:00
Hans-Christoph Steiner
1153ac24fd import: overhaul URL validation to use urllib.parse
Python provides us a lovely URL parser with some level of validation built
in.  The parsed URL is then much easier to validate.
2020-02-13 13:51:52 +01:00
Hans-Christoph Steiner
e9a6c84efd import: split URL parsing from code cloning
This makes things testable and easier to follow.
2020-02-13 13:51:52 +01:00
Hans-Christoph Steiner
bfe587979d import: make it work most of the time with git repos
This includes real tests too.
2020-02-13 13:51:52 +01:00
Hans-Christoph Steiner
5fb368916f
index: fix no JAR test case that fails on example.org
http://example.org/index-v1.jar now returns the HTTP header
"Content-Encoding: gzip" but then the reply is plain HTML. That
triggers a ContentDecodingError instead of an HTTPError, so this
changes the test to success on any RequestsException.
2020-02-05 20:33:40 +01:00
Hans-Christoph Steiner
427856b5fd
tests: use same mirrors order in all tests
This makes it so running `../fdroid update --nosign --pretty` in tests/ no
longer creates a diff in the tests/*/index* files.  It matches the order
set in tests/run-tests.
2020-01-31 15:38:10 +01:00
Hans-Christoph Steiner
6ce22bff56
update: strip newlines from name/summary/video entries
These entries are hardcoded as a single line in all the app stores, so
newlines should be stripped to get the data simple to use.  This is in
contrast with the on-disk format for Fastlane and Triple-T, which includes
a newline in the title.txt and short_description.txt files.  I think all
files in those systems are normalized to end in a newline.
2020-01-31 15:37:59 +01:00
Hans-Christoph Steiner
fdede914cd
tests: properly name CheckupdatesTest class 2020-01-31 15:37:32 +01:00
Hans-Christoph Steiner
3df276cc3c
fix all bandit B310 urllib_urlopen
"Audit url open for permitted schemes. Allowing use of ‘file:’’ or custom
schemes is often unexpected."

https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b310-urllib-urlopen
2020-01-31 15:37:30 +01:00
Marcus Hoffmann
8961485125 remove dscanner subcommand
It's unused and unmaintained. It could  potentially be revived as a
plugin at a later point.
2020-01-31 14:17:12 +01:00
Michael Pöhn
f2a80ffa3d fix tests on old python version 2020-01-28 11:43:09 +01:00
Michael Pöhn
b25eeb66a1 fix code style 2020-01-28 11:43:09 +01:00
Michael Pöhn
f21481ca81 add some simple tests for main 2020-01-28 11:43:09 +01:00
Hans-Christoph Steiner
e76a0c9d6a
git_mirror_size_limit config option to set max git mirror size
GitHub and GitLab have some kinds of limits on how big a git repo can be,
this makes that option configurable.  This also is very useful for tests.
2020-01-14 11:36:08 +01:00
Gerhard Olsson
cf9bff1d56 AutoUpdateMode: Do not include + in suffix
Add suffix to version only
2020-01-02 13:27:12 +00:00
Marcus Hoffmann
83526e09a3 update: fix unarchiving of allowed disabled algorithm
6d0b1bbe6fae0909683f2c6a154515bc4bfcb674 didn't handle the
allow_disabled_algorithm case at all, so we add it back.

This additionally fixes a (previously existing) bug where setting
allow_disabled_algorithms to True didn't move apks back from archive to
repo. Introduce a new test for this.

The disabled_algorithm archiving logic is still all over the place so
ideally that needs a future refactor.
2020-01-01 15:25:43 +01:00
Marcus Hoffmann
7f6efa74f5 tests: add test for correctly handling CVC when archiving 2019-12-24 17:35:54 +01:00
Hans-Christoph Steiner
7d40e89341
checkupdates: split out vercode parsing into testable function 2019-12-03 23:51:48 +01:00
Jochen Sprickerhof
bbee2cf707 Add unit test for string_is_integer() 2019-12-03 21:49:44 +01:00
Mike Hardy
3648ef1f2e Update scanner.TestCase to match new regex regime
scanning for analytics is now handled by firebase/play-services regexes
2019-11-02 18:54:54 +00:00
Hans-Christoph Steiner
0a8af2c55f Merge branch 'fix-readline-in-srclibs' into 'master'
build: fix bad regexs when removing signingConfig from srclibs

See merge request fdroid/fdroidserver!686
2019-10-25 08:18:08 +00:00
Hans-Christoph Steiner
afaa24f2fd
build: fix bad regexs when removing signingConfig from srclibs
I went through the source of all apps in fdroiddata for examples, and found
some that use readLine() for things totally unrelated to signingConfigs.

https://gitlab.com/fdroid/fdroiddata/merge_requests/4775#note_234132902
2019-10-23 12:44:47 +02:00
Michael Pöhn
7fa3c34e5b update tests for fixed log deployment+changelog 2019-10-15 15:19:18 +02:00
Hans-Christoph Steiner
5904aef5a7
update: parse Triple-T v2.x layout
Thanks to the very nice example from Piwigo, included here under the GPLv3+
https://github.com/Piwigo/Piwigo-Android

https://gitlab.com/fdroid/fdroiddata/merge_requests/5579#note_225834868
2019-10-04 14:11:40 +02:00
Hans-Christoph Steiner
3801db064a lint: improve HTTPS check
It was missing some domains, so I added another rule.  @IzzySoft pointed
out it was redudnant, so this removes the redudant rule and fixes the
original.

https://gitlab.com/fdroid/fdroidserver/merge_requests/681/diffs#note_225263464
2019-10-04 11:12:48 +02:00
Hans-Christoph Steiner
af4e231f7d lint: enforce HTTPS on GitHub and GitLab pages
The always provide HTTPS, so let's enforce it.
2019-10-04 11:12:48 +02:00
Hans-Christoph Steiner
17320c23f4 lint: don't trip up on projects with 'master' in the name
https://gitlab.com/fdroid/fdroiddata/merge_requests/5557#note_223283359
2019-10-04 11:12:48 +02:00
Michael Pöhn
d5ab303d83 lint: license override config option + require FSF/OSI approved licenses by default 2019-10-02 22:14:09 +02:00
Hans-Christoph Steiner
c7048f2c39 update: log errors on bad graphics, and then ignore the file
Python PIL is not so tolerant, so bad EXIF causes crashes:

  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2088, in main
    insert_localized_app_metadata(apps)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 978, in insert_localized_app_metadata
    _strip_and_copy_image(os.path.join(root, f), destdir)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 754, in _strip_and_copy_image
    in_image = Image.open(fp)
  File "/usr/lib/python3/dist-packages/PIL/Image.py", line 2687, in open
    % (filename if filename else fp))
OSError: cannot identify image file <_io.BufferedReader name='build/org.sw24softwares.starkeverben/fastlane/metadata/android/en-US/images/featureGraphic.png'>
2019-09-26 22:06:21 +02:00
Hans-Christoph Steiner
cf2c1d0270
run-tests needs a source of APKs to function
The test runs on jenkins.debian.net were failing because they
specified a dir that did not exist.
2019-09-25 12:25:39 +02:00
Hans-Christoph Steiner
25548023e0
gitlab-ci: check gradle checksums against official list 2019-09-12 14:18:07 +02:00
relan
079754c56c scanner: add a test for the local Debian Maven repo 2019-08-29 21:33:12 +03:00
Hans-Christoph Steiner
70e7e720b9
update: use graphics filename with hash in index to support caching
Using a filename based on the hash of the contents means that the caching
algorithms for fdroidclient and browsers can safely cache the file forever
using the filename, since this guarantees that the contents will never
change for a given filename.

This does not cover screenshots, only icon.png, featureGraphic.png,
tvBanner.png, and promoGraphic.png.

fdroidserver#689
fdroid-website!453
2019-08-29 08:23:51 +02:00
Hans-Christoph Steiner
508af00e84
update: only copy graphics and screenshots if mtime/size has changed
Instead of copying every time, trust the filesystem to tell us when the
file has changed.
2019-08-29 08:23:47 +02:00
Hans-Christoph Steiner
f13c41e3ba
update: remove Provides: from template.yml test case
!654
dcf3837bcb
2019-08-28 15:57:40 +02:00
Hans-Christoph Steiner
7090ce63dd Merge branch 'remove-provides-in-yml' into 'master'
Remove `Provides:` in yaml metadata

Closes #676

See merge request fdroid/fdroidserver!654
2019-08-28 13:39:59 +00:00
Michael Pöhn
8e5232076f do not delete yml metadata when raumel not installed 2019-08-27 15:38:40 +02:00
Michael Pöhn
0885303672 add additional rewritemeta test for yml 2019-08-27 15:38:40 +02:00
Michael Pöhn
3951d93196 add rewritemeta.TestCase 2019-08-27 15:38:40 +02:00
Michael Pöhn
39c589fb85 Merge branch 'update-create-template' into 'master'
make metadata template behave well for empty values

Closes #681

See merge request fdroid/fdroidserver!657
2019-07-24 11:02:38 +00:00
Michael Pöhn
d0368d0ad8 common add parse_androidmanifests_ignore test 2019-07-23 22:44:45 +02:00
Michael Pöhn
bad888856a checkupdates: add check_http ignore test 2019-07-23 22:01:18 +02:00
Michael Pöhn
23280b6029 checkupdates add check_http test 2019-07-23 22:01:18 +02:00
Michael Pöhn
1c7af1dc2c add test for checkupdates_app() 2019-07-23 22:01:18 +02:00
Michael Pöhn
e2fed09af1 make metadata template behaves well for empty values 2019-07-22 01:34:55 +02:00
Michael Pöhn
dcf3837bcb parse yaml: ignore (and warn) deprecated field: Provides 2019-07-15 15:45:02 +02:00
Michael Pöhn
6e48663230 test that write yaml does not write provides 2019-07-11 03:35:23 +02:00
Michael Pöhn
723bd110a6 test that yaml parsing does not accept provides 2019-07-11 03:35:23 +02:00
Hans-Christoph Steiner
051596dd0d
tests: conditionally disable tests that can't work with apksigner
apksigner treats MD5 signatures as valid, fdroid does not.
2019-07-10 14:35:03 +02:00
Hans-Christoph Steiner
26af94974a Merge branch 'rsync-buildslogs-to-webroot-repo' into 'master'
build: rsync buildlogs to <webroot>/repo

See merge request fdroid/fdroidserver!651
2019-07-10 08:55:04 +00:00
Michael Pöhn
2c87b5e6f9 deploy build logs: no timestamps 2019-07-06 16:48:56 +02:00
Michael Pöhn
f30983368c build: rsync buildlogs to <webroot>/repo 2019-07-06 16:48:56 +02:00
Hans-Christoph Steiner
a248a69692 tests: skip disabled_algorithms test when apksigner is present
apksigner doesn't treat MD5 signatures as deprecated, so that portion of
the tests would always fail.
2019-07-04 16:45:50 +02:00
Hans-Christoph Steiner
a9aa8788e0 tests: only run source tarball test if running from git clone 2019-07-03 22:33:15 +02:00
Hans-Christoph Steiner
aa1e958360 tests: only run hooks/pre-commit if its present (not in source tarball) 2019-07-03 20:46:30 +02:00
Hans-Christoph Steiner
a0f5ee661e tests: common.test_sign_apk requires aapt to run 2019-07-03 09:07:36 +02:00
Hans-Christoph Steiner
57b9d1e316 tests: handle when apksigner considers MD5 signatures valid 2019-07-02 22:17:06 +02:00
Michael Pöhn
66105de833 improve litecoin validation + tests 2019-05-28 11:04:43 +02:00
Michael Pöhn
06cec2041d improve bitcoin validation regex + testcases 2019-05-07 22:43:05 +02:00
Hans-Christoph Steiner
dd2f9d60f8
publish: fix stupid error in repro-signing and add integration test
stoopid mistake in ea84014f9b reported
by @CiaranG
2019-04-11 14:06:51 +02:00
Marcus
7272689ced Merge branch 'master' into 'master'
Added newer ndks, gradles, latest sdk-license, and update java 1.8 version

See merge request fdroid/fdroidserver!637
2019-04-11 11:42:18 +00:00
Michael Pöhn
67731470cc Revert "Merge branch 'write-yaml-overhaul' into 'master'"
This reverts merge request !630
2019-04-01 10:24:00 +00:00
Taco
457cf22361 Added newer ndks, gradles, latest sdk-license, and update java 1.8 version 2019-03-30 17:10:21 -04:00
Michael Pöhn
2683b37044 yml metadata write: do not use local functions 2019-03-19 01:01:18 +01:00
Michael Pöhn
881a79fa84 test writing all yaml fields 2019-03-18 22:45:35 +01:00
Michael Pöhn
63afc0acb5 use pyyaml for writing metadata instead of ruamel 2019-03-18 22:45:35 +01:00
Hans-Christoph Steiner
d03b121152
update: allow tests to pass when apksigner is not installed
This is only for the v2/v3 signatures.

fdroid/fdroidserver#627
2019-02-12 12:43:21 +01:00
Hans-Christoph Steiner
50ca3967cc update: fix tests when running without apksigner 2019-02-03 15:50:39 +01:00
Hans-Christoph Steiner
17dc231dc9 update: fix running without androguard
Soon, we can rip out all the aapt parsing stuff, but not yet!
2019-02-03 15:50:39 +01:00
Michael Pöhn
dd695c650e update: treat target and min sdk version as int 2019-02-01 09:56:21 +01:00
Hans-Christoph Steiner
d96f5ff660 support APK Signature V2 when apksigner is installed
This was done with much help from @uniqx.  This is the first level of
supporting APK Signatures v1, v2, and v3.  This is enough to include
APKs with any combo of v1/v2/v3 signatures.  For this to work at all,
apksigner and androguard 3.3.3+ must be installed.

closes #399
2019-02-01 09:17:56 +01:00
Francesco Cervigni
984d276c1b Added test aapt output files for build-tools 28.0.3 2019-01-13 19:27:02 +01:00
Michael Pöhn
31ca2092a1 yaml parsing: script build flags can now be lists 2018-11-29 21:53:37 +01:00
Michael Pöhn
c15a7508e7 write yaml script metadata as lists 2018-11-29 21:53:37 +01:00
Michael Pöhn
a21635ae2e fix metadata test case: write yaml prebuild 2018-11-29 21:53:37 +01:00
Michael Pöhn
723815a25b fix metadata.Testcase:test_rewrite_yaml_special_build_params 2018-11-29 21:53:37 +01:00
Michael Pöhn
d0a129c216 add test for parsing build field prebuild as string 2018-11-29 21:53:37 +01:00
Michael Pöhn
942de28fa5 yaml metadata: split prebuild build field to list 2018-11-29 21:53:37 +01:00
Hans-Christoph Steiner
57556aceee remove redundant open() arg: encoding='utf8'
By default, open() returns a str:
https://docs.python.org/3/library/functions.html#open

By default, str is UTF-8:
https://docs.python.org/3/library/stdtypes.html#str

This used to matter on Python 2.x, but this code is 3.x only now.
2018-10-19 15:01:34 +02:00
Hans-Christoph Steiner
bfdf581201 import: use valid placeholder values for versionCode/versionName
fdroid/fdroidserver!559
closes fdroid/fdroidserver#548
2018-10-10 16:02:34 +02:00
Michael Pöhn
67e46694d3 add test for allowing to write placeholder values to yaml metadata files 2018-10-10 16:02:34 +02:00
Hans-Christoph Steiner
3bea689f74 add another lint field type test case 2018-10-10 16:02:34 +02:00
Hans-Christoph Steiner
e33c1c74b1 tests: include info.zwanenburg.caffeinetile in index tests 2018-10-10 16:02:34 +02:00
Hans-Christoph Steiner
8b251da79f update: do not set targetSdkVersion if the APK is missing it
699b3e4c69 got it wrong for targetSdkVersion.
Also, one confusing thing is that aapt outputs "sdkVersion: '3'" for
com.politedroid_3.apk but no "sdkVersion:" for no.min.target.sdk_987.apk.
F-Droid never really supported running on android-1 or android-2, so it
seems pointless to debug support for them.
2018-10-10 15:29:02 +02:00
Hans-Christoph Steiner
85993eb2f8 lint: check fields for proper type, e.g. list vs. string
fdroid/fdroidserver#578
2018-10-09 23:31:24 +02:00
Hans-Christoph Steiner
6b57cb6b7c fix strict Application ID checks
* upper case letters are allowed at all positions
* there must be a "." separator
2018-09-24 17:07:27 +02:00
Hans-Christoph Steiner
11d46072ab use androguard primitives to speed up finding debuggable flag
androguard parses the whole APK before handing the instance back, this uses
the primitives to just find the <application android:debuggable=""> value,
then stop parsing.

#557
2018-09-21 14:56:46 +02:00
Hans-Christoph Steiner
a3cecc16a3 use partial androguard binary XML parsing to speed up APK ID lookup
Normally, androguard parses the entire APK before it is possible to get any
values from it.  This uses androguard primitives to only attempt to parse
the AndroidManifest.xml, then to quit as soon as it gets what it needs.
This greatly speeds up the parsing (1 minute vs 60 minutes).

fdroid/fdroidserver#557
2018-09-21 14:56:46 +02:00
Hans-Christoph Steiner
4c4da3d176 update: remove 'SET' debug antifeature, closes #565
oops, this slipped in in 3011953d0e

Bump the metadata version to purge the cache
2018-09-18 10:57:10 +02:00
Hans-Christoph Steiner
92c4f048de update: test scan_apk() with both aapt and androguard
#568
2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner
807bf3d26b build: reuse common methods for getting metadata from APKs
This splits out the code that gets the list of native ABIs supported, then
uses the standard methods for the rest.
2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner
487c4d02f3 handle package: line output from aapt v28
fdroid/fdroiddata!3484
fdroid/fdroiddata!3562
fdroid/fdroidserver!548
2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner
d1acef0405 tests: generate aapt output for every version to make tests easy 2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner
9d12b1dc61 add strict, tested validation of Android/F-Droid package names
Android has stricter rules than Java for Package Names, but anything the
Python regex thinks is valid must be valid according to Java's rules too.

https://developer.android.com/studio/build/application-id
2018-09-07 14:17:39 +02:00
Hans-Christoph Steiner
5d161cc9fd validate appid when reading metadata files
The metadata file must be named after the Application ID of the app it is
describing, and Android Application IDs must be valid Java Package Names.
2018-09-03 22:56:08 +02:00
Hans-Christoph Steiner
3011953d0e convert apkcache from pickle to JSON
pickle can serialize executable code, while JSON is only ever pure data.
The APK cache is only ever pure data, so no need for the security risks of
pickle.  For example, if some malicious thing gets write access on the
`fdroid update` machine, it can write out a custom tmp/apkcache which would
then be executed.  That is not possible with JSON.

This does just ignore any existing cache and rebuilds from scratch. That is
so we don't need to maintain pickle anywhere, and to ensure there are no
glitches from a conversion from pickle to JSON.

closes #163
2018-09-03 18:07:40 +02:00
Hans-Christoph Steiner
4d13a904f3 use defusedxml to avoid DoS attacks while loading XML 2018-08-29 17:44:54 +02:00
Hans-Christoph Steiner
4503e7a92a replace unneeded eval() call and support negative versionCodes 2018-08-29 17:24:04 +02:00
Hans-Christoph Steiner
05be4bc814 Merge branch '471-lint-check-unknown-keys' into 'master'
check for unknown app fields and build flags when parsing yml

Closes #471

See merge request fdroid/fdroidserver!554
2018-08-14 10:50:59 +00:00
Hans-Christoph Steiner
53bd5acb09 Merge branch 'make-python-tests-run-independent-of-cwd' into 'master'
Make python tests run independent of cwd

See merge request fdroid/fdroidserver!556
2018-08-14 10:34:47 +00:00
Michael Pöhn
82563850b7 fix test metadata case for Builds app field 2018-08-14 12:31:12 +02:00
Michael Pöhn
6a06944945 better exception and waring message when yaml metadata checks fail; build flags check optional 2018-08-14 12:31:12 +02:00
Michael Pöhn
e83221eb03 improve yaml build flag check cost; added unit test for unknown app fields, builds flags checks when parsing yaml 2018-08-14 12:31:12 +02:00
Michael Pöhn
7d5f4d3ab9 added cwd setup boiler-plate to *.TestCase where missing 2018-08-14 10:34:13 +02:00
Michael Pöhn
f153a61277 made build.TestCase run independant of cwd 2018-08-14 10:10:19 +02:00
Michael Pöhn
37e0dce73b made index.TestCase run independant of cwd 2018-08-14 10:08:03 +02:00
Michael Pöhn
edb8d105b0 made update.TestCase run independent of cwd 2018-08-14 10:02:15 +02:00
Hans-Christoph Steiner
1b37b521c8 common: fix tests when running without any Android SDK installed
fdroid/fdroidserver!552
fdroid/fdroidserver#541
2018-08-07 14:51:42 +02:00
Michael Pöhn
55466f1166 fix fixtures in tests/metadata/dump 2018-08-07 00:02:07 +02:00
Michael Pöhn
5d5617bf40 fix: minor bugs in testcases for: common, metadata, update 2018-08-06 23:53:49 +02:00
Michael Pöhn
28ec93c587 fix: run all testcase in tests/run-tests 2018-08-06 23:53:49 +02:00
Hans-Christoph Steiner
64b999ca19 publish: test case that successfully verifies based on Binaries:
This is a stupid test of the process just to exercise the most basic path.
It should always succeed since it is verifying two copes of the same APK.
2018-07-12 23:53:05 +02:00
Hans-Christoph Steiner
b03c4a7353 NoSourceSince Anti-Feature should only be added to index
Before, it was being added to the metadata, so that rewritemeta would
output it.

fdroid/fdroidserver!529
#508
2018-07-10 23:32:07 +02:00
Hans-Christoph Steiner
669401640a tests: only run gpgsign tests if gpg is present and executable
This allows the full test suite to run in Debian/Ubuntu autopkgtest with
only the Depends: installed, and nothing from Recommends:.
2018-06-25 10:52:12 +02:00
Michael Pöhn
752f6a4ba2 applied review suggestions for deploying (build) logs 2018-06-19 15:18:13 +02:00
Michael Pöhn
88e64df3ef deploying build logs to server after each individual build run 2018-06-19 12:24:52 +02:00
Michael Pöhn
4c53c71fcf added helper function for uploading build logs with rsync 2018-06-19 12:24:52 +02:00
Michael Pöhn
af980fbe7e added tests for update_serverwebroot 2018-06-19 12:24:52 +02:00
Michael Pöhn
429bc21da7 make common.TestCase run independant of cwd 2018-06-19 12:24:52 +02:00
Hans-Christoph Steiner
ff90c0246e fix PEP8 W605 invalid escape sequence
Python 3.7 will get a lot stricter with escape sequences.  They must be
valid.

* https://lintlyci.github.io/Flake8Rules/rules/W605.html
* https://docs.python.org/3/whatsnew/3.6.html#deprecated-python-behavior
2018-05-29 13:51:47 +02:00
Hans-Christoph Steiner
e180e03a5a tests: fix index-v1.json unicode test 2018-05-28 09:29:11 +02:00
Hans-Christoph Steiner
91068d8e00 tests: rename unicode test file to fix "file name too long" errors
pristine-tar and mkdir barf on this file.
2018-05-25 17:27:58 +02:00
Hans-Christoph Steiner
5ff1b5ef37 verify: exit with error code if any APK fails to verify 2018-05-25 11:56:13 +02:00
Hans-Christoph Steiner
0a15a9b65b tests: fix and test fdroid deploy 2018-05-25 10:32:09 +02:00
Hans-Christoph Steiner
1daa1063c5 tests: keep strerr totally clean on successful runs
Debian's autopkgtest defaults to considering any output on stderr as a sign
that the tests failed.  it is simple to make this achieve that standard, so
it seems worth it for now at least.
2018-05-25 10:32:01 +02:00
Michael Pöhn
6b1f242d25 added tests for common.calculate_math_string 2018-05-22 13:24:37 +02:00
Marcus Hoffmann
a4b6549927
tests: adapt to spdx v4.0.0 2018-05-16 22:49:57 +02:00
Hans-Christoph Steiner
f0795cc092 tests: git repo mirror tests require git >= 2.3 2018-05-14 15:25:33 +02:00
Hans-Christoph Steiner
2d3e7ef950 tests: allow tests to run without any Android SDK installed 2018-05-14 15:25:33 +02:00
Hans-Christoph Steiner
43c61aecee fix pylint Unused argument 2018-05-14 15:25:33 +02:00
Hans-Christoph Steiner
54b21a6d22 move pylint run to standalone gitlab-ci job 2018-05-14 15:25:33 +02:00
Hans-Christoph Steiner
b3f45ca024 split pip tests to run separately for Ubuntu/trusty and Debian/stretch
Trying to reuse the pip test run is now a lot more work than just
writing it cutstom for each target platform.

Ubuntu/trusty does not have androguard 3.1.x nor aapt >= 26.x, so
using trusty's aapt will always have errors scraping some names from
APKs.  This continues to use Google's binary.

`apt-get update` is now disabled by default by Travis, this adds it
back so that the latest files from the PPA are used.

Here's one example of an issue with pip on trusty:
https://github.com/requests/requests/issues/4006

    pip3 install --quiet -e /home/travis/build/fdroidtravis/fdroidserver
    Exception:
    Traceback (most recent call last):
      File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2482, in _dep_map
        return self.__dep_map
      File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2344, in __getattr__
        raise AttributeError(attr)
    AttributeError: _DistInfoDistribution__dep_map

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python3.4/site-packages/pip/basecommand.py", line 122, in main
        status = self.run(options, args)
      File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python3.4/site-packages/pip/commands/install.py", line 278, in run
        requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
      File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python3.4/site-packages/pip/req.py", line 1266, in prepare_files
        req_to_install.extras):
      File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2291, in requires
        dm = self._dep_map
      File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2484, in _dep_map
        self.__dep_map = self._compute_dependencies()
      File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2517, in _compute_dependencies
        common = frozenset(reqs_for_extra(None))
      File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2514, in reqs_for_extra
        if req.marker_fn(override={'extra':extra}):
      File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/_markerlib/markers.py", line 113, in marker_fn
        return eval(compiled_marker, environment)
      File "<environment marker>", line 1, in <module>
    NameError: name 'platform_system' is not defined
2018-05-14 15:25:33 +02:00
Hans-Christoph Steiner
77f7ca2f25 tests: allow the full suite to run without aapt, only androguard 2018-05-04 15:11:25 +02:00
Hans-Christoph Steiner
9cc8826a79 tests: allow running with only androguard, no Android SDK 2018-05-04 15:11:25 +02:00
J-Jamet
d6f1de2649 Fix applicationIdSuffix / versionNameSuffix #455 2018-05-04 12:24:16 +02:00
Hans-Christoph Steiner
27a5cce832 implement common.get_apk_id() using androguard 2018-05-03 13:46:42 +02:00
Hans-Christoph Steiner
98a2f70e38 fix intermittent test failure
For some reason, the parser stopped working intermittently, even
though the format has been the same since aapt 23 or earlier.  Then
also, some of the test cases pointed to symlinks that were no longer
generated, and one test app now has a blank versionName.

Strange that this wasn't caught in the gitlab-ci runs.  !484

FAIL: test_get_api_id_aapt (__main__.CommonTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./common.TestCase", line 578, in testA_get_api_id_aapt
    self.assertEqual(versionName, vn)
AssertionError: '0.1' != "0.1' platformBuildVersionName='4.3.1-1425645"
- 0.1
+ 0.1' platformBuildVersionName='4.3.1-1425645
2018-05-03 13:46:42 +02:00
Hans-Christoph Steiner
0e5ed1bb11 reusable method for checking if a value is a resId or not
I wrote this for the previous commit, but since aapt does not recognize
SDK Version values that are string resources, I made this do the same thing
2018-04-26 15:19:41 +02:00
Hans-Christoph Steiner
e17815e9f0 handle bad SDK Version values in APKs
Even though it is invalid to have *SdkVersion in AndroidManifest.xml set as
anything but an integer, sometimes people manage to get something in there.
`fdroid update` needs to be able to handle all that.

* https://developer.android.com/guide/topics/manifest/uses-sdk-element#min
* https://gitlab.com/souch/SMSbypass/blob/v0.9/app/src/main/AndroidManifest.xml#L29
* https://gitlab.com/souch/SMSbypass/blob/v0.9/app/src/main/res/values/strings.xml#L27

admin#65
2018-04-26 15:19:41 +02:00
Hans-Christoph Steiner
74c6555c71 index: ensure index.xml timestamp is in UTC, closes #481
use verbose but safe syntax so this works with all versions of sed, e.g. osx
2018-04-25 16:35:22 +02:00
Hans-Christoph Steiner
de35f1b05b update: handle APKs with a blank versionName
Instead of just crashing, first try to use the versionName as written in the
build metadata, otherwise just let it be blank.  A blank versionName will
cause fdroidclient < 1.3 to crash.  Blank versionNames are not allowed in
the .txt metadata format, only .yml.

closes #477
closes #478
closes fdroidclient#1416
closes fdroidclient#1417
closes fdroidclient#1418
fdroiddata!3061
2018-04-17 12:36:49 +02:00
Hans-Christoph Steiner
9bf9159a77 update: use only 7 chars of SHA256 for non-APK version name
#460
2018-04-17 12:36:49 +02:00
Hans-Christoph Steiner
74c0e98d82 update index metadata tests for 673a3517d3 2018-04-16 11:47:48 +02:00
Hans-Christoph Steiner
673a3517d3 add test of Android TV "activity banner" graphic
!491

https://developer.android.com/guide/topics/manifest/activity-element.html#banner
https://developer.android.com/training/tv/start/start.html#banner
2018-04-16 11:37:11 +02:00
Jonas Kalderstam
9e4459d7d1 Add test for when apk icon src could not be found
An APK (Netflix) was found to have the following icon filename:

    \u2003\u2009\n.xml

This breaks the aapt dump parsing because it iterates line by line and
this filename goes across two lines. Consequently, icon_src will be
None (default value) when it is passed to the icons parser.
2018-03-27 18:39:59 +02:00
Hans-Christoph Steiner
d98d8c2e52 update: do not crash with androguard when working with apps without icons
For example, SpeedoMeterApp.main_1.apk
2018-03-12 17:55:45 +01:00
Hans-Christoph Steiner
7d79780df0 tests: checkupdates now requires a clean git repo state 2018-03-06 20:50:23 +01:00
Hans-Christoph Steiner
8f30c892c5 VercodeOperation: only allow simple math expresssions and %c 2018-03-05 09:45:58 +01:00
Hans-Christoph Steiner
498ea5d609 lint: ban all dangerous HTML tags
* https://en.wikipedia.org/wiki/HTML_sanitization
* https://asostack.com/enhance-your-google-play-store-description-with-rich-formatting-and-emojis-5f50ff354e5f
2018-02-27 12:09:54 +01:00
Hans-Christoph Steiner
e75bf70be6 signatures: future-proof fetching app ID info from APK
We're not using platformBuildVersionName and it might go away just like it
appeared: with no good reason or announcement.
2018-02-22 15:08:55 +01:00
Hans-Christoph Steiner
089712c012 tests: do not automatically run install.TestCase, its troublesome
`fdroid install` is rarely used, if at all, and the test frequently fails
for no reason in gitlab-ci, because it can't start the adb server.
2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner
52b3436ff6 make is_apk_and_debuggable() default to using androguard before aapt 2018-02-22 15:08:53 +01:00
Hans-Christoph Steiner
40fac10ebc update: extract and store XML icons
These can then be used by the client.

#344
#392
2018-02-15 14:28:45 +01:00
Hans-Christoph Steiner
05eef5a454 metadata: add new Translation app field for URL for contributions
This is a field requested from a bunch of translators so they can easily
find where to translate apps:
https://forum.f-droid.org/t/translation-field-for-app-entries/1403/5
2018-02-05 15:45:12 +01:00
relan
86f34ee70a build: make per-build hard time limit customizable
Add "timeout=n" metadata field that overrides build timeout (in seconds).
The default is 7200, i.e. 2 hours.
2018-01-30 11:14:50 +03:00
Hans-Christoph Steiner
d3caf09421 use standard User-Agent in check-fdroid-apk 2018-01-26 10:18:41 +01:00
Hans-Christoph Steiner
513c95894c build: remove unused, unmaintained Kivy build method
This code has never been used and contains some insecure uses of shell=True
Building Kivy apps should be done with the buildozer=yes method.  The
buildozer method should probably be moved to a provisioner once that is in
place.
2018-01-23 23:16:05 +01:00
Hans-Christoph Steiner
dfbe114af3 new script to audit the FDroid.apk on https://f-droid.org
This makes sure its signed by the F-Droid key.
2018-01-05 14:08:14 +01:00
Hans-Christoph Steiner
6d32b68016 tests: correct package names in stats/known_apks.txt
This doesn't seem to affect the tests either way, but it is good to have
things correct there.
2017-12-28 23:07:26 +01:00
Hans-Christoph Steiner
e451ec0079 common: fix bug in new SHA-256 signatures for >= android-18
Luckily, this is only used in `fdroid nightly` so far.
2017-12-28 23:07:26 +01:00
Hans-Christoph Steiner
109eb928e8 aapt 26.0.0 is required to properly parse permissions and label
#236

closes #395
aapt 26.0.0 outputs the permissions correctly

closes #306
aapt 26.0.0 now outputs:  application-label:'K-9 Mail'
2017-12-28 23:07:26 +01:00
mimi89999
455dcc34d8
Add Nextcloud and DavDroid test case 2017-12-27 11:51:57 +01:00
mimi89999
90c7dd29df
gradle file: use flavour specific versionCode/versionName, fall back to parsing line by line 2017-12-23 17:12:54 +01:00
mimi89999
918bd15c45
Revert: gradle file: use flavour specific versionCode/versionName, fall back to parsing line by line 2017-12-23 12:57:34 +01:00
Hans-Christoph Steiner
173a35b190 tests: only generate keystores when that is actually being tested
Generating a keystore is quite slow since it means a new RSA key is created.
That only needs to happen in the tests that check that it actually happened,
otherwise the test can just reuse the stored test keystore.

closes #432
2017-12-20 23:46:37 +01:00
Hans-Christoph Steiner
39b76b0eda scanner: fix tests so they work on all tested platforms
The standard test configuration is needed to make the tests reliably. Also,
these tests used some odd yield logic.  Who knows what exactly failed, but
these tests should be reliable.

* https://gitlab.com/fdroid/fdroidserver/-/jobs/44984595
* https://gitlab.com/fdroid/fdroidserver/-/jobs/44984596
* https://travis-ci.org/f-droid/fdroidserver/builds/318071369
2017-12-19 22:51:40 +01:00
Hans-Christoph Steiner
86c5598307 Merge branch 'whitelist-firebase' into 'master'
Whitelist some open-source firebase libs

See merge request fdroid/fdroidserver!411
2017-12-18 12:30:30 +00:00
Jan Berkel
365834d3ea Convert to string 2017-12-15 00:58:46 +01:00
Jan Berkel
5dee23f7a6 Add a simple test for scanner 2017-12-15 00:37:52 +01:00
Hans-Christoph Steiner
bde0558d82 update: reject APKs with invalid file sig, probably Janus exploits
This just checks the first four bytes of the APK file, aka the "file
signature", to make sure it is the ZIP signature and not the DEX signature.
This was checked against the test APK, and I ran it against some known
malware and all of f-droid.org to make sure it works.

All valid ZIP files (therefore APK files) should start with the ZIP
Local File Header of four bytes.

https://www.guardsquare.com/en/blog/new-android-vulnerability-allows-attackers-modify-apps-without-affecting-their-signatures
2017-12-14 16:57:22 +01:00
Hans-Christoph Steiner
ca50adb2e5 update: switch tests to using standardized setUp() method 2017-12-14 16:57:22 +01:00
Hans-Christoph Steiner
61aac0503a Merge branch 'fixFlavor' into 'master'
Regex only for flavor blocks: flavor { ... }

See merge request fdroid/fdroidserver!407
2017-12-14 16:56:01 +01:00
Hans-Christoph Steiner
bec4f7d547 add Conversations as gradle flavor test case 2017-12-14 16:52:02 +01:00
Andrea Scarpino
19a83b6219 Add Liberapay support 2017-12-12 11:53:31 +01:00
Hans-Christoph Steiner
6228162cbd handle jarsigner/apksigner output cleanly for rational logging
These were both spamming the output with lots of confusing messages, even
when --verbose was not used.  Jarsigner especially has confusing messages,
since it has warnings that do not pertain to APK signatures at all, like
the ones about timestamps and missing Certificate Authority.

closes #405
2017-12-07 17:32:14 +01:00
Hans-Christoph Steiner
05abbfbabd gitlab-ci: move sdist test run to new fedora job
A full run of the test suite takes quite a bit of time.  This removes one
of the 3 runs from the main 'tests' job, and puts it into the Fedora job.
That test run is mostly to make sure the setup.py and source tarball are
correctly, so that doesn't affect merge requests very often.

This also tests `pip install --user`, which was not really being tested
before.
2017-12-06 20:20:17 +01:00
Marcus Hoffmann
bfe2c00834 common.testCase: fix find_sdk_tools when aapt is installed in /usr/bin
The testlogic was broken when having both aapt in /usr/bin and also as
part of the android sdk.
2017-12-06 12:30:47 +01:00
Hans-Christoph Steiner
1c3a4479ab add common.sign_apk() for nighly as test for using in publish
Since the MD5 migration was quite a bit of work, it makes sense to start
on moving away from SHA1 as much as possible while it is easy to do. SHA256
will only work in APK signatures on android-18 (4.3) or newer.  So if an
APK has a minSdkVersion of 18 or newer, then sign with SHA256.

https://issuetracker.google.com/issues/36956587
https://android-review.googlesource.com/c/platform/libcore/+/44491
2017-12-04 22:52:41 +01:00
Hans-Christoph Steiner
3f4e93bf86 Merge branch 'metadataFlavours' into 'master'
Add flavour to metadata

See merge request fdroid/fdroidserver!370
2017-12-02 20:51:24 +00:00
tobiasKaminsky
d5ecb34df7
add flavour to metadata 2017-12-01 15:25:00 +01:00
Hans-Christoph Steiner
5b22ff7dc6 tests: use standard dir setup so all tests start in same dir 2017-11-30 17:32:53 +01:00
Hans-Christoph Steiner
3ff4b656c6 tests: name temp test dir after test function that used it 2017-11-30 17:32:53 +01:00
Hans-Christoph Steiner
30b2f5a48a build: sort fdroid build --all by most recently changed first 2017-11-30 17:32:53 +01:00
Hans-Christoph Steiner
c7c40cb59f PEP8 fixes 2017-11-30 13:42:37 +01:00
tobiasKaminsky
33aee96ed9
added test case 2017-11-30 11:12:18 +01:00
Hans-Christoph Steiner
743ef9ba00 rename test files to work round filename length limits
It seems a lot of things like tar, Windows, some filesystems, etc. do not
handle long filenames. #416
2017-11-27 16:57:30 +01:00
Hans-Christoph Steiner
e33684fc9b tests: new tests for stacktraces caused by translated help strings 2017-11-22 11:40:16 +01:00
Hans-Christoph Steiner
cbd9669105 rename test file to fit within eCryptfs filename limits (closes #416) 2017-11-22 11:40:16 +01:00
Hans-Christoph Steiner
cd3e763953 tests: only test fdroid build if the right build-tools is installed 2017-11-09 14:07:40 +01:00
Hans-Christoph Steiner
a14124955a tests: fix fdroid build test with Debian gradle 2017-11-09 13:11:49 +01:00
Hans-Christoph Steiner
2ee32f3524 fix tests to work when only Debian Android SDK is installed
jenkins.debian.net only had `apt install android-sdk`, it does not have any
of Google's packages installed.
2017-11-08 14:05:17 +01:00
Hans-Christoph Steiner
17efa13183 tests: pre-set failfast as reminder of a handy time saver 2017-10-25 23:01:25 +02:00
Hans-Christoph Steiner
e0df6d2479 choose the most recent available version of Java
This came about testing on OSX, where there are often multiple versions of
the JDK installed.  This was choosing the oldest version.  It should
choose the most recent version.
2017-10-25 23:01:25 +02:00
Hans-Christoph Steiner
49bd5a6462 use Babel to compile localizations and include in source tarball
python3-babel was also included in the gitlab-ci docker image:
ffc08dbc1d
2017-10-19 23:28:13 +02:00
Hans-Christoph Steiner
7d4e354f25 FDroidException handle Exceptions that return things other than str
This is related to a4c4a16ed9
2017-10-19 20:38:43 +02:00
Hans-Christoph Steiner
af0d8ab84c tests: make pip3 install quieter for shorter CI logs 2017-10-19 20:23:57 +02:00
Michael Pöhn
788c8f97fd better error message in publish when repo_key is not set 2017-10-17 22:10:39 +02:00
Michael Pöhn
89d9d0c45b use OrderedDict for localization data in index-v1 2017-09-26 14:15:48 +02:00
Michael Pöhn
26192a00e2 make sure all timestamps in index-v1 are utc 2017-09-26 14:15:27 +02:00
Michael Pöhn
a2d361dec8 add test for index-v1.json 2017-09-26 14:12:59 +02:00
Michael Pöhn
b6b79edf11 store fdroid signing-key-fp-list in stats dir 2017-09-26 14:11:09 +02:00
Michael Pöhn
6930edf889 sort index-v1; publish now creates and stores a list of signature fingerprints 2017-09-26 14:11:09 +02:00
Michael Pöhn
bca07f794f added functions for storing/loading signer fingerprints to stats 2017-09-26 14:11:09 +02:00
Michael Pöhn
5a524d4d0c added some developer-signing key metadata to tests 2017-09-26 14:11:09 +02:00
Michael Pöhn
6a4a78429f add signer to index v1 2017-09-26 14:11:09 +02:00
Michael Pöhn
7c4b8bcac4 added parse release filename function 2017-09-26 14:11:09 +02:00
Michael Pöhn
04daa7a03a test for common.get_app_id_aapt 2017-09-26 14:11:09 +02:00
Michael Pöhn
efb0bf6ee5 add common functions for dealing with apk signatures 2017-09-26 14:11:09 +02:00
Michael Pöhn
c196f7dd7a common function for fetching sha256 signing-key fingerprint 2017-09-26 14:11:09 +02:00
Hans-Christoph Steiner
eaca20675c tests: test setting up app git repo like fdroid build does
I wrote this to try to find #379 but this test didn't trigger that bug.
2017-09-25 16:35:18 +02:00
Hans-Christoph Steiner
cbf7ba0414 tests: add setUp() method to common to handle standard stuff 2017-09-25 16:35:18 +02:00
Hans-Christoph Steiner
2764c43fa2 tests: use ci-test-app with current Gradle Android Plugin
This works around the gradle 2.x bug where versions newer than 2.9 cannot
run things.  This also then specifies the version of Gradle Android Plugin
that is included in Debian/stretch.
2017-09-19 20:16:08 +02:00
Hans-Christoph Steiner
dc569b9c18 tests: support Java setups where MD5 is not disabled by default
For platforms using Java < 1.8.0_133, MD5 is still enabled for JAR
signatures.  Its just too painful to manage all this, so support this
in the tests.
2017-09-19 20:16:08 +02:00
Hans-Christoph Steiner
373b46ab3f tests: don't run gpgsign tests on Travis CI's OSX, gpg always fails 2017-09-19 20:16:08 +02:00
Hans-Christoph Steiner
d1f6126dea tests: support Ubuntu/trusty's python3.4-venv 2017-09-19 20:13:36 +02:00
Hans-Christoph Steiner
bc192b6be0 tests: prefer GNU sed on BSD/OSX, BSD's sed has lame syntax
https://blog.remibergsma.com/2012/09/18/sed-inline-editing-different-on-mac-osx/
2017-09-19 20:13:36 +02:00
Hans-Christoph Steiner
176f539647 allow spaces in filenames
This fixes all the bugs I could find that prevented fdroid from
handling files with spaces in them.  This is more important now that
fdroid supports random media files, and Repomaker
2017-09-19 20:13:36 +02:00
Hans-Christoph Steiner
6adf309bef tests: move test APK with bad unicode filename to separate test
This filename has some messed up bytes related to bi-directional script
that is included (Left-to-Right and Right-to-Left).  GNU/Linux always
interprets filenames as pure byte sequences.  Windows and OSX store
filenames as Unicode strings.  So on OSX, the invalid filename gets
converted to a valid name.  That works fine, but the test fails because it
is compared to a file generated on Ubuntu, where it preserves the byte
sequence.

This includes an APK with a valid Unicode filename that includes
bi-directional script.
2017-09-19 20:13:36 +02:00
Hans-Christoph Steiner
5dcb48831f index: always use jarsigner for verifying JAR signatures
apksigner v0.7 (build-tools 26.0.1), Google made it require that the
AndroidManifest.xml was present in the archive before it verifies the
signature.  So this needs to stick with the jarsigner hack for JARs.
2017-09-19 20:13:36 +02:00
Hans-Christoph Steiner
acac15f74d Merge branch 'newcomers' into 'master'
allow starting without config.py

See merge request fdroid/fdroidserver!335
2017-09-16 08:07:56 +00:00
Hans-Christoph Steiner
e4ca82d80c tests: support running on platforms with old java
In order to test on travis-ci's OSX setup, its easiest to support the old
version of Java that is installed there by default.  It is pretty old, so
it does not include the April 2017 update that disables MD5:

https://blogs.oracle.com/java-platform-group/oracle-jre-will-no-longer-trust-md5-signed-code-by-default
https://www.bleepingcomputer.com/news/security/oracle-to-block-jar-files-signed-with-md5-starting-with-april-2017
https://support.ca.com/us/knowledge-base-articles.TEC1691042.html

#292 #323
2017-09-15 11:37:24 +02:00
Marcus Hoffmann
34f79b876c tests: don't touch config.py, no config is fine now 2017-09-14 02:15:57 +02:00
Michael Pöhn
3e6dfacf6c added signatures subcommand 2017-09-07 00:03:35 +02:00
Hans-Christoph Steiner
c67f8e349c metadata: handle empty YAML files without crashing
!311
2017-07-26 18:21:28 -07:00
Hans-Christoph Steiner
7bd171480f update: test metadata creation using internal and external templates
There is a hardcoded template in update.py, and there is also the
possibility for the user to create a template.yml.  This tests both of them
and cleans up the related code a bit.

#352 !310
2017-07-26 15:34:13 -07:00
Hans-Christoph Steiner
0a953814e2 update: always include name/summary/desc in index.xml if available
With the new localization support, the name/summary/description in the
metadata file becomes the global override.  So most apps are not going to
have those fields present in their metadata file.  This fixes the index.xml
generation to fall back to the localized versions of those fields when they
are not set in the metadata field.

https://forum.f-droid.org/t/what-has-happend-to-osmand
2017-07-19 15:07:50 +02:00
Hans-Christoph Steiner
523b5f4777 server: smooth out btlog transfer for offline signing setups
It turns out it is error prone to `git push` to a non-bare git repo. For
the offline signing machine, the git remote needs to be a regular git repo
in a directory on a thumbdrive so that once the thumbdrive is plugged into
an online machine, that git repo can be transferred to the online machine.
2017-07-17 11:38:15 +02:00
Hans-Christoph Steiner
25f96e1911 ensure that mirror URLs always include the repodir
Since the mirror URLs are per repo section (repo/archive), the mirror URLs
must include the repodir at the end.  This was missing for servergitmirrors

found by @cde when working on fdroidclient#35
2017-07-07 15:50:11 +02:00
Hans-Christoph Steiner
dfb07808d3 support configing buildserver VM per-build with sudo=
This adds the 'sudo' build field, which is just a script that is run as
root.  For more info, see the issue that this closes:

refs #318
closes #317
2017-07-06 17:29:23 +02:00
Hans-Christoph Steiner
4a15208b84 update: update openssl KnownVuln scan to handle all recent versions
Thanks to @bubu for reporting!
2017-07-06 13:31:42 +02:00
Hans-Christoph Steiner
df99c85ca6 support manually adding per-build antiFeatures in metadata
For cases like the OpenVPN vuln that was recently announced, it is useful
for fdroiddata maintainers to be able to mark builds that have known
vulnerabilities.
2017-07-06 12:25:48 +02:00
Michael Pöhn
071ad46b55 moved some yaml test files into a separate forlder to avoid conflicts with other test cases 2017-07-04 13:35:05 +02:00
Michael Pöhn
19d422da86 rewritemeta yaml: fixed boolen mapping for build flags 2017-07-04 11:51:08 +02:00
Michael Pöhn
fc070ee657 refactored yaml tests into metadata test case; fixed typo 2017-07-04 11:51:08 +02:00
Michael Pöhn
f1e0089931 prettify write_yaml metadata 2017-07-04 11:51:08 +02:00
Michael Pöhn
7aa92abebc yaml rewrite version code as int 2017-07-04 11:51:08 +02:00
Michael Pöhn
f9d4007ea6 rewrite metadata builds list to yaml 2017-07-04 11:51:08 +02:00
Michael Pöhn
735b7d0f33 fixed pyflakes/pep warings 2017-07-04 11:51:08 +02:00
Michael Pöhn
0f90ab9aac rewrite to yaml works for app data now (builds still missing) 2017-07-04 11:51:08 +02:00
Torsten Grote
0bbe0c5db0
Break up the scan_apk() method and rename it to process_apk() 2017-07-03 18:37:01 -03:00
Hans-Christoph Steiner
5f0817d7bf tests: make sure apkcache gets created 2017-07-03 10:02:51 +02:00
Hans-Christoph Steiner
746d4bd4cf update: allow_disabled_algorithms option to keep MD5 sigs in repo
The new policy is to move APKs with invalid signatures to the archive,
and only add those APKs to the archive's index if they have valid MD5
signatures.

closes #323
closes #292
2017-07-03 10:02:51 +02:00
Hans-Christoph Steiner
e1492148fa fix "Archive Policy:" field, APKs can move in/out of archive
The original logic was checking keepversions against the len() of ALL the
APKs in the repo/archive.  The correct thing is to check against the
number of APKs available for the given packageName/appid.

closes #166
2017-06-28 23:53:42 +02:00
Hans-Christoph Steiner
be2926ffc8 tests: test moving files to and from the archive
#166
2017-06-28 23:53:42 +02:00
Hans-Christoph Steiner
214c9f7a2b lint: check file extension for metadata
This helps keep fdroiddata clean, on @krt's request.

closes #222
2017-06-28 23:14:57 +02:00
lb@lb520
bab8f8ea02 Test metadata files updated. 2017-06-21 08:26:52 +02:00
Hans-Christoph Steiner
d06f05e9c8 update: do not crash on screenshots without texts/graphics
`fdroid update` crashed for apps that only had screenshots but no graphics
or localized texts because destdir was not being set in that case.  This
fixes that and adds a test case.

closes #320
!286
2017-06-12 21:25:54 +02:00
Hans-Christoph Steiner
8776221988 check signature and OpenSSL after APK has proven valid
If working with a random grabbag of APKs, there can be all sorts of
issues like corrupt entries in the ZIP, bad signatures, signatures that
are invalid since they use MD5, etc.  Moving these two checks later means
that the APKs can be renamed still.

This does change how common.getsig() works.  For years, it returned
None if the signature check failed.  Now that I've started working
with giant APK collections gathered from the wild, I can see that
`fdroid update` needs to be able to first index what's there, then
make decisions based on that information.  So that means separating
the getsig() fingerprint fetching from the APK signature verification.

This is not hugely security sensitive, since the APKs still have to
get past the Android checks, e.g. update signature checks.  Plus the
APK hash is already included in the signed index.
2017-06-01 17:45:29 +02:00
Hans-Christoph Steiner
372c8b418d strip file extension from generated name for non-APKs
With a generic file, the file name is the only guaranteed name metadata
field.  So if the name is not specified in the metadata, then the name
is set to the filename.  This changes that so that the file extension is
stripped from that generated name.
2017-06-01 16:01:05 +02:00
Hans-Christoph Steiner
9471bf2731 regexs for getting packageName and versionCode from filenames
This is useful for parsing APK files, which can include packageName,
versionCode, and optionally 7 char signing key ID (i.e. <sig>).
This also can set the packageName and versionCoe for non APK files, so
that it is easy to assign them to metadata files, and to allow for
upgrades by setting the versionCode in the filename.
2017-06-01 16:01:05 +02:00
Hans-Christoph Steiner
ceac6d25cb index.xml cannot handle APKs with the same packageName/versionCode
Really, it is the fdroidclient parser of index.xml that fails, due to the
hardcoded expectation that there will only ever be a single APK for any
given versionCode.  We keep index.xml backwards compatible for old
clients, and use index-v1.json to support new things.  Having multiple
APKs that have the same packageName and versionCode will break the client
v0.103.* since that version uses index-v1.json, but still has the hard-
coded database parsing stuff.

#153
2017-06-01 16:01:05 +02:00
Hans-Christoph Steiner
4053f03d77 update: add --rename-apks to force APK filenames to fdroid standard
uses the standard package.name_123.apk.  If that exists, it appends the
shasum.  If that exists, then its a duplicate, so its deleted. This should
help @SergeWinters with his 12,000 APKs.
2017-06-01 16:01:05 +02:00
Hans-Christoph Steiner
0f4cbc7224 allow APKs with same packageName/versionCode but different signer
There are many APKs out in the wild that claim to be the same app and
version and each other, but they are signed by different keys.  fdroid
should be able to index these, and work with them.   This supports having
the developer's signature via reproducible builds, random collections of
APKs like repomaker, etc.
2017-06-01 10:30:41 +02:00
Hans-Christoph Steiner
afac1b2efd add basic test for fdroid scanner
There was no test coverage at all for this command, this is a very basic
test that should prevent things like 2626858450
2017-05-30 21:13:49 +02:00
Hans-Christoph Steiner
114d5dc8c6 make tests pass new lint rules 2017-05-26 21:03:38 +02:00
Hans-Christoph Steiner
17cd07f1a7 fix pylint unused-argument 2017-05-23 22:34:16 +02:00
Hans-Christoph Steiner
12c80f9062 update outdated pylint setup
The logilab-astng lib is dead, replaced by python-astroid.  The crazy astng
plugin is no longer needed also.

#281
2017-05-23 22:34:16 +02:00
Hans-Christoph Steiner
c749c68486 test script for vmtools 2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
0345d2c79f update: find aapt when it is not in the PATH
This is some very messy logic built up since 2010.  This will all go away
once we have a python3 version of androguard available.

The removed imports and `dir(APK)` is to silence pyflakes

closes #303
2017-05-18 17:21:05 +02:00
Hans-Christoph Steiner
c348186ad6 update: fix fastlane scraping, it uses a subdir called 'images'
https://commons.wikimedia.org/wiki/File:GetChromium_FeatureGraphic_1024x500.png
2017-05-18 16:13:17 +02:00
thez3ro
06598ae406
use androguard if aapt isn't found 2017-05-04 23:35:17 +02:00
Torsten Grote
8d424f19ec
Support ETag when downloading repository index 2017-05-02 15:37:02 -03:00
Hans-Christoph Steiner
cdef5bcd92 downcase all 'localized' key names to match the rest of index-v1
This is a little omission.  keys that are used in metadata/*.yml all start
with an UpperCase letter, but in fdroidserver, index-v1.json, and
fdroidclient, it is all camelCase with lowercase first letter. The keys
from the 'localized' section are currently never in metadata/*.yml, so
these keys never get downcase.  This change will break fdroidclient
versions that do not also have this change, but since we're in alpha, that
should be fine.

If support for a 'localized' section is added to metadata/*.yml, then the
keys there should probably be UpperCase CamelCase to match the other keys.
2017-04-27 21:12:49 +02:00
Hans-Christoph Steiner
82095c7a9a add basic test for Triple-T Gradle Play Publisher scraping 2017-04-19 10:05:24 +02:00
Hans-Christoph Steiner
cb49f57c06 support "Author Web Site" as metadata field
Fastlane Supply, Triple-T Gradle Play Publisher, and many app stores
include the possibility to specify a website for the author, as distinct
from the website for the app.

closes #204
2017-04-19 10:05:24 +02:00
Hans-Christoph Steiner
9589d13ef2 update: include "What's New" texts when they are available
This uses the "What's New" entry for the CurrentVersionCode and includes it
as the current WhatsNew metadata for the App class.

Things like fastlane supply and Google Play support a "What's New" entry
per-APK, but fdroidclient does not current use anything but the current
version of this data.  Right now, it seems we probably only want to have
the latest WhatsNew in the index to save space.

In theory, we could make the WhatsNew data structure follow the structure
of fastlane/Play, but that would quite a bit of complexity for something
that might never be used.

fdroidclient#910
2017-04-19 10:05:24 +02:00
Hans-Christoph Steiner
31d92367ed git mirror test requires git >= 2.3 2017-04-13 00:49:36 +02:00
Hans-Christoph Steiner
17219d56f4 server: support git mirrors with the offline signing setup 2017-04-12 15:04:04 +02:00
Hans-Christoph Steiner
f75b547795 server: remove unused --sync-from-local-copy-dir
This option was not hooked up at all, and does not make sense as a command
line argument.  It should just be a config.py item.  In that case, the
presence of config.py marks the current dir as a repo, so there is no
longer a need to test for a dir called repo/ as a safety.  This makes the
setup easier, since sync_from_localcopy() now creates repo/ for the user.
2017-04-12 00:45:22 +02:00
Hans-Christoph Steiner
299ba86948 server: git push for binary transparency in fdroid server update
Since `fdroid server update` is the place where all uploads to servers
happens, it makes sense to also handle the git push for the binary
transparency log here instead of `fdroid btlog`
2017-04-10 23:02:12 +02:00
Hans-Christoph Steiner
f3b55ab0f0 fix mistake: lists use .append() not .add()
This was a mistake in 5ad3486741
2017-04-10 23:02:12 +02:00
Hans-Christoph Steiner
136d58dbae btlog: test with fdroid update without requiring ssh access
Yay git!  Just use a filesystem git remote instead of ssh.
2017-04-03 23:17:03 +02:00
Hans-Christoph Steiner
53b69f507e allow index.TestCase to work when run using any path 2017-04-03 20:24:21 +02:00
Hans-Christoph Steiner
d6310ed52c Merge branch 'invalid-apk-fix' into 'master'
Fix bug when scanning invalid APKs and add a test for it

See merge request !246
2017-04-03 17:38:51 +00:00
Torsten Grote
8f6fa2955d
Fix bug when scanning invalid APKs and add a test for it 2017-04-03 13:07:49 -03:00
Torsten Grote
a23da47118
Add method for downloading (and verifying) a repository index
This includes some test cases to test the new code.
2017-04-03 09:02:07 -03:00
Michael Pöhn
8b51e40d63 avoid duplicate value assignments when updating config files 2017-04-02 12:08:01 +02:00
Torsten Grote
9f765ed6f7
Move index signing methods into signindex.py 2017-03-29 12:11:40 -03:00
Hans-Christoph Steiner
223c793201 prefer apksigner if installed, jarsigner sucks
Google has their own utility for verifying APK signatures on a desktop
machine since Java's jarsigner is bad for the task.  For example, it
acts as if an unsigned APK validates.  And to check whether an APK is
unsigned using jarsigner is difficult.

apksigner also does the v2 signatures, so it will have to be used
eventually anyway.  It is already in Debian/stretch and can be
available in jessie-backports if need be.

https://android.googlesource.com/platform/tools/apksig
https://packages.debian.org/apksigner
2017-03-22 10:51:12 +01:00
Hans-Christoph Steiner
998b6245e9 verify: ensure only a single signature is in compared APK
The ZIP format allows multiple entries with the exact same filename, and on
top of that, it does not allow deleting or updating entries.  To make the
`fdroid verify` procedure failsafe, it needs to create a new temporary APK
that is made up on the contents of the "unsigned APK" and the signature
from the "signed APK".  Since it would be possible to give a signed APK as
in the unsigned one's position, `fdroid verify` was not able to update the
signature since it was just adding the new signature to the end of the ZIP
file.  When reading a ZIP, the first entry is used.
2017-03-22 10:51:12 +01:00
Hans-Christoph Steiner
866528de5b signindex: support signing index-v1.jar
This is a bit different than index.jar: instead of their being index.xml
and index_unsigned.jar, the presense of index-v1.json means that there is
unsigned data.  That file is then stuck into a jar and signed by the
signing process.  index-v1.json is never published to the repo.  It is
included in the binary transparency log, if that is enabled.
2017-03-17 14:12:03 +01:00
Hans-Christoph Steiner
fa657ce720 move update.signjar() to common so it can also be used in signindex 2017-03-17 13:55:40 +01:00
Hans-Christoph Steiner
713d9195c3 ensure that app.Categories is always a list of strings
This just makes it easier for people writing build recipes.  Rewriting will
output a list of strings as well.

The test index.xml and categories.txt are updated to include the new number
categories, and the changed CurrentVersionCode to 2147483647 (MAX_VALUE)
2017-03-17 13:55:40 +01:00
Hans-Christoph Steiner
7044a909ca include graphics and screenshots from repo in V1 index
If the repo has store graphics and/or screenshots, then include those in
the metadata.  This follows the possible graphics for Google Play, and the
file naming scheme of the open source 'fastlane' tool for managing those
files.

https://github.com/fastlane/fastlane/blob/1.109.0/supply/README.md#images-and-screenshots
https://support.google.com/googleplay/android-developer/answer/1078870?
https://android-developers.blogspot.com/2011/10/android-market-featured-image.html

Signed-off-by: Hans-Christoph Steiner <hans@eds.org>
2017-03-17 13:55:40 +01:00
Hans-Christoph Steiner
6c2cf2ccdd convert internal dict keys to match fdroidclient's Apk class
This syncs up the field names between the fdroiddata .yml files, the keys
used in the implementation in fdroidserver, the index data format, and the
final data structures in fdroidclient.  This makes it easier for devs to
follow, and makes the Jackson parsing library automatically handle
converting the data from the index file to Java instances.

This bumps the metadata version since the apkcache will have to be
discarded.

Here are the name changes:
* apkname --> apkName
* id --> packageName
* sha256 --> hash
* version --> versionName
* versioncode --> versionCode

tests/repo/index.xml was changed only to bump the metadata version
from 17 to 18.
2017-03-17 13:55:40 +01:00
Hans-Christoph Steiner
c9aa26d89e add index V1 format, a direct translation of internal dict
Python encode/decode libs work directly with dicts, so the internal dict
can just be passed directly to any of these libs (pyyaml, pyjson, msgpack,
simplejson, etc).  This still generates the exact same index.xml as before.

This converts the internal format for the repo timestamp to a datetime
instance, which can be easily converted to UNIX time in seconds for XML
and UNIX time in milliseconds for the new index formats.  UNIX time in
milliseconds is directly serialized into a java.util.Date instance by
Jackson.
2017-03-17 13:55:40 +01:00
Hans-Christoph Steiner
b6d8916881 Merge branch 'metadata_output_vars' into 'master'
metadata output-lines now substitute $$VERSION$$/etc.

Closes #263

See merge request !228
2017-03-17 11:37:18 +00:00
Hans-Christoph Steiner
a737255ce4 add a simple test of the binary transparency log
This makes sure it gets generated with two commits: README and first log.
2017-03-16 19:22:32 +01:00
Michael Pöhn
d2570a493e metadata output-lines now substitute $$VERSION$$/etc. 2017-03-16 09:24:05 +01:00
Hans-Christoph Steiner
6aca906863 include non-APKs that match metadata and fdroid-generated filenames
Since it is now possible to build and include arbitrary files, like OTA
update ZIP files, the update procedure needs to look for non-APK files that
match the packageName_versionCode pattern of fdroid-generated files.

!193
admin#14
privileged-extension#9
2017-03-10 18:37:18 +01:00
Hans-Christoph Steiner
d92ac1e8f6 fix metadata dump test to run on CI servers 2017-02-28 12:08:04 +01:00
Hans-Christoph Steiner
8b85c860a6 rename lastupdated to lastUpdated to match fdroid client
This lets index-v1 be parsed directly into class instances because the
field/instance var names match exactly.  The original index v0 element
must retain the 'lastupdated' name for backwards compatibility.
2017-02-28 12:08:04 +01:00
Hans-Christoph Steiner
8decd3b504 remove TeX manual, it moved to https://gitlab.com/fdroid/fdroid-website
The whole TeX manual has been folded into the new Jekyll-based website, so
this is no longer needed at all.
2017-02-24 14:01:31 +01:00
Hans-Christoph Steiner
4e39621601 provide warning if config items will not preserve order
If a group of items are enclosed in {}, then that will be a Python set,
which does not preserve order.  To preserve order, the data must be either
a tuple () or list [].
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
388c336e76 build: ensure test is running on git commit that this code works with
Since https://gitlab.com/fdroid/ci-test-app is a separate git repo, things
with incompatible changes could get out of sync.  Therefore, this test
should specify which git commit is runs against.

For example, the .fdroid.yml file is still a moving target.  Just now, the
keys had the spaces removed as part of this MR.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
593613043e normalize Build TYPE_STRING data based on .txt
Unfortunately, things like versionCode must be strings.  That should be
converted to be ints throughout.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
c178cfb843 normalize Build TYPE_LIST data based on .txt
In the future, we should have better internal datatypes for this stuff,
i.e. instead of gradle: ['yes'] for True, actually use a boolean.  For now,
make the YAML and JSON metadata produce the same internal data as .txt.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
f7ae398aae add script for mass testing metadata parsing after changes
This requires manually running it.  I suppose it would be possible to
include a snapshot of the dumped internal representation for each release,
then make the tests run automatically against that.  Right now, the dump is
17megs of YAML.  Seems large to include in this git repo.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
e0f39a7e7b rename Build fields: version -> versionName, vercode -> versionCode
Since the YAML/JSON/etc. field names are now exactly the same as the field
names used in the internal dict in the Build class, this is a global rename

This keeps with the standard names used in Android:
https://developer.android.com/guide/topics/manifest/manifest-element.html
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
c0bc3afda9 convert metadata.Build to a subclass of dict
Like with the App class in the commit before, this makes it a lot
easier to work with this data when converting between the internal
formats and external formats like YAML, JSON, MsgPack, protobuf, etc.

The one unfortunate thing here is Build.update. It becomes
dict.update(), which is a method not an attribute.
build.get('update') or build['update'] could be used, but that would
be oddly inconsistent. So instead the field is renamed to
'androidupdate', except for in the .txt v0 metadata files. This better
describes what field does anyway, since it runs `android update`.

Build.update is only referenced in two places right next to each other
for the ant builds, so this change still seems worthwhile.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
b7fc7f2228 convert App to subclass of dict to support parsing/dumping libs
Python is heavily based on its core data types, and dict is one of the more
important ones.  Even classes are basically a wrapper around a dict. This
converts metadata.App to be a subclass of dict so it can behave like a dict
when being dumped and loaded.  This makes its drastically easier to use
different data formats for build metadata and for sending data to the
client.  This approach will ultimately mean we no longer have to maintain
custom parsing and dumping code.

This also means then that the YAML/JSON field names will not have spaces in
them, and they will match exactly what it used as the dict keys once the
data is parsed, as well as matching exactly the instance attribute names:

* CurrentVersion: 1.2.6
* app['CurrentVersion'] == '1.2.6'
* app.CurrentVersion == '1.2.6'

Inspired by:
https://goodcode.io/articles/python-dict-object/
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
56ef716a4e test for original "v0" index XML compatibility
The original index.xml format needs to stay around for backwards
compatibility, but we shouldn't touch it anymore once the new format is in
place.  This is a test to make sure `fdroid update` can still generate the
correct XML.

install_list and uninstall_list should be tuples or lists in order to
ensure that the order is preserved.

These tests also check that the added and lastupdated dates are
working correct, based on the dates in tests/stats/known_apks.txt. I
could see no useful way to test the timestamp, it is just hardcoded
using a regexp search-and-replace.  Running these tests manually might
require deleting tmp/apkcache.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
36272656fc gitlab-ci: workaround "ImportError: No module named 'packaging'"
https://github.com/pypa/setuptools/issues/937

fdroid/ci-images#1
2017-02-06 17:28:07 +01:00
Hans-Christoph Steiner
d08bd32a35 update URL for git repo of test app 2017-01-20 12:12:20 +01:00
Hans-Christoph Steiner
b1a943d562 convert metadata test dumps to YAML for easy comparison
When making code changes related to the metadata parsing, it is useful to
see how the internal format has changed by seeing the differences in the
dump files.  Those files are currently in the binary .pickle format.  This
just straight converts them to YAML, which is a text format, so that normal
diff tools work to see changes.

The dump files are named .yaml instead of .yml since .yml is used for hand-
edited YAML files for fdroiddata/metadata, while these dump files here are
a human readable form of a Python pickle.
2016-12-05 20:16:54 +01:00
Hans-Christoph Steiner
ce3efe4168 remove support for XML app metadata, its broken
JSON and YAML are very closely related, so supporting both of them is
basically almost no extra work.  Both are also closely related to how
Python works with dicts and pickles. XML is a very different beast, and its
not popular for this kind of thing anyway, so just purge it.
2016-12-05 20:16:54 +01:00
Hans-Christoph Steiner
a4e4310803 allow metadata to be embedded in source repos via .fdroid.yml
This allows a source repo to include a complete metadata file so that it
can be built directly in place using `fdroid build`.  If that app is then
included in fdroiddata, it will first load the source repo type and URL
from fdroiddata, then read .fdroid.yml if it exists, then include the rest
of the metadata as specified in fdroiddata, so that fdroiddata has
precedence over the metadata in the source code.

This lets `fdroid build` apps without having a whole fdroiddata setup, but
instead just directly in place in the source code.  This also lets devs
optionallu maintain the fdroid metadata as part of their app, rather than
in fdroiddata without loosing any control.  This should make it easier to
spread around the maintenance load.
2016-11-16 23:28:03 +01:00
Hans-Christoph Steiner
b4a39ee272 switch import test to custom, small test app
The test project should be moved to https://gitlab.com/fdroid/ci-test-app
2016-11-16 23:28:03 +01:00
Hans-Christoph Steiner
1f55a40caa properly parse build metadata list types like gradle=
Something like `gradle: yes` in YAML will be parsed as a boolean, since
'yes' is officially defined as a boolean true in YAML.  For metadata fields
that need to be lists, this needs to be converted.  Same goes for a single
string like `gradle: customFlavor`.
2016-11-16 23:28:03 +01:00
Daniel Martí
82b1d7ad14 all: make newer pycodestyle happy
Apparently the "two empty lines" rule is now stricter.
2016-11-15 20:55:06 +00:00
Hans-Christoph Steiner
56d51fcd6b gpg-sign all valid files in the repo, including source tarballs
This makes sure there is a GPG signature on any file that is included in
the repo, including APKs, OBB, source tarballs, media files, OTA update
ZIPs, etc.  Having a GPG signature is more important on non-APK files since
they mostly do not have any signature mechanism of their own.

This also adds basic tests of adding non-APK/OBB files to a repo with
`fdroid update`.

closes #232
2016-11-07 14:53:01 +01:00
Hans-Christoph Steiner
84e09cd2a2 allow arbitrary build products, not only APKs
This makes it so that the final build product can be specified in output=
and it'll work no matter if its an APK or not.  This was developed around
the case of building the OTA update.zip for the Privileged Extension. It
should work for any build process in theory but it has not yet been tested.

https://gitlab.com/fdroid/privileged-extension/issues/9
2016-11-07 14:53:01 +01:00
Hans-Christoph Steiner
47d9fd330d remove unused 'apps' argument from update.scan_apks() 2016-11-02 16:11:52 +01:00
Hans-Christoph Steiner
008b4a31eb skip fdroid import test if gitlab is not available
This prevents CI build failures when gitlab is deploying or has other
issues that might cause a 500.
2016-09-15 23:20:18 +02:00
Hans-Christoph Steiner
6126b55136 rename server request from "delete" to "uninstall"
This matches the Android API's current Intent action for this, rather than
the deprecated one:

https://gitlab.com/fdroid/fdroidclient/blob/v0.101-alpha5/app/src/main/java/org/fdroid/fdroid/installer/DefaultInstallerActivity.java#L147
https://developer.android.com/reference/android/content/Intent.html#ACTION_UNINSTALL_PACKAGE
https://developer.android.com/reference/android/content/Intent.html#ACTION_DELETE
2016-08-24 23:05:34 +02:00
Hans-Christoph Steiner
85632ba00e set up install/delete lists for "push" commands from server
It is now possible for the server operator to specify lists of apps that
must be installed or deleted on the client (aka "push installs).  If
the user has opted in, or the device is already setup to respond to
these requests, then fdroidclient will automatically install/delete
the packageNames listed.  This is protected by the same signing key
as the app index metadata.

It generates single XML elements with the data set in the attributes. This
keeps the XML compact and easily extensible, e.g. for adding versionCode,
signingKey, etc as attributes:

    <install packageName="com.fsck.k9"/>
    <install packageName="at.bitfire.davdroid"/>
    <delete packageName="com.facebook.orca"/>

Copyright: 2016 Blue Jay Wireless
Signed-off-by: Hans-Christoph Steiner <hans@eds.org>

closes #177
2016-08-16 21:02:15 +02:00
Hans-Christoph Steiner
5f8beaa2c6 tests for finding the proper OBB files for each APK 2016-07-07 22:34:41 +02:00
Hans-Christoph Steiner
1be263e870 FDroidPopen must have a locale to support UTF-8 filenames
`fdroid update` should be able to handle any valid filename (hopefully
aapt doesn't barf on them).  To handle that, the environment where the
shell commands are run in needs to have a UTF-8 locale set. If LANG is
not set, things default to ASCII and UTF-8 filenames fail.

This also renames test APK with lots of Unicode chars as a test case.

closes #167
2016-06-27 20:30:32 +02:00
Daniel Martí
5de678e48b all: switch to jdk8 as default
Also, remove jdk7 as it will become unused. We added jdk8 for
retrolambda, and now that we will use jdk8 as the default, jdk7 is
unnecessary as retrolambda can work fine with just jdk8.

This removes it from the buildserver, and the new CI image also only has
jdk8 from jessie-backports.

Fixes #185.
2016-06-21 11:47:37 +01:00
Hans-Christoph Steiner
c352605768 add force_build_tools config option
This replaces the current default behavior of always forcing the
build_tools version and allows the user to set build-tools forcing in
config.py.

closes #147
2016-06-21 10:29:56 +02:00
Hans-Christoph Steiner
85b65f0941 tests: pyvenv --system-site-packages is too buggy on python 3.4
It always wants to install packages into /usr/lib/python3.4/site-packages

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 295, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install
    requirement.install(install_options, global_options, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install
    self.move_wheel_files(self.source_dir, root=root)
  File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in move_wheel_files
    pycompile=self.pycompile,
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 214, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 176, in clobber
    os.makedirs(dest)
  File "/usr/lib/python3.4/os.py", line 237, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.4/site-packages'
2016-06-14 22:40:50 +02:00
Hans-Christoph Steiner
1b7a8f85fc parse targetSdkVersion from APKs
The default targetSdkVersion is minSdkVersion, according to the docs:
https://developer.android.com/guide/topics/manifest/uses-sdk-element.html#target

https://gitlab.com/fdroid/fdroidclient/issues/682
2016-06-14 11:43:07 +02:00