445 Commits

Author	SHA1	Message	Date
Hans-Christoph Steiner	3801db064a	lint: improve HTTPS check ... It was missing some domains, so I added another rule. @IzzySoft pointed out it was redudnant, so this removes the redudant rule and fixes the original. https://gitlab.com/fdroid/fdroidserver/merge_requests/681/diffs#note_225263464	2019-10-04 11:12:48 +02:00
Hans-Christoph Steiner	af4e231f7d	lint: enforce HTTPS on GitHub and GitLab pages ... The always provide HTTPS, so let's enforce it.	2019-10-04 11:12:48 +02:00
Hans-Christoph Steiner	17320c23f4	lint: don't trip up on projects with 'master' in the name ... https://gitlab.com/fdroid/fdroiddata/merge_requests/5557#note_223283359	2019-10-04 11:12:48 +02:00
Michael Pöhn	d5ab303d83	lint: license override config option + require FSF/OSI approved licenses by default	2019-10-02 22:14:09 +02:00
Hans-Christoph Steiner	c7048f2c39	update: log errors on bad graphics, and then ignore the file ... Python PIL is not so tolerant, so bad EXIF causes crashes: File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2088, in main insert_localized_app_metadata(apps) File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 978, in insert_localized_app_metadata _strip_and_copy_image(os.path.join(root, f), destdir) File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 754, in _strip_and_copy_image in_image = Image.open(fp) File "/usr/lib/python3/dist-packages/PIL/Image.py", line 2687, in open % (filename if filename else fp)) OSError: cannot identify image file <_io.BufferedReader name='build/org.sw24softwares.starkeverben/fastlane/metadata/android/en-US/images/featureGraphic.png'>	2019-09-26 22:06:21 +02:00
Hans-Christoph Steiner	cf2c1d0270	run-tests needs a source of APKs to function ... The test runs on jenkins.debian.net were failing because they specified a dir that did not exist.	2019-09-25 12:25:39 +02:00
Hans-Christoph Steiner	25548023e0	gitlab-ci: check gradle checksums against official list	2019-09-12 14:18:07 +02:00
relan	079754c56c	scanner: add a test for the local Debian Maven repo	2019-08-29 21:33:12 +03:00
Hans-Christoph Steiner	70e7e720b9	update: use graphics filename with hash in index to support caching ... Using a filename based on the hash of the contents means that the caching algorithms for fdroidclient and browsers can safely cache the file forever using the filename, since this guarantees that the contents will never change for a given filename. This does not cover screenshots, only icon.png, featureGraphic.png, tvBanner.png, and promoGraphic.png. fdroidserver#689 fdroid-website!453	2019-08-29 08:23:51 +02:00
Hans-Christoph Steiner	508af00e84	update: only copy graphics and screenshots if mtime/size has changed ... Instead of copying every time, trust the filesystem to tell us when the file has changed.	2019-08-29 08:23:47 +02:00
Hans-Christoph Steiner	f13c41e3ba	update: remove Provides: from template.yml test case ... !654 dcf3837bcb	2019-08-28 15:57:40 +02:00
Hans-Christoph Steiner	7090ce63dd	Merge branch 'remove-provides-in-yml' into 'master' ... Remove `Provides:` in yaml metadata Closes #676 See merge request fdroid/fdroidserver!654	2019-08-28 13:39:59 +00:00
Michael Pöhn	8e5232076f	do not delete yml metadata when raumel not installed	2019-08-27 15:38:40 +02:00
Michael Pöhn	0885303672	add additional rewritemeta test for yml	2019-08-27 15:38:40 +02:00
Michael Pöhn	3951d93196	add rewritemeta.TestCase	2019-08-27 15:38:40 +02:00
Michael Pöhn	39c589fb85	Merge branch 'update-create-template' into 'master' ... make metadata template behave well for empty values Closes #681 See merge request fdroid/fdroidserver!657	2019-07-24 11:02:38 +00:00
Michael Pöhn	d0368d0ad8	common add parse_androidmanifests_ignore test	2019-07-23 22:44:45 +02:00
Michael Pöhn	bad888856a	checkupdates: add check_http ignore test	2019-07-23 22:01:18 +02:00
Michael Pöhn	23280b6029	checkupdates add check_http test	2019-07-23 22:01:18 +02:00
Michael Pöhn	1c7af1dc2c	add test for checkupdates_app()	2019-07-23 22:01:18 +02:00
Michael Pöhn	e2fed09af1	make metadata template behaves well for empty values	2019-07-22 01:34:55 +02:00
Michael Pöhn	dcf3837bcb	parse yaml: ignore (and warn) deprecated field: Provides	2019-07-15 15:45:02 +02:00
Michael Pöhn	6e48663230	test that write yaml does not write provides	2019-07-11 03:35:23 +02:00
Michael Pöhn	723bd110a6	test that yaml parsing does not accept provides	2019-07-11 03:35:23 +02:00
Hans-Christoph Steiner	051596dd0d	tests: conditionally disable tests that can't work with apksigner ... apksigner treats MD5 signatures as valid, fdroid does not.	2019-07-10 14:35:03 +02:00
Hans-Christoph Steiner	26af94974a	Merge branch 'rsync-buildslogs-to-webroot-repo' into 'master' ... build: rsync buildlogs to <webroot>/repo See merge request fdroid/fdroidserver!651	2019-07-10 08:55:04 +00:00
Michael Pöhn	2c87b5e6f9	deploy build logs: no timestamps	2019-07-06 16:48:56 +02:00
Michael Pöhn	f30983368c	build: rsync buildlogs to <webroot>/repo	2019-07-06 16:48:56 +02:00
Hans-Christoph Steiner	a248a69692	tests: skip disabled_algorithms test when apksigner is present ... apksigner doesn't treat MD5 signatures as deprecated, so that portion of the tests would always fail.	2019-07-04 16:45:50 +02:00
Hans-Christoph Steiner	a9aa8788e0	tests: only run source tarball test if running from git clone	2019-07-03 22:33:15 +02:00
Hans-Christoph Steiner	aa1e958360	tests: only run hooks/pre-commit if its present (not in source tarball)	2019-07-03 20:46:30 +02:00
Hans-Christoph Steiner	a0f5ee661e	tests: common.test_sign_apk requires aapt to run	2019-07-03 09:07:36 +02:00
Hans-Christoph Steiner	57b9d1e316	tests: handle when apksigner considers MD5 signatures valid	2019-07-02 22:17:06 +02:00
Michael Pöhn	66105de833	improve litecoin validation + tests	2019-05-28 11:04:43 +02:00
Michael Pöhn	06cec2041d	improve bitcoin validation regex + testcases	2019-05-07 22:43:05 +02:00
Hans-Christoph Steiner	dd2f9d60f8	publish: fix stupid error in repro-signing and add integration test ... stoopid mistake in ea84014f9b reported by @CiaranG	2019-04-11 14:06:51 +02:00
Marcus	7272689ced	Merge branch 'master' into 'master' ... Added newer ndks, gradles, latest sdk-license, and update java 1.8 version See merge request fdroid/fdroidserver!637	2019-04-11 11:42:18 +00:00
Michael Pöhn	67731470cc	Revert "Merge branch 'write-yaml-overhaul' into 'master'" ... This reverts merge request !630	2019-04-01 10:24:00 +00:00
Taco	457cf22361	Added newer ndks, gradles, latest sdk-license, and update java 1.8 version	2019-03-30 17:10:21 -04:00
Michael Pöhn	2683b37044	yml metadata write: do not use local functions	2019-03-19 01:01:18 +01:00
Michael Pöhn	881a79fa84	test writing all yaml fields	2019-03-18 22:45:35 +01:00
Michael Pöhn	63afc0acb5	use pyyaml for writing metadata instead of ruamel	2019-03-18 22:45:35 +01:00
Hans-Christoph Steiner	d03b121152	update: allow tests to pass when apksigner is not installed ... This is only for the v2/v3 signatures. fdroid/fdroidserver#627	2019-02-12 12:43:21 +01:00
Hans-Christoph Steiner	50ca3967cc	update: fix tests when running without apksigner	2019-02-03 15:50:39 +01:00
Hans-Christoph Steiner	17dc231dc9	update: fix running without androguard ... Soon, we can rip out all the aapt parsing stuff, but not yet!	2019-02-03 15:50:39 +01:00
Michael Pöhn	dd695c650e	update: treat target and min sdk version as int	2019-02-01 09:56:21 +01:00
Hans-Christoph Steiner	d96f5ff660	support APK Signature V2 when apksigner is installed ... This was done with much help from @uniqx. This is the first level of supporting APK Signatures v1, v2, and v3. This is enough to include APKs with any combo of v1/v2/v3 signatures. For this to work at all, apksigner and androguard 3.3.3+ must be installed. closes #399	2019-02-01 09:17:56 +01:00
Francesco Cervigni	984d276c1b	Added test aapt output files for build-tools 28.0.3	2019-01-13 19:27:02 +01:00
Michael Pöhn	31ca2092a1	yaml parsing: script build flags can now be lists	2018-11-29 21:53:37 +01:00
Michael Pöhn	c15a7508e7	write yaml script metadata as lists	2018-11-29 21:53:37 +01:00
Michael Pöhn	a21635ae2e	fix metadata test case: write yaml prebuild	2018-11-29 21:53:37 +01:00
Michael Pöhn	723815a25b	fix metadata.Testcase:test_rewrite_yaml_special_build_params	2018-11-29 21:53:37 +01:00
Michael Pöhn	d0a129c216	add test for parsing build field prebuild as string	2018-11-29 21:53:37 +01:00
Michael Pöhn	942de28fa5	yaml metadata: split prebuild build field to list	2018-11-29 21:53:37 +01:00
Hans-Christoph Steiner	57556aceee	remove redundant open() arg: encoding='utf8' ... By default, open() returns a str: https://docs.python.org/3/library/functions.html#open By default, str is UTF-8: https://docs.python.org/3/library/stdtypes.html#str This used to matter on Python 2.x, but this code is 3.x only now.	2018-10-19 15:01:34 +02:00
Hans-Christoph Steiner	bfdf581201	import: use valid placeholder values for versionCode/versionName ... fdroid/fdroidserver!559 closes fdroid/fdroidserver#548	2018-10-10 16:02:34 +02:00
Michael Pöhn	67e46694d3	add test for allowing to write placeholder values to yaml metadata files	2018-10-10 16:02:34 +02:00
Hans-Christoph Steiner	3bea689f74	add another lint field type test case	2018-10-10 16:02:34 +02:00
Hans-Christoph Steiner	e33c1c74b1	tests: include info.zwanenburg.caffeinetile in index tests	2018-10-10 16:02:34 +02:00
Hans-Christoph Steiner	8b251da79f	update: do not set targetSdkVersion if the APK is missing it ... 699b3e4c69 got it wrong for targetSdkVersion. Also, one confusing thing is that aapt outputs "sdkVersion: '3'" for com.politedroid_3.apk but no "sdkVersion:" for no.min.target.sdk_987.apk. F-Droid never really supported running on android-1 or android-2, so it seems pointless to debug support for them.	2018-10-10 15:29:02 +02:00
Hans-Christoph Steiner	85993eb2f8	lint: check fields for proper type, e.g. list vs. string ... fdroid/fdroidserver#578	2018-10-09 23:31:24 +02:00
Hans-Christoph Steiner	6b57cb6b7c	fix strict Application ID checks ... * upper case letters are allowed at all positions * there must be a "." separator	2018-09-24 17:07:27 +02:00
Hans-Christoph Steiner	11d46072ab	use androguard primitives to speed up finding debuggable flag ... androguard parses the whole APK before handing the instance back, this uses the primitives to just find the <application android:debuggable=""> value, then stop parsing. #557	2018-09-21 14:56:46 +02:00
Hans-Christoph Steiner	a3cecc16a3	use partial androguard binary XML parsing to speed up APK ID lookup ... Normally, androguard parses the entire APK before it is possible to get any values from it. This uses androguard primitives to only attempt to parse the AndroidManifest.xml, then to quit as soon as it gets what it needs. This greatly speeds up the parsing (1 minute vs 60 minutes). fdroid/fdroidserver#557	2018-09-21 14:56:46 +02:00
Hans-Christoph Steiner	4c4da3d176	update: remove 'SET' debug antifeature, closes #565 ... oops, this slipped in in 3011953d0e Bump the metadata version to purge the cache	2018-09-18 10:57:10 +02:00
Hans-Christoph Steiner	92c4f048de	update: test scan_apk() with both aapt and androguard ... #568	2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner	807bf3d26b	build: reuse common methods for getting metadata from APKs ... This splits out the code that gets the list of native ABIs supported, then uses the standard methods for the rest.	2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner	487c4d02f3	handle package: line output from aapt v28 ... fdroid/fdroiddata!3484 fdroid/fdroiddata!3562 fdroid/fdroidserver!548	2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner	d1acef0405	tests: generate aapt output for every version to make tests easy	2018-09-18 10:55:51 +02:00
Hans-Christoph Steiner	9d12b1dc61	add strict, tested validation of Android/F-Droid package names ... Android has stricter rules than Java for Package Names, but anything the Python regex thinks is valid must be valid according to Java's rules too. https://developer.android.com/studio/build/application-id	2018-09-07 14:17:39 +02:00
Hans-Christoph Steiner	5d161cc9fd	validate appid when reading metadata files ... The metadata file must be named after the Application ID of the app it is describing, and Android Application IDs must be valid Java Package Names.	2018-09-03 22:56:08 +02:00
Hans-Christoph Steiner	3011953d0e	convert apkcache from pickle to JSON ... pickle can serialize executable code, while JSON is only ever pure data. The APK cache is only ever pure data, so no need for the security risks of pickle. For example, if some malicious thing gets write access on the `fdroid update` machine, it can write out a custom tmp/apkcache which would then be executed. That is not possible with JSON. This does just ignore any existing cache and rebuilds from scratch. That is so we don't need to maintain pickle anywhere, and to ensure there are no glitches from a conversion from pickle to JSON. closes #163	2018-09-03 18:07:40 +02:00
Hans-Christoph Steiner	4d13a904f3	use defusedxml to avoid DoS attacks while loading XML	2018-08-29 17:44:54 +02:00
Hans-Christoph Steiner	4503e7a92a	replace unneeded eval() call and support negative versionCodes	2018-08-29 17:24:04 +02:00
Hans-Christoph Steiner	05be4bc814	Merge branch '471-lint-check-unknown-keys' into 'master' ... check for unknown app fields and build flags when parsing yml Closes #471 See merge request fdroid/fdroidserver!554	2018-08-14 10:50:59 +00:00
Hans-Christoph Steiner	53bd5acb09	Merge branch 'make-python-tests-run-independent-of-cwd' into 'master' ... Make python tests run independent of cwd See merge request fdroid/fdroidserver!556	2018-08-14 10:34:47 +00:00
Michael Pöhn	82563850b7	fix test metadata case for Builds app field	2018-08-14 12:31:12 +02:00
Michael Pöhn	6a06944945	better exception and waring message when yaml metadata checks fail; build flags check optional	2018-08-14 12:31:12 +02:00
Michael Pöhn	e83221eb03	improve yaml build flag check cost; added unit test for unknown app fields, builds flags checks when parsing yaml	2018-08-14 12:31:12 +02:00
Michael Pöhn	7d5f4d3ab9	added cwd setup boiler-plate to *.TestCase where missing	2018-08-14 10:34:13 +02:00
Michael Pöhn	f153a61277	made build.TestCase run independant of cwd	2018-08-14 10:10:19 +02:00
Michael Pöhn	37e0dce73b	made index.TestCase run independant of cwd	2018-08-14 10:08:03 +02:00
Michael Pöhn	edb8d105b0	made update.TestCase run independent of cwd	2018-08-14 10:02:15 +02:00
Hans-Christoph Steiner	1b37b521c8	common: fix tests when running without any Android SDK installed ... fdroid/fdroidserver!552 fdroid/fdroidserver#541	2018-08-07 14:51:42 +02:00
Michael Pöhn	55466f1166	fix fixtures in tests/metadata/dump	2018-08-07 00:02:07 +02:00
Michael Pöhn	5d5617bf40	fix: minor bugs in testcases for: common, metadata, update	2018-08-06 23:53:49 +02:00
Michael Pöhn	28ec93c587	fix: run all testcase in tests/run-tests	2018-08-06 23:53:49 +02:00
Hans-Christoph Steiner	64b999ca19	publish: test case that successfully verifies based on Binaries: ... This is a stupid test of the process just to exercise the most basic path. It should always succeed since it is verifying two copes of the same APK.	2018-07-12 23:53:05 +02:00
Hans-Christoph Steiner	b03c4a7353	NoSourceSince Anti-Feature should only be added to index ... Before, it was being added to the metadata, so that rewritemeta would output it. fdroid/fdroidserver!529 #508	2018-07-10 23:32:07 +02:00
Hans-Christoph Steiner	669401640a	tests: only run gpgsign tests if gpg is present and executable ... This allows the full test suite to run in Debian/Ubuntu autopkgtest with only the Depends: installed, and nothing from Recommends:.	2018-06-25 10:52:12 +02:00
Michael Pöhn	752f6a4ba2	applied review suggestions for deploying (build) logs	2018-06-19 15:18:13 +02:00
Michael Pöhn	88e64df3ef	deploying build logs to server after each individual build run	2018-06-19 12:24:52 +02:00
Michael Pöhn	4c53c71fcf	added helper function for uploading build logs with rsync	2018-06-19 12:24:52 +02:00
Michael Pöhn	af980fbe7e	added tests for update_serverwebroot	2018-06-19 12:24:52 +02:00
Michael Pöhn	429bc21da7	make common.TestCase run independant of cwd	2018-06-19 12:24:52 +02:00
Hans-Christoph Steiner	ff90c0246e	fix PEP8 W605 invalid escape sequence ... Python 3.7 will get a lot stricter with escape sequences. They must be valid. * https://lintlyci.github.io/Flake8Rules/rules/W605.html * https://docs.python.org/3/whatsnew/3.6.html#deprecated-python-behavior	2018-05-29 13:51:47 +02:00
Hans-Christoph Steiner	e180e03a5a	tests: fix index-v1.json unicode test	2018-05-28 09:29:11 +02:00
Hans-Christoph Steiner	91068d8e00	tests: rename unicode test file to fix "file name too long" errors ... pristine-tar and mkdir barf on this file.	2018-05-25 17:27:58 +02:00
Hans-Christoph Steiner	5ff1b5ef37	verify: exit with error code if any APK fails to verify	2018-05-25 11:56:13 +02:00
Hans-Christoph Steiner	0a15a9b65b	tests: fix and test fdroid deploy	2018-05-25 10:32:09 +02:00