490 Commits

Author	SHA1	Message	Date
Marcus Hoffmann	68c072c72e	Revert "scanner: add --json option for outputting machine readable results" ... This reverts commit cdaf62e5d9. See: https://gitlab.com/fdroid/fdroidserver/-/merge_requests/748#note_347769371	2020-05-24 10:24:40 +02:00
Hans-Christoph Steiner	cdaf62e5d9	scanner: add --json option for outputting machine readable results ... * makes per-build entries in per-app entries * `fdroid scanner --json --verbose` will output logging messages to stderr * removed " at line N" from one message to make them uniform keys * this will be used in issuebot	2020-05-14 21:58:48 +02:00
Hans-Christoph Steiner	a0e3b01e94	metadata: parsed srclibs must always return a dict as the container	2020-05-14 16:28:13 +02:00
Michael Pöhn	b076e8cba7	add srclib spec parser	2020-04-25 17:22:48 +02:00
Michael Pöhn	d24484a950	simple testcase for common.run_yamllint	2020-04-24 15:47:47 +02:00
Michael Pöhn	4e69ff582f	run yamllint when parsing failed and also on fdroid lint runs	2020-04-24 15:47:31 +02:00
Michael Pöhn	5741e6930b	yml srclibs: support Subdir as list	2020-04-16 11:31:59 +02:00
Michael Pöhn	1ac7d612b1	yml srclibs: support multiline Prepare	2020-04-16 11:31:59 +02:00
Michael Pöhn	58776da694	get tests working	2020-04-16 11:31:59 +02:00
Michael Pöhn	ee3d8d2f18	srclib: yml parsing + tests for yml and txt	2020-04-16 11:31:59 +02:00
Michael Pöhn	fa7885063b	scrlib: add test for getsrclib	2020-04-16 11:31:59 +02:00
Michael Pöhn	286220fe11	srclib: add test for getsrclibvcs	2020-04-16 11:31:59 +02:00
Michael Pöhn	2c86d38028	srclib: no spaces in dict key	2020-04-16 11:31:59 +02:00
Michael Pöhn	c6a97939f1	rename parse_srclib to parese_txt_srclib + test case	2020-04-16 11:31:59 +02:00
Hans-Christoph Steiner	b7901952a1	deploy: make androidobservatory and virustotal functions reusable ... This should not change the logic at all, just make the loop runs into standalone functions.	2020-03-11 22:13:39 +01:00
Hans-Christoph Steiner	ab2291475b	import: mv reusable functions to common.py to avoid import_proxy.py ... import is a strict keyword in Python, so it is not possible to import a module called 'import', even with things like: * import fdroidserver.import * from fdroidserver import import	2020-03-11 13:41:13 +01:00
Marcus	9d24f2e4a7	add opencollective metadata and index field	2020-03-10 14:56:03 +00:00
Hans-Christoph Steiner	202291d66c	integration test for creating and deploying status JSON files ... ----------------------------	2020-02-19 14:50:37 +01:00
Hans-Christoph Steiner	4bb590b6e5	publish: write status JSON	2020-02-19 14:50:24 +01:00
Hans-Christoph Steiner	271b74af7d	fix remove_signing_keys() for Kotlin gradles files (*.gradle.kts)	2020-02-13 22:32:51 +01:00
Hans-Christoph Steiner	3de2d0f56f	add basic test suite for gradlew-fdroid ... !707 fdroiddata#6216 The se.manyver app is licensed MPL, the files came from: 81d247a6cd	2020-02-13 22:32:51 +01:00
Hans-Christoph Steiner	83ffeb855f	prefer build.gradle with Android Plugin as source of package/version/code ... These days, the location that overrides all the others is in the android{} block of the build.gradle file that loads the com.android.application plugin. So this should be the preferred place to read these values. test files GPL licensed: https://github.com/Integreight/1Sheeld-Android-App	2020-02-13 22:32:51 +01:00
Hans-Christoph Steiner	1153ac24fd	import: overhaul URL validation to use urllib.parse ... Python provides us a lovely URL parser with some level of validation built in. The parsed URL is then much easier to validate.	2020-02-13 13:51:52 +01:00
Hans-Christoph Steiner	e9a6c84efd	import: split URL parsing from code cloning ... This makes things testable and easier to follow.	2020-02-13 13:51:52 +01:00
Hans-Christoph Steiner	bfe587979d	import: make it work most of the time with git repos ... This includes real tests too.	2020-02-13 13:51:52 +01:00
Hans-Christoph Steiner	5fb368916f	index: fix no JAR test case that fails on example.org ... http://example.org/index-v1.jar now returns the HTTP header "Content-Encoding: gzip" but then the reply is plain HTML. That triggers a ContentDecodingError instead of an HTTPError, so this changes the test to success on any RequestsException.	2020-02-05 20:33:40 +01:00
Hans-Christoph Steiner	427856b5fd	tests: use same mirrors order in all tests ... This makes it so running `../fdroid update --nosign --pretty` in tests/ no longer creates a diff in the tests/*/index* files. It matches the order set in tests/run-tests.	2020-01-31 15:38:10 +01:00
Hans-Christoph Steiner	6ce22bff56	update: strip newlines from name/summary/video entries ... These entries are hardcoded as a single line in all the app stores, so newlines should be stripped to get the data simple to use. This is in contrast with the on-disk format for Fastlane and Triple-T, which includes a newline in the title.txt and short_description.txt files. I think all files in those systems are normalized to end in a newline.	2020-01-31 15:37:59 +01:00
Hans-Christoph Steiner	fdede914cd	tests: properly name CheckupdatesTest class	2020-01-31 15:37:32 +01:00
Hans-Christoph Steiner	3df276cc3c	fix all bandit B310 urllib_urlopen ... "Audit url open for permitted schemes. Allowing use of ‘file:’’ or custom schemes is often unexpected." https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b310-urllib-urlopen	2020-01-31 15:37:30 +01:00
Marcus Hoffmann	8961485125	remove dscanner subcommand ... It's unused and unmaintained. It could potentially be revived as a plugin at a later point.	2020-01-31 14:17:12 +01:00
Michael Pöhn	f2a80ffa3d	fix tests on old python version	2020-01-28 11:43:09 +01:00
Michael Pöhn	b25eeb66a1	fix code style	2020-01-28 11:43:09 +01:00
Michael Pöhn	f21481ca81	add some simple tests for main	2020-01-28 11:43:09 +01:00
Hans-Christoph Steiner	e76a0c9d6a	git_mirror_size_limit config option to set max git mirror size ... GitHub and GitLab have some kinds of limits on how big a git repo can be, this makes that option configurable. This also is very useful for tests.	2020-01-14 11:36:08 +01:00
Gerhard Olsson	cf9bff1d56	AutoUpdateMode: Do not include + in suffix ... Add suffix to version only	2020-01-02 13:27:12 +00:00
Marcus Hoffmann	83526e09a3	update: fix unarchiving of allowed disabled algorithm ... 6d0b1bbe6fae0909683f2c6a154515bc4bfcb674 didn't handle the allow_disabled_algorithm case at all, so we add it back. This additionally fixes a (previously existing) bug where setting allow_disabled_algorithms to True didn't move apks back from archive to repo. Introduce a new test for this. The disabled_algorithm archiving logic is still all over the place so ideally that needs a future refactor.	2020-01-01 15:25:43 +01:00
Marcus Hoffmann	7f6efa74f5	tests: add test for correctly handling CVC when archiving	2019-12-24 17:35:54 +01:00
Hans-Christoph Steiner	7d40e89341	checkupdates: split out vercode parsing into testable function	2019-12-03 23:51:48 +01:00
Jochen Sprickerhof	bbee2cf707	Add unit test for string_is_integer()	2019-12-03 21:49:44 +01:00
Mike Hardy	3648ef1f2e	Update scanner.TestCase to match new regex regime ... scanning for analytics is now handled by firebase/play-services regexes	2019-11-02 18:54:54 +00:00
Hans-Christoph Steiner	0a8af2c55f	Merge branch 'fix-readline-in-srclibs' into 'master' ... build: fix bad regexs when removing signingConfig from srclibs See merge request fdroid/fdroidserver!686	2019-10-25 08:18:08 +00:00
Hans-Christoph Steiner	afaa24f2fd	build: fix bad regexs when removing signingConfig from srclibs ... I went through the source of all apps in fdroiddata for examples, and found some that use readLine() for things totally unrelated to signingConfigs. https://gitlab.com/fdroid/fdroiddata/merge_requests/4775#note_234132902	2019-10-23 12:44:47 +02:00
Michael Pöhn	7fa3c34e5b	update tests for fixed log deployment+changelog	2019-10-15 15:19:18 +02:00
Hans-Christoph Steiner	5904aef5a7	update: parse Triple-T v2.x layout ... Thanks to the very nice example from Piwigo, included here under the GPLv3+ https://github.com/Piwigo/Piwigo-Android https://gitlab.com/fdroid/fdroiddata/merge_requests/5579#note_225834868	2019-10-04 14:11:40 +02:00
Hans-Christoph Steiner	3801db064a	lint: improve HTTPS check ... It was missing some domains, so I added another rule. @IzzySoft pointed out it was redudnant, so this removes the redudant rule and fixes the original. https://gitlab.com/fdroid/fdroidserver/merge_requests/681/diffs#note_225263464	2019-10-04 11:12:48 +02:00
Hans-Christoph Steiner	af4e231f7d	lint: enforce HTTPS on GitHub and GitLab pages ... The always provide HTTPS, so let's enforce it.	2019-10-04 11:12:48 +02:00
Hans-Christoph Steiner	17320c23f4	lint: don't trip up on projects with 'master' in the name ... https://gitlab.com/fdroid/fdroiddata/merge_requests/5557#note_223283359	2019-10-04 11:12:48 +02:00
Michael Pöhn	d5ab303d83	lint: license override config option + require FSF/OSI approved licenses by default	2019-10-02 22:14:09 +02:00
Hans-Christoph Steiner	c7048f2c39	update: log errors on bad graphics, and then ignore the file ... Python PIL is not so tolerant, so bad EXIF causes crashes: File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2088, in main insert_localized_app_metadata(apps) File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 978, in insert_localized_app_metadata _strip_and_copy_image(os.path.join(root, f), destdir) File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 754, in _strip_and_copy_image in_image = Image.open(fp) File "/usr/lib/python3/dist-packages/PIL/Image.py", line 2687, in open % (filename if filename else fp)) OSError: cannot identify image file <_io.BufferedReader name='build/org.sw24softwares.starkeverben/fastlane/metadata/android/en-US/images/featureGraphic.png'>	2019-09-26 22:06:21 +02:00
Hans-Christoph Steiner	cf2c1d0270	run-tests needs a source of APKs to function ... The test runs on jenkins.debian.net were failing because they specified a dir that did not exist.	2019-09-25 12:25:39 +02:00
Hans-Christoph Steiner	25548023e0	gitlab-ci: check gradle checksums against official list	2019-09-12 14:18:07 +02:00
relan	079754c56c	scanner: add a test for the local Debian Maven repo	2019-08-29 21:33:12 +03:00
Hans-Christoph Steiner	70e7e720b9	update: use graphics filename with hash in index to support caching ... Using a filename based on the hash of the contents means that the caching algorithms for fdroidclient and browsers can safely cache the file forever using the filename, since this guarantees that the contents will never change for a given filename. This does not cover screenshots, only icon.png, featureGraphic.png, tvBanner.png, and promoGraphic.png. fdroidserver#689 fdroid-website!453	2019-08-29 08:23:51 +02:00
Hans-Christoph Steiner	508af00e84	update: only copy graphics and screenshots if mtime/size has changed ... Instead of copying every time, trust the filesystem to tell us when the file has changed.	2019-08-29 08:23:47 +02:00
Hans-Christoph Steiner	f13c41e3ba	update: remove Provides: from template.yml test case ... !654 dcf3837bcb	2019-08-28 15:57:40 +02:00
Hans-Christoph Steiner	7090ce63dd	Merge branch 'remove-provides-in-yml' into 'master' ... Remove `Provides:` in yaml metadata Closes #676 See merge request fdroid/fdroidserver!654	2019-08-28 13:39:59 +00:00
Michael Pöhn	8e5232076f	do not delete yml metadata when raumel not installed	2019-08-27 15:38:40 +02:00
Michael Pöhn	0885303672	add additional rewritemeta test for yml	2019-08-27 15:38:40 +02:00
Michael Pöhn	3951d93196	add rewritemeta.TestCase	2019-08-27 15:38:40 +02:00
Michael Pöhn	39c589fb85	Merge branch 'update-create-template' into 'master' ... make metadata template behave well for empty values Closes #681 See merge request fdroid/fdroidserver!657	2019-07-24 11:02:38 +00:00
Michael Pöhn	d0368d0ad8	common add parse_androidmanifests_ignore test	2019-07-23 22:44:45 +02:00
Michael Pöhn	bad888856a	checkupdates: add check_http ignore test	2019-07-23 22:01:18 +02:00
Michael Pöhn	23280b6029	checkupdates add check_http test	2019-07-23 22:01:18 +02:00
Michael Pöhn	1c7af1dc2c	add test for checkupdates_app()	2019-07-23 22:01:18 +02:00
Michael Pöhn	e2fed09af1	make metadata template behaves well for empty values	2019-07-22 01:34:55 +02:00
Michael Pöhn	dcf3837bcb	parse yaml: ignore (and warn) deprecated field: Provides	2019-07-15 15:45:02 +02:00
Michael Pöhn	6e48663230	test that write yaml does not write provides	2019-07-11 03:35:23 +02:00
Michael Pöhn	723bd110a6	test that yaml parsing does not accept provides	2019-07-11 03:35:23 +02:00
Hans-Christoph Steiner	051596dd0d	tests: conditionally disable tests that can't work with apksigner ... apksigner treats MD5 signatures as valid, fdroid does not.	2019-07-10 14:35:03 +02:00
Hans-Christoph Steiner	26af94974a	Merge branch 'rsync-buildslogs-to-webroot-repo' into 'master' ... build: rsync buildlogs to <webroot>/repo See merge request fdroid/fdroidserver!651	2019-07-10 08:55:04 +00:00
Michael Pöhn	2c87b5e6f9	deploy build logs: no timestamps	2019-07-06 16:48:56 +02:00
Michael Pöhn	f30983368c	build: rsync buildlogs to <webroot>/repo	2019-07-06 16:48:56 +02:00
Hans-Christoph Steiner	a248a69692	tests: skip disabled_algorithms test when apksigner is present ... apksigner doesn't treat MD5 signatures as deprecated, so that portion of the tests would always fail.	2019-07-04 16:45:50 +02:00
Hans-Christoph Steiner	a9aa8788e0	tests: only run source tarball test if running from git clone	2019-07-03 22:33:15 +02:00
Hans-Christoph Steiner	aa1e958360	tests: only run hooks/pre-commit if its present (not in source tarball)	2019-07-03 20:46:30 +02:00
Hans-Christoph Steiner	a0f5ee661e	tests: common.test_sign_apk requires aapt to run	2019-07-03 09:07:36 +02:00
Hans-Christoph Steiner	57b9d1e316	tests: handle when apksigner considers MD5 signatures valid	2019-07-02 22:17:06 +02:00
Michael Pöhn	66105de833	improve litecoin validation + tests	2019-05-28 11:04:43 +02:00
Michael Pöhn	06cec2041d	improve bitcoin validation regex + testcases	2019-05-07 22:43:05 +02:00
Hans-Christoph Steiner	dd2f9d60f8	publish: fix stupid error in repro-signing and add integration test ... stoopid mistake in ea84014f9b reported by @CiaranG	2019-04-11 14:06:51 +02:00
Marcus	7272689ced	Merge branch 'master' into 'master' ... Added newer ndks, gradles, latest sdk-license, and update java 1.8 version See merge request fdroid/fdroidserver!637	2019-04-11 11:42:18 +00:00
Michael Pöhn	67731470cc	Revert "Merge branch 'write-yaml-overhaul' into 'master'" ... This reverts merge request !630	2019-04-01 10:24:00 +00:00
Taco	457cf22361	Added newer ndks, gradles, latest sdk-license, and update java 1.8 version	2019-03-30 17:10:21 -04:00
Michael Pöhn	2683b37044	yml metadata write: do not use local functions	2019-03-19 01:01:18 +01:00
Michael Pöhn	881a79fa84	test writing all yaml fields	2019-03-18 22:45:35 +01:00
Michael Pöhn	63afc0acb5	use pyyaml for writing metadata instead of ruamel	2019-03-18 22:45:35 +01:00
Hans-Christoph Steiner	d03b121152	update: allow tests to pass when apksigner is not installed ... This is only for the v2/v3 signatures. fdroid/fdroidserver#627	2019-02-12 12:43:21 +01:00
Hans-Christoph Steiner	50ca3967cc	update: fix tests when running without apksigner	2019-02-03 15:50:39 +01:00
Hans-Christoph Steiner	17dc231dc9	update: fix running without androguard ... Soon, we can rip out all the aapt parsing stuff, but not yet!	2019-02-03 15:50:39 +01:00
Michael Pöhn	dd695c650e	update: treat target and min sdk version as int	2019-02-01 09:56:21 +01:00
Hans-Christoph Steiner	d96f5ff660	support APK Signature V2 when apksigner is installed ... This was done with much help from @uniqx. This is the first level of supporting APK Signatures v1, v2, and v3. This is enough to include APKs with any combo of v1/v2/v3 signatures. For this to work at all, apksigner and androguard 3.3.3+ must be installed. closes #399	2019-02-01 09:17:56 +01:00
Francesco Cervigni	984d276c1b	Added test aapt output files for build-tools 28.0.3	2019-01-13 19:27:02 +01:00
Michael Pöhn	31ca2092a1	yaml parsing: script build flags can now be lists	2018-11-29 21:53:37 +01:00
Michael Pöhn	c15a7508e7	write yaml script metadata as lists	2018-11-29 21:53:37 +01:00
Michael Pöhn	a21635ae2e	fix metadata test case: write yaml prebuild	2018-11-29 21:53:37 +01:00
Michael Pöhn	723815a25b	fix metadata.Testcase:test_rewrite_yaml_special_build_params	2018-11-29 21:53:37 +01:00
Michael Pöhn	d0a129c216	add test for parsing build field prebuild as string	2018-11-29 21:53:37 +01:00
Michael Pöhn	942de28fa5	yaml metadata: split prebuild build field to list	2018-11-29 21:53:37 +01:00
Hans-Christoph Steiner	57556aceee	remove redundant open() arg: encoding='utf8' ... By default, open() returns a str: https://docs.python.org/3/library/functions.html#open By default, str is UTF-8: https://docs.python.org/3/library/stdtypes.html#str This used to matter on Python 2.x, but this code is 3.x only now.	2018-10-19 15:01:34 +02:00