1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-14 19:10:11 +01:00
Commit Graph

2804 Commits

Author SHA1 Message Date
Hans-Christoph Steiner
0837289935 scanner: fix regex for matching URLs in gradle maven{} blocks
closes #465

This script generated gradle-maven-blocks.yaml:
```python
import os
import re
import yaml

pat = re.compile(r'\smaven\s*{[^}]+}')

finds = set()
for root, dirs, files in os.walk('.'):
    for f in files:
        if '.gradle' in f:
            with open(os.path.join(root, f), errors='surrogateescape') as fp:
                contents = fp.read()
            for m in pat.findall(contents):
                finds.add(m)

with open('finds.yaml', 'w') as fp:
    yaml.dump(sorted(finds), fp, default_flow_style=False)

```
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
6590f3869e scanner: error/warn on dex/gz/zip, closes #394 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
d7b3bca1e7 build: pass --verbose flag through to scanner 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
ee54dbce87 scanner: safely check options, for things using this API
https://gitlab.com/fdroid/fdroidserver/-/issues/771#note_353495799
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
75acf63efa scanner: remove all gradle wrapper files 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
2f5d780c49 scanner: always setup JSON data structure so functions work as API 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
84f225f2f2 scanner: make problem descriptions translationable 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
d898ad0412 scanner: make AARs and JARs trigger an error
refs #491
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner
3bc246ccad scanner: ignore well known image types that are set executable 2020-06-10 19:42:23 +02:00
Michael Pöhn
a97b3ca4dd implement plugin system review suggestsions 2020-06-10 18:43:23 +02:00
Michael Pöhn
77167e098e plugin system: regex instead of import bases plugin parsing 2020-06-10 18:43:11 +02:00
Michael Pöhn
b257a3411a stick to default python sys.path 2020-06-10 12:11:47 +02:00
Michael Pöhn
3a3803ea2d raise excepten when starting broken plugin 2020-06-10 12:11:47 +02:00
Michael Pöhn
bf815251ec rough plugin system implementation 2020-06-10 12:11:47 +02:00
Marcus Hoffmann
581e433832 strip_and_copy_image: abort on broken symlinks
Also guard against other processes removing the files we are about to
copy.

closes fdroid/fdroidserver#783
2020-06-04 12:58:11 +02:00
Hans-Christoph Steiner
13d9a122bf metadata: validate STRING and INT build field types
This converts float/int to string for things like commit: or versionName:.
For versionCode, which must be an integer, it throws an exception if the
data is any other type.
2020-06-02 22:06:01 +02:00
Hans-Christoph Steiner
b2daf96284 lint: fix pyflakes 2020-06-02 22:05:18 +02:00
Hans-Christoph Steiner
6c5887e906
lint: make --force-yamllint error if yamllint is not installed
https://gitlab.com/fdroid/fdroidserver/-/merge_requests/753#note_353829401
2020-06-02 21:56:05 +02:00
Hans-Christoph Steiner
6030445be0 logging.warn() was deprecated in Python 3.3, use logging.warning()
sed -i 's,logging\.warn(,logging.warning(,g' fdroid */*.*

https://docs.python.org/3.3/library/logging.html#logging.Logger.warning
2020-05-27 08:35:14 +02:00
Hans-Christoph Steiner
67332d83a5 scanner: add --json option for outputting machine readable results
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot

This is a second attempt with tests for how `fdroid build` calls the
scanner functions. closes #771.  It was previously merged in !748 then
reverted in 68c072c72e
2020-05-27 08:35:14 +02:00
relan
9ff77cfd1a build: clean buildSrc/build
The buildSrc directory contains custom build logic written in Kotlin.
Before this change we had to 'scandelete' buildSrc/build in the build
recipes becase 'gradle clean' leaves binary artifacts there.
2020-05-26 07:37:20 +00:00
relan
ddfbd1cc47 build: fix directories removal
The 'dirs' array contains a single-level listing of a directory, e. g.
['app', 'build', 'build.gradle', 'gradle', '.gradle']. Multi-component
paths like 'build/tmp' could never be found in this array and thus were
never removed.

Call shutil.rmtree() without checking that the argument is in 'dirs'. If
it exists and is a directory, it'll be removed. Otherwise shutil.rmtree()
will do nothing.
2020-05-26 07:37:20 +00:00
Marcus Hoffmann
68c072c72e Revert "scanner: add --json option for outputting machine readable results"
This reverts commit cdaf62e5d9.

See: https://gitlab.com/fdroid/fdroidserver/-/merge_requests/748#note_347769371
2020-05-24 10:24:40 +02:00
Hans-Christoph Steiner
cdaf62e5d9 scanner: add --json option for outputting machine readable results
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot
2020-05-14 21:58:48 +02:00
Hans-Christoph Steiner
a0e3b01e94 metadata: parsed srclibs must always return a dict as the container 2020-05-14 16:28:13 +02:00
Hans-Christoph Steiner
410901d3bd Merge branch 'yamllint' into 'master'
yamllint

See merge request fdroid/fdroidserver!721
2020-05-14 12:36:19 +00:00
Hans-Christoph Steiner
df563d339a
fix pep8 E741 ambiguous variable name 'l' 2020-05-14 12:11:36 +02:00
Hans-Christoph Steiner
1106795583 deploy: stop uploading to androidobservatory if its already up there 2020-05-14 11:55:09 +02:00
Hans-Christoph Steiner
30f2d62597 import: fix --omit-disable flag, its a boolean 2020-05-14 11:55:09 +02:00
Hans-Christoph Steiner
ba854ab24a support Cordova's gradle file
https://cordova.apache.org/docs/en/latest/guide/platforms/android/index.html
2020-05-14 11:55:09 +02:00
Hans-Christoph Steiner
77367f1bff server: disable extraneous HTTP errors in upload_apk_to_virustotal
This should be in the lowest level so that both the low level call,
upload_apk_to_virustotal(), and the next level up, upload_to_virustotal(),
both inherit this.
2020-05-14 11:55:09 +02:00
Torsten Grote
db96753902
Fix nightly --archive-older command line option 2020-05-11 10:23:26 -03:00
Michael Pöhn
810a6587df add --force-yamllint option to lint 2020-04-25 17:39:23 +02:00
Michael Pöhn
b076e8cba7 add srclib spec parser 2020-04-25 17:22:48 +02:00
Michael Pöhn
ed2c5f6f5b run yamllint on srclibs when running 'fdroid lint' 2020-04-25 15:57:21 +02:00
Michael Pöhn
975538a7a7 make yamllint optional 2020-04-24 15:47:47 +02:00
Michael Pöhn
4e69ff582f run yamllint when parsing failed and also on fdroid lint runs 2020-04-24 15:47:31 +02:00
Jochen Sprickerhof
30c654459f Use ImportError for Python < 3.6 (Closes: !734)
ModuleNotFoundError is a subclass of ImportError, so this should not
change anything.
2020-04-17 23:26:06 +02:00
Michael Pöhn
911e86fc87 support copying yml srclibs to buildserver vm during builds 2020-04-16 11:32:37 +02:00
Michael Pöhn
5741e6930b yml srclibs: support Subdir as list 2020-04-16 11:31:59 +02:00
Michael Pöhn
1ac7d612b1 yml srclibs: support multiline Prepare 2020-04-16 11:31:59 +02:00
Michael Pöhn
aa020af040 srclibs: forward yml parsing error cause 2020-04-16 11:31:59 +02:00
Michael Pöhn
ee3d8d2f18 srclib: yml parsing + tests for yml and txt 2020-04-16 11:31:59 +02:00
Michael Pöhn
2c86d38028 srclib: no spaces in dict key 2020-04-16 11:31:59 +02:00
Michael Pöhn
c6a97939f1 rename parse_srclib to parese_txt_srclib + test case 2020-04-16 11:31:59 +02:00
Hans-Christoph Steiner
c47f9ef123
index: xml.dom.minidom no longer sorts attribs
It seems now that xml.dom.minidom preserves the order of attributes, rather
than sorting them.  We assume alpha-sort, so this manually

This diff in the test suite running on Debian/testing pointed it out:
https://gitlab.com/fdroid/fdroidserver/-/jobs/486970383

```diff
--- /builds/fdroid/fdroidserver/tests/repo/index.xml	2020-04-11 13:36:57.000000000 +0000
+++ repo/index.xml	2020-04-11 13:41:44.000000000 +0000
@@ -1,6 +1,6 @@
 <?xml version="1.0" encoding="utf-8"?>
 <fdroid>
-	<repo icon="fdroid-icon.png" name="My First F-Droid Repo Demo" pubkey="308204e1308202c9a003020102020434597643300d06092a864886f70d01010b050030213110300e060355040b1307462d44726f6964310d300b06035504031304736f7661301e170d3136303931333230313930395a170d3434303133303230313930395a30213110300e060355040b1307462d44726f6964310d300b06035504031304736f766130820222300d06092a864886f70d01010105000382020f003082020a028202010086ef94b5aacf2ba4f38c875f4194b44f5644392e3715575d7c92828577e692c352b567172823851c8c72347fbc9d99684cd7ca3e1db3e4cca126382c53f2a5869fb4c19bdec989b2930501af3e758ff40588915fe96b10076ce3346a193a0277d79e83e30fd8657c20e35260dd085aa32eac7c4b85786ffefbf1555cafe2bc928443430cdbba48cfbe701e12ae86e676477932730d4fc7c00af820aef85038a5b4df084cf6470d110dc4c49ea1b749b80b34709d199b3db516b223625c5de4501e861f7d261b3838f8f616aa78831d618d41d25872dc810c9b2087b5a9e146ca95be740316dcdbcb77314e23ab87d4487913b800b1113c0603ea2294188b71d3e49875df097b56f9151211fc6832f9790c5c83d17481f14ad37915fd164f4fd713f6732a15f4245714b84cd665bdbd085660ea33ad7d7095dcc414f09e3903604a40facc2314a115c0045bb50e9df38efb57e1b8e7cc105f340a26eeb46aba0fa6672953eee7f1f92dcb408e561909bbd4bdf4a4948c4d57c467d21aa238c34ba43be050398be963191fa2b49828bc1e4eeed224b40dbe9dc3e570890a71a974a2f4527edb1b07105071755105edcb2af2f269facfb89180903a572a99b46456e80d4a01685a80b233278805f2c876678e731f4ec4f52075aeef6b2b023efbb8a3637ef507c4c37c27e428152ec1817fcba640ad601cb09f72f0fbe2d274a2410203010001a321301f301d0603551d0e04160414c28bf33dd5a9a17338e5b1d1a6edd8c7d141ed0b300d06092a864886f70d01010b0500038202010084e20458b2aafd7fc27146b0986f9324f4260f244920417a77c9bf15e2e2d22d2725bdd8093ec261c3779c3ca03312516506f9410075b90595b41345956d8eb2786fb5994f195611382c2b99dba13381b0100a30bc9e6e47248bf4325e2f6eec9d789216dc7536e753bf1f4be603d9fa2e6f5e192b4eb988b8cdb0bb1e8668a9225426f7d4636479f73ed24ad1d2657c31e63c93d9679b9080171b3bd1bf10a3b92b80bd790fbf62d3644900cd08eae8b9bf9c2567be98dc8cdd2ae19a8d57a3e3e2de899f81f1279f578989e6af906f80c8c2b67651730ee7e568c1af5bcb845b6d685dc55332a9984aeceaea3b7e883447edf1c76b155d95253e39b9710eaa22efa6c81468829702b5dce7126538f3ca70c2f0ad9a5795435fdb1f715f20d60359ef9a9926c7050116e802df651727447848827815f70bd82af3cedd08783156102d2d8ce995c4c43b8e47e91a3e6927f3505a5d395e6bebb84542c570903eeab4382a1c2151f1471c7a06a34dc4d268d8fa72e93bdcd2dccc4302ecac47b9e7e3d8bc9b46d21cd097874a24d529548018dc190ff568c6aa428f0a5eedff1a347730931c74f19277538e49647a4ad7254f4c1ec7d4da12cce9e1fad9607534e66ab40a56b473d9d7e3d563fd03cad2052bad365c5a29f8ae54f09b60dbca3ea768d7767cbe1c133ca08ce725c1c1370f4aab8e5b6e286f52dc0be8d0982b5a" timestamp="1480431575" url="https://MyFirstFDroidRepo.org/fdroid/repo" version="21">
+	<repo name="My First F-Droid Repo Demo" icon="fdroid-icon.png" url="https://MyFirstFDroidRepo.org/fdroid/repo" version="21" timestamp="1480431575" pubkey="308204e1308202c9a003020102020434597643300d06092a864886f70d01010b050030213110300e060355040b1307462d44726f6964310d300b06035504031304736f7661301e170d3136303931333230313930395a170d3434303133303230313930395a30213110300e060355040b1307462d44726f6964310d300b06035504031304736f766130820222300d06092a864886f70d01010105000382020f003082020a028202010086ef94b5aacf2ba4f38c875f4194b44f5644392e3715575d7c92828577e692c352b567172823851c8c72347fbc9d99684cd7ca3e1db3e4cca126382c53f2a5869fb4c19bdec989b2930501af3e758ff40588915fe96b10076ce3346a193a0277d79e83e30fd8657c20e35260dd085aa32eac7c4b85786ffefbf1555cafe2bc928443430cdbba48cfbe701e12ae86e676477932730d4fc7c00af820aef85038a5b4df084cf6470d110dc4c49ea1b749b80b34709d199b3db516b223625c5de4501e861f7d261b3838f8f616aa78831d618d41d25872dc810c9b2087b5a9e146ca95be740316dcdbcb77314e23ab87d4487913b800b1113c0603ea2294188b71d3e49875df097b56f9151211fc6832f9790c5c83d17481f14ad37915fd164f4fd713f6732a15f4245714b84cd665bdbd085660ea33ad7d7095dcc414f09e3903604a40facc2314a115c0045bb50e9df38efb57e1b8e7cc105f340a26eeb46aba0fa6672953eee7f1f92dcb408e561909bbd4bdf4a4948c4d57c467d21aa238c34ba43be050398be963191fa2b49828bc1e4eeed224b40dbe9dc3e570890a71a974a2f4527edb1b07105071755105edcb2af2f269facfb89180903a572a99b46456e80d4a01685a80b233278805f2c876678e731f4ec4f52075aeef6b2b023efbb8a3637ef507c4c37c27e428152ec1817fcba640ad601cb09f72f0fbe2d274a2410203010001a321301f301d0603551d0e04160414c28bf33dd5a9a17338e5b1d1a6edd8c7d141ed0b300d06092a864886f70d01010b0500038202010084e20458b2aafd7fc27146b0986f9324f4260f244920417a77c9bf15e2e2d22d2725bdd8093ec261c3779c3ca03312516506f9410075b90595b41345956d8eb2786fb5994f195611382c2b99dba13381b0100a30bc9e6e47248bf4325e2f6eec9d789216dc7536e753bf1f4be603d9fa2e6f5e192b4eb988b8cdb0bb1e8668a9225426f7d4636479f73ed24ad1d2657c31e63c93d9679b9080171b3bd1bf10a3b92b80bd790fbf62d3644900cd08eae8b9bf9c2567be98dc8cdd2ae19a8d57a3e3e2de899f81f1279f578989e6af906f80c8c2b67651730ee7e568c1af5bcb845b6d685dc55332a9984aeceaea3b7e883447edf1c76b155d95253e39b9710eaa22efa6c81468829702b5dce7126538f3ca70c2f0ad9a5795435fdb1f715f20d60359ef9a9926c7050116e802df651727447848827815f70bd82af3cedd08783156102d2d8ce995c4c43b8e47e91a3e6927f3505a5d395e6bebb84542c570903eeab4382a1c2151f1471c7a06a34dc4d268d8fa72e93bdcd2dccc4302ecac47b9e7e3d8bc9b46d21cd097874a24d529548018dc190ff568c6aa428f0a5eedff1a347730931c74f19277538e49647a4ad7254f4c1ec7d4da12cce9e1fad9607534e66ab40a56b473d9d7e3d563fd03cad2052bad365c5a29f8ae54f09b60dbca3ea768d7767cbe1c133ca08ce725c1c1370f4aab8e5b6e286f52dc0be8d0982b5a">
 		<description>This is a repository of apps to be used with F-Droid. Applications in this repository are either official binaries built by the original application developers, or are binaries built from source by the admin of f-droid.org using the tools on https://gitlab.com/u/fdroid. </description>
 		<mirror>http://foobarfoobarfoobar.onion/fdroid/repo</mirror>
 		<mirror>https://foo.bar/fdroid/repo</mirror>
@@ -94,9 +94,9 @@
 			<added>2017-12-22</added>
 			<sig>056c9f1554c40ba59a2103009c82b420</sig>
 			<permissions>ACCESS_NETWORK_STATE,ACCESS_WIFI_STATE,CHANGE_WIFI_MULTICAST_STATE,INTERNET,READ_EXTERNAL_STORAGE,WRITE_EXTERNAL_STORAGE</permissions>
-			<uses-permission maxSdkVersion="18" name="android.permission.READ_EXTERNAL_STORAGE"/>
-			<uses-permission maxSdkVersion="18" name="android.permission.WRITE_EXTERNAL_STORAGE"/>
-			<uses-permission-sdk-23 maxSdkVersion="27" name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS"/>
+			<uses-permission name="android.permission.READ_EXTERNAL_STORAGE" maxSdkVersion="18"/>
+			<uses-permission name="android.permission.WRITE_EXTERNAL_STORAGE" maxSdkVersion="18"/>
+			<uses-permission-sdk-23 name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS" maxSdkVersion="27"/>
 		</package>
 	</application>
 	<application id="fake.ota.update">
@@ -182,9 +182,9 @@
 			<added>2013-12-31</added>
 			<sig>eb41d4d6082bb3e81c3d58dbf7fc7332</sig>
 			<permissions>ACCESS_NETWORK_STATE,ACCESS_WIFI_STATE,BLUETOOTH,BLUETOOTH_ADMIN,CHANGE_NETWORK_STATE,CHANGE_WIFI_MULTICAST_STATE,CHANGE_WIFI_STATE,INTERNET,NFC,RECEIVE_BOOT_COMPLETED</permissions>
-			<uses-permission maxSdkVersion="22" name="android.permission.ACCESS_NETWORK_STATE"/>
-			<uses-permission maxSdkVersion="18" name="android.permission.BLUETOOTH_ADMIN"/>
-			<uses-permission-sdk-23 maxSdkVersion="25" name="android.permission.WRITE_SETTINGS"/>
+			<uses-permission name="android.permission.ACCESS_NETWORK_STATE" maxSdkVersion="22"/>
+			<uses-permission name="android.permission.BLUETOOTH_ADMIN" maxSdkVersion="18"/>
+			<uses-permission-sdk-23 name="android.permission.WRITE_SETTINGS" maxSdkVersion="25"/>
 		</package>
 	</application>
 	<application id="obb.main.twoversions">
```
2020-04-15 21:53:39 +02:00
Jochen Sprickerhof
86beac22e2 Use libarchive instead of the Python implementation 2020-04-15 18:27:13 +00:00
Hans-Christoph Steiner
e85573d0e1 Merge branch 'git-push-logging' into 'master'
Show debug messages to give clue why pushing git mirror failed

See merge request fdroid/fdroidserver!731
2020-04-15 12:56:04 +00:00
Hans-Christoph Steiner
f39732987e Merge branch 'patch-1' into 'master'
Import - sudo refresh repos too and use the newer npm

See merge request fdroid/fdroidserver!730
2020-04-15 12:53:56 +00:00
Izzy
46a464a74b
get_app_from_url: add codeberg.org as git url 2020-04-15 09:13:08 +02:00