kamp/fdroidserver
1
0
Fork 0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-19 05:10:10 +01:00
Code Activity
7,942 Commits 1 Branch 59 Tags 43 MiB
b0c05842d8
Commit Graph

258 Commits

Author SHA1 Message Date
Hans-Christoph Steiner
052e22284b gitlab-ci: show clear error message when one step of job fails 2020-05-11 17:40:21 +02:00
Jochen Sprickerhof
86beac22e2 Use libarchive instead of the Python implementation 2020-04-15 18:27:13 +00:00
Marcus
9d24f2e4a7 add opencollective metadata and index field 2020-03-10 14:56:03 +00:00
Hans-Christoph Steiner
ed46afe262
gitlab-ci: ensure git is installed for pip_install job 2020-02-20 16:40:31 +01:00
Hans-Christoph Steiner
3de2d0f56f add basic test suite for gradlew-fdroid 
!707
fdroiddata#6216

The se.manyver app is licensed MPL, the files came from:
81d247a6cd
 2020-02-13 22:32:51 +01:00
Hans-Christoph Steiner
0fa1f91a23
gitlab-ci: long timeout and many retries for pip installs 2020-01-31 15:38:05 +01:00
Hans-Christoph Steiner
3df276cc3c
fix all bandit B310 urllib_urlopen 
"Audit url open for permitted schemes. Allowing use of ‘file:’’ or custom
schemes is often unexpected."

https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b310-urllib-urlopen
 2020-01-31 15:37:30 +01:00
Hans-Christoph Steiner
d8f3d94997
gitlab-ci: remove dscanner exclusions from bandit 2020-01-31 15:37:28 +01:00
Hans-Christoph Steiner
cca78114cb
gitlab-ci: fedora no longer installs difftools by default 2019-11-05 15:30:34 +01:00
Hans-Christoph Steiner
8d3512763d
gitlab-ci: use a template for a complete apt CI setup 
# Conflicts:
#	.gitlab-ci.yml
 2019-09-23 11:50:51 +02:00
Hans-Christoph Steiner
0e40387805
gitlab-ci: switch debian/testing back to pure testing, no sid 
The sid packages were needed as a temporary workaround while the new
androguard packages were settling into Debian.
 2019-09-23 11:33:00 +02:00
Hans-Christoph Steiner
25548023e0
gitlab-ci: check gradle checksums against official list 2019-09-12 14:18:07 +02:00
Hans-Christoph Steiner
a9b8687e94 gitlab-ci: the ubuntu_lts test also tests the PPA 2019-07-02 22:17:06 +02:00
Hans-Christoph Steiner
e2351f6c53 gitlab-ci: move pip job to Xenial, Trusty is over 2019-07-02 22:17:06 +02:00
Michael Pöhn
716f84ec5e use actually working bandit version when running tests on alpine 2019-05-24 23:14:20 +02:00
Taco
457cf22361 Added newer ndks, gradles, latest sdk-license, and update java 1.8 version 2019-03-30 17:10:21 -04:00
Michael Pöhn
102340ec5a fix fedora ci tests: install @development-tools 
Sometime pip dependencies required a C compiler because they need to
compile something during installation.
 2019-03-18 16:01:34 +01:00
Hans-Christoph Steiner
7133cede89 gitlab-ci: add cache, extend timeouts/retries to stabilize Fedora job 
This is happening too often:
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
  Curl error (28): Timeout was reached for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f29&arch=x86_64 [Connection timed out after 30001 milliseconds]
ERROR: Job failed: exit code 1

https://gitlab.com/fdroid/fdroidserver/-/jobs/152719443

# Conflicts:
#	.gitlab-ci.yml
 2019-01-29 13:40:55 +01:00
Hans-Christoph Steiner
3b84a82728 gitlab-ci: fix locale checks in pip_install job 2019-01-29 13:01:25 +01:00
Hans-Christoph Steiner
6925083e57 gitlab-ci: compile locales using gettext and babel 2019-01-28 14:42:29 +01:00
Hans-Christoph Steiner
9a524fa85d bump RELEASE_COMMIT_ID for fixing metadata_v0 test 
fdroid/fdroidserver!564
 2018-12-06 12:30:46 +01:00
Hans-Christoph Steiner
e10c12ffc4 gitlab-ci: fix tests on Fedora 
* install `which` and `find` commands used in ./tests/run-tests
* sdkmanager on Fedora needs JAVA_HOME to be set
 2018-12-06 12:15:27 +01:00
Hans-Christoph Steiner
d9e9cc20aa gitlab-ci: install python3-defusedxml in debian_testing and ubuntu_lts 
fdroid/fdroidserver!578
 2018-09-19 16:48:00 +02:00
Hans-Christoph Steiner
0cd1e0b172 gitlab-ci: include fdroid in bandit scans 2018-09-07 10:34:56 +02:00
Hans-Christoph Steiner
3ffe2860f3 gitlab-ci: add 'bandit' security scanner to all runs 
bandit is used by Radically Open Security and is part of the GitLab Ultimate
Static Application Security Testing (SAST) suite.

https://docs.gitlab.com/ee/user/project/merge_requests/sast.html
 2018-08-29 17:48:06 +02:00
Michael Pöhn
14730be812 bump RELEASE_COMMIT_ID for fixing metadata_v0 test 2018-08-02 18:18:07 +02:00
Hans-Christoph Steiner
60ee69b8bd gitlab-ci: pylint<2.0 workaround to avoid typed-ast's gcc requirement 
To keep those tests light and small, no gcc please!
* https://gitlab.com/eighthave/fdroidserver/-/jobs/82274815
 2018-07-18 17:11:21 +02:00
Hans-Christoph Steiner
e3bd293f43 gitlab-ci: point to fixed NoSourceSince commit 2018-07-10 23:48:25 +02:00
Hans-Christoph Steiner
6ca09e1bb2 gitlab-ci: update metadata_v0 test for latest metadata fields 
This is a lot easier than trying to do some elaborate multiline sed regexp!

https://gitlab.com/fdroid/fdroidserver/merge_requests/529#note_86955227
 2018-07-10 17:26:07 +02:00
Hans-Christoph Steiner
a87df29135 gitlab-ci: pep8 has been replaced by pycodestyle 2018-05-29 11:28:08 +02:00
Hans-Christoph Steiner
14127bf418 gitlab-ci: combine all lint/syntax/safety checks into a single job 
This should make it easier to accept merge requests where there are only
cosmetic problems with them.  pep8/pylint/pyflakes runs can then be disabled
in the 'test' job by not installing the in the ci-images-server base image.
 2018-05-25 12:12:40 +02:00
Hans-Christoph Steiner
74fb07b302 gitlab-ci: switch pyup_io_safety_check to Alpine to be lighter 2018-05-25 09:54:08 +02:00
Hans-Christoph Steiner
6570e85a2b gitlab-ci: new test case of pip installs on Ubuntu/trusty 
Ubuntu/trusty is used as the base image for CI systems like Travis, as well
as Microsoft Subsystem for Linux.  SO we need to provide working options.
 2018-05-14 15:25:33 +02:00
Hans-Christoph Steiner
54b21a6d22 move pylint run to standalone gitlab-ci job 2018-05-14 15:25:33 +02:00
Hans-Christoph Steiner
3c9cc59c38 gitlab-ci: include fdroiddata yml files in metadata_v0 test 2018-05-14 15:25:33 +02:00
Hans-Christoph Steiner
0bd276de1c gitlab-ci: add new security scanner pyup.io/safety 
https://pyup.io/safety/
 2018-05-14 15:25:33 +02:00
Hans-Christoph Steiner
21a18cf26b gitlab-ci: fix ubuntu_lts test now that its bionic rather than xenial 
The new ubuntu:latest image is not set up for non-interactive, so this
gitlab-ci job now needs to do that.
 2018-05-04 15:11:25 +02:00
Hans-Christoph Steiner
02107cc5bc gitlab-ci: make ubuntu_lts job test of fdroid/fdroidserver PPA 
Now that androguard is working, there should be no need for a specific aapt
version.  The aapt included in Ubuntu LTS should always work fine when
androguard handles the bulk of the work.
 2018-03-09 12:00:31 +01:00
Hans-Christoph Steiner
01a73071c7 gitlab-ci: set metadata_v0 test to use 1.0.2 as the baseline 2018-02-23 22:48:44 +01:00
Hans-Christoph Steiner
f420a037d5 gitlab-ci: remove apt-get progress dumps from build log 2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner
0e9252db37 gitlab-ci: try to download PPA keys until they succeed 
This download occasionally fails, so this keeps retrying till it succeeds.
The CI job has a time limit, so no need to figure out an exit condition.
 2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner
89498208fc gitlab-ci: test against latest build-tools 27.0.3 2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner
11bed21071 gitlab-ci: androguard from Debian/unstable til it stabilizes 2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner
9f553186e8 gitlab-ci: switch debian_testing to use androguard 2018-02-20 16:30:06 +01:00
Hans-Christoph Steiner
05eef5a454 metadata: add new Translation app field for URL for contributions 
This is a field requested from a bunch of translators so they can easily
find where to translate apps:
https://forum.f-droid.org/t/translation-field-for-app-entries/1403/5
 2018-02-05 15:45:12 +01:00
Hans-Christoph Steiner
b170ef7ce8 fix metadata test: remove timeout= from comparison 2018-01-30 11:20:30 +01:00
Hans-Christoph Steiner
513c95894c build: remove unused, unmaintained Kivy build method 
This code has never been used and contains some insecure uses of shell=True
Building Kivy apps should be done with the buildozer=yes method.  The
buildozer method should probably be moved to a provisioner once that is in
place.
 2018-01-23 23:16:05 +01:00
Hans-Christoph Steiner
22563bdf17 gitlab-ci: make metadata_v0 test work even when tags are missing 
This uses the commit ID of the release tags, rather than the release tag
itself so that contributor forks do not need to include the tags in them
for this test to work.

The COMMIT_ID should be bumped after each release, so that the list of sed
hacks needed does not continuously grow.
 2018-01-22 14:00:20 +01:00
Hans-Christoph Steiner
fba4fa6ca6 get minimum aapt version from fdroidserver/common.py for CI tests 2018-01-03 14:54:30 +01:00
Hans-Christoph Steiner
c8e32c2802 gitlab-ci: and one last stoopid error in debian_testing target 2017-12-29 16:06:45 +01:00
Hans-Christoph Steiner
c7762ad7d7 gitlab-ci: fix syntax error 2017-12-29 15:35:04 +01:00
Hans-Christoph Steiner
6816ef15dd gitlab-ci: temp disable some tests on Debian/testing 
build-tools needs an update before these tests will work again.
 2017-12-29 15:29:28 +01:00
Hans-Christoph Steiner
b70ede6761 gitlab-ci: set Fedora build to use a supported build-tools version 
Using 25.0.2, not all of the permissions were being output:

--- /builds/fdroid/fdroidserver/tests/repo/index.xml	2017-12-28 22:33:53.624704459 +0000
+++ repo/index.xml	2017-12-28 22:41:52.207849097 +0000
@@ -35,7 +35,6 @@
 			<added>2017-12-22</added>
 			<sig>2d337e40aef77564bf62781ac424595c</sig>
 			<permissions>ACCESS_NETWORK_STATE,ACCESS_WIFI_STATE,CHANGE_WIFI_MULTICAST_STATE,INTERNET,READ_EXTERNAL_STORAGE,WRITE_EXTERNAL_STORAGE</permissions>
-			<uses-permission maxSdkVersion="18" name="android.permission.READ_EXTERNAL_STORAGE"/>
 			<uses-permission maxSdkVersion="18" name="android.permission.WRITE_EXTERNAL_STORAGE"/>
 			<uses-permission-sdk-23 maxSdkVersion="27" name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS"/>
 		</package>
 2017-12-29 15:12:49 +01:00
Hans-Christoph Steiner
8b9b18ceed fix metadata_v0 tests 
The old metadata format didn't know anything of LiberapayID, so no need to
check for it.
 2017-12-12 16:54:35 +01:00
Hans-Christoph Steiner
05abbfbabd gitlab-ci: move sdist test run to new fedora job 
A full run of the test suite takes quite a bit of time.  This removes one
of the 3 runs from the main 'tests' job, and puts it into the Fedora job.
That test run is mostly to make sure the setup.py and source tarball are
correctly, so that doesn't affect merge requests very often.

This also tests `pip install --user`, which was not really being tested
before.
 2017-12-06 20:20:17 +01:00
Hans-Christoph Steiner
dda9c8b774 gitlab-ci: add pip install test on Arch Linux 2017-12-06 20:20:17 +01:00
Hans-Christoph Steiner
1b1475c982 gitlab-ci: add test runs on Ubuntu, Debian/testing, Fedora 2017-12-06 12:30:47 +01:00
Hans-Christoph Steiner
30c0d5f4d5 gitlab-ci: ensure release tag is present for metadata_v0 tests 
Forks won't necessarily have all the tags, so this ensures that they are
present.
 2017-09-15 11:35:30 +02:00
Marcus Hoffmann
baef08725b CI: bump fdroid version to test against to 0.8 remove legacy handling 
Bump to latest released version, remove all handling of things that
changed in a backward incompatible way before.
 2017-08-31 21:06:34 +02:00
Hans-Christoph Steiner
619c66e1dc gitlab-ci: filter new build fields in metadata_v0 test 2017-07-19 16:24:28 +02:00
Hans-Christoph Steiner
dfb07808d3 support configing buildserver VM per-build with sudo= 
This adds the 'sudo' build field, which is just a script that is run as
root.  For more info, see the issue that this closes:

refs #318
closes #317
 2017-07-06 17:29:23 +02:00
Hans-Christoph Steiner
df99c85ca6 support manually adding per-build antiFeatures in metadata 
For cases like the OpenVPN vuln that was recently announced, it is useful
for fdroiddata maintainers to be able to mark builds that have known
vulnerabilities.
 2017-07-06 12:25:48 +02:00
Hans-Christoph Steiner
b29fcde16b gitlab-ci: remove hacks need to support Debian/jessie 2017-07-04 14:53:21 +02:00
Hans-Christoph Steiner
607d1d2553 gitlab-ci: switch to new Debian/stretch image 
ci-images-server!1
 2017-07-03 20:41:26 +02:00
Hans-Christoph Steiner
f2432f7fa4 gitlab-ci: apt upgrade so that tests run with current updates 
The MD5 signature stuff was failing in tests because the CI image was
using a quite old version of Java's jarsigner, which had not yet disabled
MD5.
 2017-07-03 10:02:51 +02:00
Hans-Christoph Steiner
7f9f47496f gitlab-ci: fix metadata_v0 test to run on the right commits 
Checking out master will often mean its testing the wrong commit, since
merge requests rarely are in master.
 2017-06-28 23:14:57 +02:00
Hans-Christoph Steiner
6e72ffb043 gitlab-ci: fix metadata_v0 test for new buildozer build flag 
Just remove it, since 0.7.0 does not know about that build flag.
 2017-06-22 16:33:11 +02:00
Hans-Christoph Steiner
5e7e17e800 gitlab-ci: properly quote colons 2017-06-20 20:27:23 +02:00
Hans-Christoph Steiner
4939b8a530 gitlab-ci: ignore blank Description in metadata_v0 test 
We'll eventually be moving to having the internal representation use a
'' or None when there is no Description, so no use in fixing this.
 2017-06-20 18:15:12 +02:00
Hans-Christoph Steiner
2c6945dac7 gitlab-ci: add index v0 metadata parsing test 
This test is very handy for making sure the old index.xml v0 format does
not inadvertantly change.
 2017-05-30 21:13:49 +02:00
Hans-Christoph Steiner
e58ad330f4 encode filenames as bytes to handle all locale setups 
This was failing on environments that did not have any LANG or LC_* locale
variables set.  This is a valid setup, and is common in headless setups, so
it needs to be handled.

This also adds a new pass of the test suite without the locale env vars set
so that this situation is also tests on gitlab-ci, not only gpjenkins.

The error this caused was:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-18: ordinal not in range(128)
 2017-04-03 20:24:00 +02:00
Hans-Christoph Steiner
36272656fc gitlab-ci: workaround "ImportError: No module named 'packaging'" 
https://github.com/pypa/setuptools/issues/937

fdroid/ci-images#1
 2017-02-06 17:28:07 +01:00
Daniel Martí
926aafa58b Bump CI image, now with build-tools 25.0.2 2016-12-23 17:46:09 +01:00
Daniel Martí
2e96fc1cd9 CI: use new docker image with texinfo 
docs/gendocs.sh uses makeinfo.
 2016-10-31 14:02:39 +00:00
Daniel Martí
148f077ad8 Bump ci images and default build-tools 2016-10-23 17:04:58 +01:00
Daniel Martí
fd7a8c83c8 all: bump to build-tools 24.0.2 
Update the docker image to include it there too.
 2016-09-10 11:45:39 +02:00
Daniel Martí
a006f277bc CI: bump docker image 2016-08-21 11:11:01 +02:00
Daniel Martí
70777233d0 CI: bump docker image 2016-08-01 00:57:32 +02:00
Daniel Martí
5de678e48b all: switch to jdk8 as default 
Also, remove jdk7 as it will become unused. We added jdk8 for
retrolambda, and now that we will use jdk8 as the default, jdk7 is
unnecessary as retrolambda can work fine with just jdk8.

This removes it from the buildserver, and the new CI image also only has
jdk8 from jessie-backports.

Fixes #185.
 2016-06-21 11:47:37 +01:00
Daniel Martí
f68830127e CI: bump docker image 2016-06-13 19:45:12 +01:00
Daniel Martí
4ed95b1fa6 CI: remove debug ls 2016-06-13 19:44:52 +01:00
Hans-Christoph Steiner
733ef52424 gitlab-ci: make sure pip3 install dirs exist 
It doesn't want to create them itself.
 2016-06-09 12:15:11 +02:00
Daniel Martí
38790830f4 CI: Update image, not using base anymore 
Since base is now 200MB smaller, this one is smaller too. The
fdroidserver-only deps are now in the client image - mainly python3-dev,
gcc and all the build deps for stuff like pillow.
 2016-04-29 20:27:39 +01:00
Daniel Martí
bf0e5dde71 CI: Install venv instead of virtualenv 2016-04-14 16:16:58 +01:00
Daniel Martí
ec9087d057 Add and switch to build-tools 23.0.3 
Also bump to the CI image which contains it.
 2016-04-05 10:49:30 +01:00
Daniel Martí
a6e857985a CI: Use the new base image 
Lighter, since it doesn't include all the stuff required to build and
test the client that we don't need here.
 2016-03-29 12:44:00 +01:00
Daniel Martí
221ec07a69 CI: Bump image, now has -dev and gcc 
Also, use a specific tag instead of "latest" to avoid breaking old
tags/commits when updating the image.
 2016-03-17 13:36:06 +00:00
Daniel Martí
f2681dcb36 CI: Use pip install to grab all deps 2016-03-11 13:27:01 +00:00
Daniel Martí
c2cf6b1982 CI: Install missing gcc and python-dev packages 2016-03-05 19:22:23 +00:00
Daniel Martí
d4f7097638 CI: No longer needed to install any deps 
All the deps come installed in the image now. This saves up a little
time (especially the apt-get update) and makes the CI script much
simpler.
 2016-02-29 23:38:54 +00:00
Daniel Martí
6264222fb6 CI: Use our own image 2016-02-29 19:56:02 +00:00
Daniel Martí
8106d4ffde CI: Install Android SDK only if necessary 2016-02-27 19:56:32 +00:00
Daniel Martí
66e82cb077 Bump build-tools to 23.0.2 2015-11-05 11:56:13 +01:00
Daniel Martí
82624cd7ca Bump sdk to 24.4.1 2015-10-24 19:02:39 +02:00
Daniel Martí
055759cf76 Rewrite scanner logic 
Initially, the scanner used libmagic which used magic numbers in the file's
content to detect what kind of file it appears to be. Since that library isn't
available on all systems, we added support for two other libraries, mimetypes
amongst them.

The issue with mimetypes is that it only uses the file's extension, not its
actual content. So this ends in variable behaviour depending on what system
you're using fdroidserver on. For example, an executable binary without
extension would be ignored if mimetypes was being used.

We now drop all libraries - mimetypes too as it depends on the system's
mime.types file - and instead check extensions ourselves. On top of that, do
a simple binary content check to find binary executables that don't have an
extension.

The new in-house code without any dependencies doesn't add any new checks, so
no builds should break. The current checks still work:

 % fdroid scanner app.openconnect:1029
[...]
Found executable binary at assets/raw/armeabi/curl
Found executable binary at assets/raw/mips/curl
Found executable binary at assets/raw/x86/curl
Found JAR file at lib/XposedBridgeApi-54.jar
Found JAR file at libs/acra-4.5.0.jar
Found JAR file at libs/openconnect-wrapper.jar
Found JAR file at libs/stoken-wrapper.jar
Found shared library at libs/armeabi/libopenconnect.so
Found shared library at libs/armeabi/libstoken.so
Found shared library at libs/mips/libopenconnect.so
Found shared library at libs/mips/libstoken.so
Found shared library at libs/x86/libopenconnect.so
Found shared library at libs/x86/libstoken.so
 2015-09-13 22:17:37 -07:00
Daniel Martí
41443edd55 Bump build-tools 2015-09-05 23:02:03 -04:00
Daniel Martí
f23011f408 CI: set up all SDK components in PATH 2015-09-01 21:12:41 -07:00
Daniel Martí
c633b21f6c CI: Split up package installing 
Advantages:
* Easier to tell why we need each package
* apt-get install output is less scary/huge
* CI job is split in more, smaller steps easier to debug
 2015-09-01 15:23:06 -07:00
Hans-Christoph Steiner
9855d4089c install build dependencies for Pillow 
When Pillow is installed with pip, it needs to compile the C code
against a few libraries, like libjpeg.  This also adds python-git
as a dep.
 2015-09-01 23:48:28 +02:00
Daniel Martí
88dcc9d272 CI: add missing python-virtualenv package 2015-09-01 11:34:04 -07:00
Hans-Christoph Steiner
242e9d2fb9 gitlab-ci: install all android packages at once 
the `android` utility is pretty stupid, it doesn't really cache the
package index info.  So each time it is run, it tries to fetch the
indexes from the network.  This combines all android package installs
to a single command to make things run quicker.
 2015-09-01 19:57:40 +02:00
Hans-Christoph Steiner
e6c0be8898 move tests into common script for jenkins and gitlab-ci 2015-09-01 19:52:37 +02:00
Daniel Martí
4a4bb3e7a5 CI: Add missing yaml dependency 2015-09-01 10:42:16 -07:00
Daniel Martí
79749fe8b4 GitLab CI: Java is required to set up the SDK 2015-08-28 16:34:42 -07:00
Daniel Martí
9e99270680 Gitlab CI: install SDK 2015-08-28 16:11:54 -07:00
Daniel Martí
1eb83dcfea Gitlab CI: run tests too 2015-08-28 16:08:01 -07:00
Daniel Martí
4e14d9ceb4 Gitlab CI: Rename static analyzers task to "check" 2015-08-27 16:07:45 -07:00
Daniel Martí
12aea08580 First try at Gitlab CI integration 2015-08-27 16:00:02 -07:00