1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-05 06:50:10 +01:00
Commit Graph

4886 Commits

Author SHA1 Message Date
Hans-Christoph Steiner
bb758d3f00 build: apt-get purge sudo after using it for sudo= build field
Once `sudo` has been used to execute the commands in sudo=, then it should
be removed from the build VM.  That prevents any other part of the build
from using sudo.  That means that all commands run with `sudo` must be
committed to fdroiddata.git, making them very visible.

closes #424
2017-12-11 22:11:16 +01:00
Hans-Christoph Steiner
4af8f196e3 metadata: error if .fdroid.txt exists, it is unsupported
Using .txt format has bugs when used embedded in an app's git repo, so
tell the user to use YAML or JSON.

#364
2017-12-11 22:11:16 +01:00
Hans-Christoph Steiner
1d83d4396c makebuildserver: quiet rsync for copy_caches_from_host 2017-12-11 22:11:16 +01:00
Hans-Christoph Steiner
4835d7c20c Merge branch 'rational-jarsigner-logging' into 'master'
handle jarsigner/apksigner output cleanly for rational logging

Closes #405

See merge request fdroid/fdroidserver!404
2017-12-11 20:27:06 +00:00
tobiasKaminsky
03f301470e
regex only for flavor blocks: flavor { ... } and nothing else 2017-12-11 14:29:32 +01:00
Hans-Christoph Steiner
6640f276b0 Revert "makebuildserver: update SHA-256 for platform-27_r01.zip"
Looks like Google switched back to the old binary, which I guess is
good news?
https://issuetracker.google.com/issues/70292819

This reverts commit 956660085a.

!401
2017-12-08 09:44:10 +01:00
Hans-Christoph Steiner
5ac943a3f2 Merge branch 'yml_completion' into 'master'
bash completion: use correct yml suffix

See merge request fdroid/fdroidserver!406
2017-12-07 23:10:18 +00:00
Marcus Hoffmann
9270e68fe2 bash completion: use correct yml suffix 2017-12-07 23:29:34 +01:00
Hans-Christoph Steiner
4f43099c88 Merge branch 'more-nightly' into 'master'
More `fdroid nightly` polish

Closes #423

See merge request fdroid/fdroidserver!402
2017-12-07 22:11:40 +00:00
Hans-Christoph Steiner
7b52722d12 nightly: replace / from fingerprint in SSH key filename, fixes #423
The SSH key fingerprint is used in the filename.  The base64 used for SSH
key fingerprints includes /.  Not all keys will end up having a / in them.
For those that do, this will crash since the ssh key filename ends up being
non-existent dirs:

$ fdroid nightly
Importing keystore /home/mhoffmann/.android/debug.keystore to /tmp/.cqswaeo8/.keystore.p12...
MAC verified OK
writing RSA key
CRITICAL: Unknown exception found!
Traceback (most recent call last):
  File "/usr/lib/python3.6/shutil.py", line 544, in move
    os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/.cqswaeo8/.privkey' -> '/tmp/.cqswaeo8/debug_keystore_PZtS/4Tzk4dpzKiX9AAf1GrhAVi9U7UE1aYEHr6evKo_id_rsa'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mhoffmann/projects/oss/fdroidserver/fdroid", line 156, in <module>
    main()
  File "/home/mhoffmann/projects/oss/fdroidserver/fdroid", line 132, in main
    mod.main()
  File "/home/mhoffmann/projects/oss/fdroidserver/fdroidserver/nightly.py", line 284, in main
    privkey = _ssh_key_from_debug_keystore()
  File "/home/mhoffmann/projects/oss/fdroidserver/fdroidserver/nightly.py", line 73, in _ssh_key_from_debug_keystore
    shutil.move(privkey, ssh_private_key_file)
  File "/usr/lib/python3.6/shutil.py", line 558, in move
    copy_function(src, real_dst)
  File "/usr/lib/python3.6/shutil.py", line 257, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.6/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/.cqswaeo8/debug_keystore_PZtS/4Tzk4dpzKiX9AAf1GrhAVi9U7UE1aYEHr6evKo_id_rsa'
2017-12-07 22:39:32 +01:00
Hans-Christoph Steiner
5d56841f8a nightly: prompt user to create a debug.keystore if its not there
#423
2017-12-07 22:39:32 +01:00
Hans-Christoph Steiner
c823d4c4a8 nightly: fix QR icon.png generation 2017-12-07 22:39:32 +01:00
Hans-Christoph Steiner
5d54446efc update: do not replace Name/Summary from template unless blank
`fdroid nightly` needs this change so it can set the Summary using the
template.
2017-12-07 22:39:32 +01:00
Hans-Christoph Steiner
4afe5aefd5 nightly: create app metadata using template of parsed data 2017-12-07 22:39:32 +01:00
Marcus
87018d45e2 Merge branch 'remove-fd-commit' into 'master'
remove fd-commit, no active devs use it, and requires Auto Name/Name

See merge request fdroid/fdroidserver!392
2017-12-07 20:31:09 +00:00
Hans-Christoph Steiner
adc0c23db4 Merge branch 'help_message_fix' into 'master'
metadata: make help for common -W option clearer

See merge request fdroid/fdroidserver!350
2017-12-07 20:03:17 +00:00
Marcus Hoffmann
0f18005104 metadata: make help for metadata -W option clearer
specify possible choices and provide clearer help text
2017-12-07 20:28:27 +01:00
Hans-Christoph Steiner
6228162cbd handle jarsigner/apksigner output cleanly for rational logging
These were both spamming the output with lots of confusing messages, even
when --verbose was not used.  Jarsigner especially has confusing messages,
since it has warnings that do not pertain to APK signatures at all, like
the ones about timestamps and missing Certificate Authority.

closes #405
2017-12-07 17:32:14 +01:00
Hans-Christoph Steiner
7f4d84684e Merge branch 'syntaxfix' into 'master'
correct "usage" output (--help; see #405)

See merge request fdroid/fdroidserver!403
2017-12-07 14:17:25 +00:00
Izzy
0a1fe3dc7e correct "usage" output (--help; see #405) 2017-12-07 14:51:27 +01:00
Hans-Christoph Steiner
0e37a18d83 makebuildserver: fix join() syntax error, it needs a list/tuple
If only there was a way to test this without taking hours to run...

fixes 964ef996a0
2017-12-07 11:52:52 +01:00
relan
8ffed8750c Merge branch 'update-platform-27_r01' into 'master'
makebuildserver: update SHA-256 for platform-27_r01.zip

See merge request fdroid/fdroidserver!401
2017-12-07 06:05:46 +00:00
Hans-Christoph Steiner
6902160e89 remove fd-commit, no active devs use it, and requires Auto Name/Name
fd-commit and checkupdates both require that there are two name fields,
AutoName: and Name:.  This is only used for the commit messages.  Since the
current devs do it manually, we can remove the fd-commit shell script, then
focus on checkupdates when revamping AutoName/Name.

https://botbot.me/freenode/fdroid-dev/msg/82539152
2017-12-06 22:48:08 +01:00
Hans-Christoph Steiner
956660085a makebuildserver: update SHA-256 for platform-27_r01.zip
Someone forgot to call this _r02.zip:
-ro.​build.​version.​incremental=4402310
+ro.​build.​version.​incremental=4458339

https://verification.f-droid.org/build-metadata/platform-27_r01.html

!364
2017-12-06 22:05:09 +01:00
Hans-Christoph Steiner
6fc507da39 Merge branch 'gitlab-ci-testing-xenial-fedora' into 'master'
gitlab CI runs on  Debian/testing,  Ubuntu/xenial,  Fedora

See merge request fdroid/fdroidserver!398
2017-12-06 19:44:58 +00:00
Hans-Christoph Steiner
05abbfbabd gitlab-ci: move sdist test run to new fedora job
A full run of the test suite takes quite a bit of time.  This removes one
of the 3 runs from the main 'tests' job, and puts it into the Fedora job.
That test run is mostly to make sure the setup.py and source tarball are
correctly, so that doesn't affect merge requests very often.

This also tests `pip install --user`, which was not really being tested
before.
2017-12-06 20:20:17 +01:00
Hans-Christoph Steiner
1a77c6af38 init: fix test for aapt when no aapt has been found
Just give a proper error message rather than this stack trace:

Traceback (most recent call last):
  File "/home/hans/code/fdroid/server/fdroid", line 156, in <module>
    main()
  File "/home/hans/code/fdroid/server/fdroid", line 132, in main
    mod.main()
  File "/export/share/code/fdroid/server/fdroidserver/init.py", line 148, in main
    if os.path.isfile(aapt):
  File "/usr/lib/python3.5/genericpath.py", line 30, in isfile
    st = os.stat(path)
2017-12-06 20:20:17 +01:00
Hans-Christoph Steiner
964ef996a0 makebuildserver: make copy_caches_from_host do rsync like fdroid build
This rsync hung because of an SSH unknown key prompt.  Since this is just
the vm host sshing to the vm guest, it is not essential to check the host
keys.
2017-12-06 20:20:17 +01:00
Hans-Christoph Steiner
a0a68c7a13 setup requires Babel aka python3-babel to compile translations
https://forum.f-droid.org/t/f-droid-server-building-error/1670
2017-12-06 20:20:17 +01:00
Hans-Christoph Steiner
dda9c8b774 gitlab-ci: add pip install test on Arch Linux 2017-12-06 20:20:17 +01:00
Hans-Christoph Steiner
1b1475c982 gitlab-ci: add test runs on Ubuntu, Debian/testing, Fedora 2017-12-06 12:30:47 +01:00
Marcus Hoffmann
bfe2c00834 common.testCase: fix find_sdk_tools when aapt is installed in /usr/bin
The testlogic was broken when having both aapt in /usr/bin and also as
part of the android sdk.
2017-12-06 12:30:47 +01:00
Hans-Christoph Steiner
67e6cbe793 hooks/pre-commit: make ruby and dash tests optional
These are only used for checking syntax in buildserver/Vagrantfile.
Not requiring ruby makes doing CI tests on lots of distros easier
and faster. dash is an 'essential' package on Debian derivs, so
those tests will always be run somewhere.
2017-12-06 12:30:47 +01:00
Hans-Christoph Steiner
a2978a5526 common: aapt 24.0.0 (v0.2-2964546) is now required
Without a recent aapt, the <uses-permission-sdk-23> tag will not be found.
2017-12-06 12:30:47 +01:00
Hans-Christoph Steiner
657b64f6ed Merge branch 'lint-fixes' into 'master'
Lint fixes, plus changing the standard link format in descriptions

See merge request fdroid/fdroidserver!397
2017-12-06 09:41:39 +00:00
Hans-Christoph Steiner
8e1c39f791 Merge branch 'rsync_improvements' into 'master'
Rsync improvements

See merge request fdroid/fdroidserver!400
2017-12-06 08:55:19 +00:00
Hans-Christoph Steiner
b31239803a lint: greatly expand the list of link shorteners to ban
Since we are now getting credit for fighting trackers, might as well step
up the fight!

gleaned from these sources:
* https://bit.do/list-of-url-shorteners.php
* https://www.hashtags.org/featured/list-of-url-shorteners/
* http://l-lists.com/en/lists/gvaoif.html
2017-12-06 09:54:25 +01:00
Hans-Christoph Steiner
05616b33a7 lint: enforce HTTPS and shortener ban in descriptions as well 2017-12-06 09:54:25 +01:00
Hans-Christoph Steiner
42a9833536 lint: switch links to plain URLS rather than mediawiki syntax
fdroidclient#1000
2017-12-06 09:54:25 +01:00
Hans-Christoph Steiner
8588b89eff lint: add more VCS HTTPS checks
I manually checked that these work with HTTPS. fdroiddata!2710 should fix
all of these issues.
2017-12-06 09:54:25 +01:00
Marcus Hoffmann
1bfba12124 build: write out full rsync options
Also put target host:dir on one line to make it more readable
2017-12-05 21:52:22 +01:00
Marcus Hoffmann
e12e1b6a5c build: better logging output on rsync failures
Save rsync error output and combine that with the command invocation
into an FDroidException which can be logged to the wiki.

This additionally sets -q for rsync to only print errors.
2017-12-05 21:52:11 +01:00
Torsten Grote
2bb1445cd6 Merge branch 'nightly-fixes' into 'master'
more `fdroid nightly` polishing

See merge request fdroid/fdroidserver!399
2017-12-05 17:42:57 +00:00
Hans-Christoph Steiner
c33a71a945 fix hg pull, was stupid mistake in 7bba20c662
fdroid/fdroidserver!396
2017-12-05 16:55:58 +01:00
Hans-Christoph Steiner
bb643eddcf jenkins-setup-build-environment: delete libvirt images before test run
profitbricks-build7-amd64 was running out of disk space when running this
job...
2017-12-05 12:31:13 +01:00
Hans-Christoph Steiner
4561ea59a6 nightly: use shutil.move() only so all ops work across filesystems
https://gitlab.com/fdroid/fdroidserver/merge_requests/377#note_49998712
2017-12-05 09:13:19 +01:00
Hans-Christoph Steiner
2983c35361 shutil.move() in apk_strip_signature() to work across filesystems
os.rename() only works if source and destination are on the same file
system, shutil.move() works across file systems.

OSError: [Errno 18] Invalid cross-device link: '/builds/eighthave/fdroidclient/app/build/outputs/apk/app-debug.apk' -> '/tmp/tmp966vh75f/tmp.apk'
2017-12-04 22:52:41 +01:00
Hans-Christoph Steiner
bf913703c5 nightly: only use read_config to load final, generated config.py
This needs to use the config loading routine to find Java `keytool`, but
since it doesn't need to fully load the config, isolate that usage in the
function.  Then read_config() is only ever called once, as is it meant to
be used, once the config.py is generated.

Using `from . import common; common.config = foo` will not always work,
due to some oddities to how the `from` imports work. So the full module
has to be imported in order to make sure its always properly set.
2017-12-04 22:52:41 +01:00
Hans-Christoph Steiner
8a61b0b945 nightly: resign APKs with provided debug.keystore
Rather than needing to run a command before and after the build, in order
to first install the debug.keystore, then after to fetch and publish the
APK, this makes `fdroid nightly` just resign the APK with the provided
debug.keystore.  Then `fdroid nightly` can be run as the final step in a CI
build, and still ensure that the APKs are always signed by the provided
debug.keystore.
2017-12-04 22:52:41 +01:00
Hans-Christoph Steiner
1c3a4479ab add common.sign_apk() for nighly as test for using in publish
Since the MD5 migration was quite a bit of work, it makes sense to start
on moving away from SHA1 as much as possible while it is easy to do. SHA256
will only work in APK signatures on android-18 (4.3) or newer.  So if an
APK has a minSdkVersion of 18 or newer, then sign with SHA256.

https://issuetracker.google.com/issues/36956587
https://android-review.googlesource.com/c/platform/libcore/+/44491
2017-12-04 22:52:41 +01:00