1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-14 02:50:12 +01:00
Commit Graph

4361 Commits

Author SHA1 Message Date
Michael Pöhn
8abd3f1cbc auto-lookup vm provider based on available executables; more fault tolerant vagrant package 2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
bba6b8ab0a fixed reading libvirt box image size 2017-05-23 20:06:06 +02:00
Michael Pöhn
1bd51966b8 vmtools debug logging for check_output calls 2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
5dbcd0e9bd added box handling to vmtools 2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
01b6473823 refactored kvm_package to vmtools 2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
5580a685db added makebuildserver option for keeping vagrant box
This is very useful for debugging this process, and also for people
who might want to keep a working copy of the box.
2017-05-23 20:06:06 +02:00
Michael Pöhn
d180aa2658 fix virsh destroy parameters 2017-05-23 20:06:06 +02:00
Michael Pöhn
4347c10d9d use configured vm provider in when calling destroy in makebuildserver 2017-05-23 20:06:06 +02:00
Michael Pöhn
a414aa00ff use overhauled mv destroy code in build.py 2017-05-23 20:06:06 +02:00
Michael Pöhn
92fada803e overhauled and moved destroying builder vm to vmtools.py 2017-05-23 20:04:08 +02:00
Michael Pöhn
fb03e17849 fdroid build: added sleep after destroy/undefine 2017-05-23 20:04:08 +02:00
Michael Pöhn
718d01dea2 makebuildserver added sleep after destroy/undefine 2017-05-23 20:04:08 +02:00
Michael Pöhn
3c4b1dec84 makebuildserver more robust codepath for vagrant destroy 2017-05-23 20:04:08 +02:00
Michael Pöhn
7e8f7c65bc makebuildserver added failsafe when destroy vagrant vm 2017-05-23 20:04:08 +02:00
Michael Pöhn
70a827d59a makebuildserver use virsh instead of libvirt for forcing domain off 2017-05-23 20:04:08 +02:00
Michael Pöhn
8e5446068b makebuildserver debug logging when initial provisioning fails 2017-05-23 20:04:08 +02:00
Michael Pöhn
16b609215e overhauled makebuildserver libvirt vm cleanup 2017-05-23 20:04:08 +02:00
Michael Pöhn
440509cf8a makebuildserver debugging log details 2017-05-23 20:04:08 +02:00
Michael Pöhn
2aa5523011 makebuildserver prune gloabal vagrant status when purging broken VMs 2017-05-23 20:04:08 +02:00
Michael Pöhn
8cfd3ca770 delete .vagrant dir when cleaning up buildserver vm 2017-05-23 20:04:08 +02:00
Michael Pöhn
437ff7c3f0 jenkins makebuildserver fail if vagrant box was not created 2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
2993674aa8 calculate correct size for buildserver-box in makebuildserver
https://gitlab.com/fdroid/fdroidserver/issues/238#note_24000153
"Our hard-coded image size meta-data (1000) is for some interpreted as less
than the size of the box-image by my kvm setup. This makes grub/initrd
refuse to boot. So I've changed the metadata size to 9999 which resulted in
an actually booting vm. I can log in on the builder-vm via virt-manager
and virsh.
2017-05-23 20:04:08 +02:00
Michael Pöhn
413c3836d5 auto-reset broken builder vm 2017-05-23 20:04:08 +02:00
Michael Pöhn
ce3c959ce5 scan vm provider again after re-creating builder vm 2017-05-23 20:04:08 +02:00
Michael Pöhn
c54e0565d7 added some debug listings to jenkins makebuildserver script 2017-05-23 20:04:08 +02:00
Michael Pöhn
34cddd3be8 delete associated libvirt domain/image when destroying builder vm 2017-05-23 20:04:08 +02:00
Michael Pöhn
fad98eeb7f build: fixed kvm snapshot support; makebuildserver: setup kvm ssh credentials 2017-05-23 20:04:08 +02:00
Michael Pöhn
fa2d44ee94 added libvirt vm-provider support to build.py 2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
69e4b91d3f makebuildserver: package up KVM VM as a vagrant box
`vagrant package` does not work with KVM, so we have to hack together our
own until someone implements it (suppose we should do it). This is a hacked
up version based on:
d7d440ea8f/tools/create_box.sh

#238
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
988ac21e7f buildserver: make --clean destroy reliably
This prevents v.destroy() from running if Vagrantfile.yaml does not exist,
since that is required for vagrant to run: is the core config including the
name of the box, etc.  Otherwise, it would exit with an error.

This also does complete cleanup when using libvirt.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
299ed82a88 buildserver: consolidate boxfile export code into one block
This is just for clarity, and moving more code into the main() function.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
daade7656a buildserver: add copy_caches_from_host config option
For people using slow, expensive, and/or flaky internet, liberal use of
caching can make a huge difference.  The restricted environment of the
gpjenkins box has been a good test environment for this (Tor-only,
whitelist of allowed IPs to visit, home internet connection).
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
4cde71552f buildserver: run_via_vagrant_ssh() to run cmds via python-vagrant
This moves the last vagrant call in a subprocess.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
6464ec55b7 buildserver: move code into main() method to always stop thread
By running the whole program in a main() function, it can be wrapped in
try/finally in order to stop the background display thread.  This is also
done in ./fdroid, its standard practice for Python CLI utilities.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
675500ad88 buildserver: display verbose logging in a background tail 2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
4b03c3d42d buildserver: replace custom code with python-vagrant
I ran into some annoying issues with UTF-8 output in the vagrant logs, and
it was hard to solve.  So I switched to using python-vagrant, which handles
it all for us.  Its been around since 2012, has a number of contributors,
and is still actively maintained, so it seems like a good bet. I also
packaged it for Debian, including a backport in jessie-backports.

On Debian/jessie, do `apt-get install python3-vagrant/jessie-backports`
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
7ef0d5dfd8 include class like UNIX tail -f for displaying logs
This allows fdroidserver to easily log activity while displaying it at the
same time.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
c62e3fd0cc move bulk of reproducible_fdroid_build_apps.sh to jenkins-build
This lets us quickly and frequently test things.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
1710428de1 Merge branch 'exceptions' into 'master'
Replace sys.exit() in non-main functions by exceptions

See merge request !273
2017-05-22 20:50:54 +00:00
Torsten Grote
1fcd8e63a3
Replace sys.exit() in non-main functions by exceptions
Also move all exceptions into one module
2017-05-22 16:51:11 -03:00
Michael Pöhn
a8420817cb fix calling vagrant global-status 2017-05-22 17:40:19 +02:00
Michael Pöhn
592a292847 use stable version of priv ext instead of latest when testing in makebuildserver 2017-05-22 16:04:30 +02:00
Hans-Christoph Steiner
84dc415837 jenkins-build-makebuildserver: retry git fetch until it works
It seems that gitlab gives 500 errors a bit too frequently, so keep
retrying the `git pull` until it works so that there isn't a error email
sent out over failed pulls.
2017-05-22 16:04:30 +02:00
Hans-Christoph Steiner
780b0e9502 Merge branch 'indexing-fixes' into 'master'
indexing fixes

Closes #303

See merge request !270
2017-05-22 09:32:39 +00:00
Hans-Christoph Steiner
0345d2c79f update: find aapt when it is not in the PATH
This is some very messy logic built up since 2010.  This will all go away
once we have a python3 version of androguard available.

The removed imports and `dir(APK)` is to silence pyflakes

closes #303
2017-05-18 17:21:05 +02:00
Hans-Christoph Steiner
c348186ad6 update: fix fastlane scraping, it uses a subdir called 'images'
https://commons.wikimedia.org/wiki/File:GetChromium_FeatureGraphic_1024x500.png
2017-05-18 16:13:17 +02:00
Hans-Christoph Steiner
77a65be9db update: skip packages without metadata in index V1
If a package does not have a matching "app" metadata file, then it should
not be added to the index. index V0 already does this properly.
2017-05-18 16:13:17 +02:00
Hans-Christoph Steiner
a17b95a860 update: improved logging
the aapt warnings were very verbose, and the other now includes the
file name type for screenshots, rather than the filename twice.
2017-05-18 16:13:17 +02:00
Hans-Christoph Steiner
5d705452f5 update: allow repo files to use _ in the file names
_ is a valid character for Java package names, so it should also work in
the repo file naming scheme. This makes it so it only splits the file
name based on the last _.
2017-05-18 16:13:17 +02:00
Hans-Christoph Steiner
84bb41a91f metadata: switch from deprecated cgi.escape to html.escape
cgi.escape is deprecated in Python 3.x and has security issues:
https://bugs.python.org/issue26398

html.escape() differs from cgi.escape() by its defaults to quote=True:

 s = html.escape( """& < " ' >""" )   # s = '&amp; &lt; &quot; &#x27; &gt;'
2017-05-18 16:13:17 +02:00