1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-09 17:00:12 +01:00
Commit Graph

4265 Commits

Author SHA1 Message Date
Michael Pöhn
b01d48a4fd makebuildserver deal with apt cache lock 2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
acf25a3999 build server use up instead or resume; logging 2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
c749c68486 test script for vmtools 2017-05-23 20:06:06 +02:00
Michael Pöhn
3187d2cbcf revised build server creation 2017-05-23 20:06:06 +02:00
Michael Pöhn
8abd3f1cbc auto-lookup vm provider based on available executables; more fault tolerant vagrant package 2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
bba6b8ab0a fixed reading libvirt box image size 2017-05-23 20:06:06 +02:00
Michael Pöhn
1bd51966b8 vmtools debug logging for check_output calls 2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
5dbcd0e9bd added box handling to vmtools 2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
01b6473823 refactored kvm_package to vmtools 2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
5580a685db added makebuildserver option for keeping vagrant box
This is very useful for debugging this process, and also for people
who might want to keep a working copy of the box.
2017-05-23 20:06:06 +02:00
Michael Pöhn
d180aa2658 fix virsh destroy parameters 2017-05-23 20:06:06 +02:00
Michael Pöhn
4347c10d9d use configured vm provider in when calling destroy in makebuildserver 2017-05-23 20:06:06 +02:00
Michael Pöhn
a414aa00ff use overhauled mv destroy code in build.py 2017-05-23 20:06:06 +02:00
Michael Pöhn
92fada803e overhauled and moved destroying builder vm to vmtools.py 2017-05-23 20:04:08 +02:00
Michael Pöhn
fb03e17849 fdroid build: added sleep after destroy/undefine 2017-05-23 20:04:08 +02:00
Michael Pöhn
718d01dea2 makebuildserver added sleep after destroy/undefine 2017-05-23 20:04:08 +02:00
Michael Pöhn
3c4b1dec84 makebuildserver more robust codepath for vagrant destroy 2017-05-23 20:04:08 +02:00
Michael Pöhn
7e8f7c65bc makebuildserver added failsafe when destroy vagrant vm 2017-05-23 20:04:08 +02:00
Michael Pöhn
70a827d59a makebuildserver use virsh instead of libvirt for forcing domain off 2017-05-23 20:04:08 +02:00
Michael Pöhn
8e5446068b makebuildserver debug logging when initial provisioning fails 2017-05-23 20:04:08 +02:00
Michael Pöhn
16b609215e overhauled makebuildserver libvirt vm cleanup 2017-05-23 20:04:08 +02:00
Michael Pöhn
440509cf8a makebuildserver debugging log details 2017-05-23 20:04:08 +02:00
Michael Pöhn
2aa5523011 makebuildserver prune gloabal vagrant status when purging broken VMs 2017-05-23 20:04:08 +02:00
Michael Pöhn
8cfd3ca770 delete .vagrant dir when cleaning up buildserver vm 2017-05-23 20:04:08 +02:00
Michael Pöhn
437ff7c3f0 jenkins makebuildserver fail if vagrant box was not created 2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
2993674aa8 calculate correct size for buildserver-box in makebuildserver
https://gitlab.com/fdroid/fdroidserver/issues/238#note_24000153
"Our hard-coded image size meta-data (1000) is for some interpreted as less
than the size of the box-image by my kvm setup. This makes grub/initrd
refuse to boot. So I've changed the metadata size to 9999 which resulted in
an actually booting vm. I can log in on the builder-vm via virt-manager
and virsh.
2017-05-23 20:04:08 +02:00
Michael Pöhn
413c3836d5 auto-reset broken builder vm 2017-05-23 20:04:08 +02:00
Michael Pöhn
ce3c959ce5 scan vm provider again after re-creating builder vm 2017-05-23 20:04:08 +02:00
Michael Pöhn
c54e0565d7 added some debug listings to jenkins makebuildserver script 2017-05-23 20:04:08 +02:00
Michael Pöhn
34cddd3be8 delete associated libvirt domain/image when destroying builder vm 2017-05-23 20:04:08 +02:00
Michael Pöhn
fad98eeb7f build: fixed kvm snapshot support; makebuildserver: setup kvm ssh credentials 2017-05-23 20:04:08 +02:00
Michael Pöhn
fa2d44ee94 added libvirt vm-provider support to build.py 2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
69e4b91d3f makebuildserver: package up KVM VM as a vagrant box
`vagrant package` does not work with KVM, so we have to hack together our
own until someone implements it (suppose we should do it). This is a hacked
up version based on:
d7d440ea8f/tools/create_box.sh

#238
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
988ac21e7f buildserver: make --clean destroy reliably
This prevents v.destroy() from running if Vagrantfile.yaml does not exist,
since that is required for vagrant to run: is the core config including the
name of the box, etc.  Otherwise, it would exit with an error.

This also does complete cleanup when using libvirt.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
299ed82a88 buildserver: consolidate boxfile export code into one block
This is just for clarity, and moving more code into the main() function.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
daade7656a buildserver: add copy_caches_from_host config option
For people using slow, expensive, and/or flaky internet, liberal use of
caching can make a huge difference.  The restricted environment of the
gpjenkins box has been a good test environment for this (Tor-only,
whitelist of allowed IPs to visit, home internet connection).
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
4cde71552f buildserver: run_via_vagrant_ssh() to run cmds via python-vagrant
This moves the last vagrant call in a subprocess.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
6464ec55b7 buildserver: move code into main() method to always stop thread
By running the whole program in a main() function, it can be wrapped in
try/finally in order to stop the background display thread.  This is also
done in ./fdroid, its standard practice for Python CLI utilities.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
675500ad88 buildserver: display verbose logging in a background tail 2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
4b03c3d42d buildserver: replace custom code with python-vagrant
I ran into some annoying issues with UTF-8 output in the vagrant logs, and
it was hard to solve.  So I switched to using python-vagrant, which handles
it all for us.  Its been around since 2012, has a number of contributors,
and is still actively maintained, so it seems like a good bet. I also
packaged it for Debian, including a backport in jessie-backports.

On Debian/jessie, do `apt-get install python3-vagrant/jessie-backports`
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
7ef0d5dfd8 include class like UNIX tail -f for displaying logs
This allows fdroidserver to easily log activity while displaying it at the
same time.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
c62e3fd0cc move bulk of reproducible_fdroid_build_apps.sh to jenkins-build
This lets us quickly and frequently test things.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
1710428de1 Merge branch 'exceptions' into 'master'
Replace sys.exit() in non-main functions by exceptions

See merge request !273
2017-05-22 20:50:54 +00:00
Torsten Grote
1fcd8e63a3
Replace sys.exit() in non-main functions by exceptions
Also move all exceptions into one module
2017-05-22 16:51:11 -03:00
Michael Pöhn
a8420817cb fix calling vagrant global-status 2017-05-22 17:40:19 +02:00
Michael Pöhn
592a292847 use stable version of priv ext instead of latest when testing in makebuildserver 2017-05-22 16:04:30 +02:00
Hans-Christoph Steiner
84dc415837 jenkins-build-makebuildserver: retry git fetch until it works
It seems that gitlab gives 500 errors a bit too frequently, so keep
retrying the `git pull` until it works so that there isn't a error email
sent out over failed pulls.
2017-05-22 16:04:30 +02:00
Hans-Christoph Steiner
780b0e9502 Merge branch 'indexing-fixes' into 'master'
indexing fixes

Closes #303

See merge request !270
2017-05-22 09:32:39 +00:00
Hans-Christoph Steiner
0345d2c79f update: find aapt when it is not in the PATH
This is some very messy logic built up since 2010.  This will all go away
once we have a python3 version of androguard available.

The removed imports and `dir(APK)` is to silence pyflakes

closes #303
2017-05-18 17:21:05 +02:00
Hans-Christoph Steiner
c348186ad6 update: fix fastlane scraping, it uses a subdir called 'images'
https://commons.wikimedia.org/wiki/File:GetChromium_FeatureGraphic_1024x500.png
2017-05-18 16:13:17 +02:00