Hans-Christoph Steiner
c749c68486
test script for vmtools
2017-05-23 20:06:06 +02:00
Michael Pöhn
3187d2cbcf
revised build server creation
2017-05-23 20:06:06 +02:00
Michael Pöhn
8abd3f1cbc
auto-lookup vm provider based on available executables; more fault tolerant vagrant package
2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
bba6b8ab0a
fixed reading libvirt box image size
2017-05-23 20:06:06 +02:00
Michael Pöhn
1bd51966b8
vmtools debug logging for check_output calls
2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
5dbcd0e9bd
added box handling to vmtools
2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
01b6473823
refactored kvm_package to vmtools
2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
5580a685db
added makebuildserver option for keeping vagrant box
...
This is very useful for debugging this process, and also for people
who might want to keep a working copy of the box.
2017-05-23 20:06:06 +02:00
Michael Pöhn
d180aa2658
fix virsh destroy parameters
2017-05-23 20:06:06 +02:00
Michael Pöhn
4347c10d9d
use configured vm provider in when calling destroy in makebuildserver
2017-05-23 20:06:06 +02:00
Michael Pöhn
a414aa00ff
use overhauled mv destroy code in build.py
2017-05-23 20:06:06 +02:00
Michael Pöhn
92fada803e
overhauled and moved destroying builder vm to vmtools.py
2017-05-23 20:04:08 +02:00
Michael Pöhn
fb03e17849
fdroid build: added sleep after destroy/undefine
2017-05-23 20:04:08 +02:00
Michael Pöhn
718d01dea2
makebuildserver added sleep after destroy/undefine
2017-05-23 20:04:08 +02:00
Michael Pöhn
3c4b1dec84
makebuildserver more robust codepath for vagrant destroy
2017-05-23 20:04:08 +02:00
Michael Pöhn
7e8f7c65bc
makebuildserver added failsafe when destroy vagrant vm
2017-05-23 20:04:08 +02:00
Michael Pöhn
70a827d59a
makebuildserver use virsh instead of libvirt for forcing domain off
2017-05-23 20:04:08 +02:00
Michael Pöhn
8e5446068b
makebuildserver debug logging when initial provisioning fails
2017-05-23 20:04:08 +02:00
Michael Pöhn
16b609215e
overhauled makebuildserver libvirt vm cleanup
2017-05-23 20:04:08 +02:00
Michael Pöhn
440509cf8a
makebuildserver debugging log details
2017-05-23 20:04:08 +02:00
Michael Pöhn
2aa5523011
makebuildserver prune gloabal vagrant status when purging broken VMs
2017-05-23 20:04:08 +02:00
Michael Pöhn
8cfd3ca770
delete .vagrant dir when cleaning up buildserver vm
2017-05-23 20:04:08 +02:00
Michael Pöhn
437ff7c3f0
jenkins makebuildserver fail if vagrant box was not created
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
2993674aa8
calculate correct size for buildserver-box in makebuildserver
...
https://gitlab.com/fdroid/fdroidserver/issues/238#note_24000153
"Our hard-coded image size meta-data (1000) is for some interpreted as less
than the size of the box-image by my kvm setup. This makes grub/initrd
refuse to boot. So I've changed the metadata size to 9999 which resulted in
an actually booting vm. I can log in on the builder-vm via virt-manager
and virsh.
2017-05-23 20:04:08 +02:00
Michael Pöhn
413c3836d5
auto-reset broken builder vm
2017-05-23 20:04:08 +02:00
Michael Pöhn
ce3c959ce5
scan vm provider again after re-creating builder vm
2017-05-23 20:04:08 +02:00
Michael Pöhn
c54e0565d7
added some debug listings to jenkins makebuildserver script
2017-05-23 20:04:08 +02:00
Michael Pöhn
34cddd3be8
delete associated libvirt domain/image when destroying builder vm
2017-05-23 20:04:08 +02:00
Michael Pöhn
fad98eeb7f
build: fixed kvm snapshot support; makebuildserver: setup kvm ssh credentials
2017-05-23 20:04:08 +02:00
Michael Pöhn
fa2d44ee94
added libvirt vm-provider support to build.py
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
69e4b91d3f
makebuildserver: package up KVM VM as a vagrant box
...
`vagrant package` does not work with KVM, so we have to hack together our
own until someone implements it (suppose we should do it). This is a hacked
up version based on:
d7d440ea8f/tools/create_box.sh#238
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
988ac21e7f
buildserver: make --clean destroy reliably
...
This prevents v.destroy() from running if Vagrantfile.yaml does not exist,
since that is required for vagrant to run: is the core config including the
name of the box, etc. Otherwise, it would exit with an error.
This also does complete cleanup when using libvirt.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
299ed82a88
buildserver: consolidate boxfile export code into one block
...
This is just for clarity, and moving more code into the main() function.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
daade7656a
buildserver: add copy_caches_from_host config option
...
For people using slow, expensive, and/or flaky internet, liberal use of
caching can make a huge difference. The restricted environment of the
gpjenkins box has been a good test environment for this (Tor-only,
whitelist of allowed IPs to visit, home internet connection).
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
4cde71552f
buildserver: run_via_vagrant_ssh() to run cmds via python-vagrant
...
This moves the last vagrant call in a subprocess.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
6464ec55b7
buildserver: move code into main() method to always stop thread
...
By running the whole program in a main() function, it can be wrapped in
try/finally in order to stop the background display thread. This is also
done in ./fdroid, its standard practice for Python CLI utilities.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
675500ad88
buildserver: display verbose logging in a background tail
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
4b03c3d42d
buildserver: replace custom code with python-vagrant
...
I ran into some annoying issues with UTF-8 output in the vagrant logs, and
it was hard to solve. So I switched to using python-vagrant, which handles
it all for us. Its been around since 2012, has a number of contributors,
and is still actively maintained, so it seems like a good bet. I also
packaged it for Debian, including a backport in jessie-backports.
On Debian/jessie, do `apt-get install python3-vagrant/jessie-backports`
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
7ef0d5dfd8
include class like UNIX tail -f for displaying logs
...
This allows fdroidserver to easily log activity while displaying it at the
same time.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
c62e3fd0cc
move bulk of reproducible_fdroid_build_apps.sh to jenkins-build
...
This lets us quickly and frequently test things.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
1710428de1
Merge branch 'exceptions' into 'master'
...
Replace sys.exit() in non-main functions by exceptions
See merge request !273
2017-05-22 20:50:54 +00:00
Torsten Grote
1fcd8e63a3
Replace sys.exit() in non-main functions by exceptions
...
Also move all exceptions into one module
2017-05-22 16:51:11 -03:00
Michael Pöhn
a8420817cb
fix calling vagrant global-status
2017-05-22 17:40:19 +02:00
Michael Pöhn
592a292847
use stable version of priv ext instead of latest when testing in makebuildserver
2017-05-22 16:04:30 +02:00
Hans-Christoph Steiner
84dc415837
jenkins-build-makebuildserver: retry git fetch until it works
...
It seems that gitlab gives 500 errors a bit too frequently, so keep
retrying the `git pull` until it works so that there isn't a error email
sent out over failed pulls.
2017-05-22 16:04:30 +02:00
Hans-Christoph Steiner
780b0e9502
Merge branch 'indexing-fixes' into 'master'
...
indexing fixes
Closes #303
See merge request !270
2017-05-22 09:32:39 +00:00
Hans-Christoph Steiner
0345d2c79f
update: find aapt when it is not in the PATH
...
This is some very messy logic built up since 2010. This will all go away
once we have a python3 version of androguard available.
The removed imports and `dir(APK)` is to silence pyflakes
closes #303
2017-05-18 17:21:05 +02:00
Hans-Christoph Steiner
c348186ad6
update: fix fastlane scraping, it uses a subdir called 'images'
...
https://commons.wikimedia.org/wiki/File:GetChromium_FeatureGraphic_1024x500.png
2017-05-18 16:13:17 +02:00
Hans-Christoph Steiner
77a65be9db
update: skip packages without metadata in index V1
...
If a package does not have a matching "app" metadata file, then it should
not be added to the index. index V0 already does this properly.
2017-05-18 16:13:17 +02:00
Hans-Christoph Steiner
a17b95a860
update: improved logging
...
the aapt warnings were very verbose, and the other now includes the
file name type for screenshots, rather than the filename twice.
2017-05-18 16:13:17 +02:00
