buildserver running in qemu/kvm to support KVM on KVM
jenkins.debian.net runs in QEMU/KVM instances, so in order to run the F-Droid buildserver there, it needs to work inside of a KVM guest. The best way I found to do that is to create QEMU/KVM instances via KVM's "nested" virtualization support. This collection of commits enables using QEMU/KVM as the buildserver when `./makebuildserver` detects that it is running inside of KVM. Otherwise, the old behavior is default: running in VirtualBox.
I have run these tests inside of ubuntu/16.04 on bare metal, which uses VirtualBox, and ubuntu/16.04 KVM guest, which uses QEMU/KVM. It'll also run on the Guardian Project jenkins box, which is Debian/jessie.
@mvdan @CiaranG @krt
See merge request !168
I think the `android update sdk` tool is installing all of the m2 files
that are present in the temp cache, and it seems to do it in order of
newest to oldest. Well done, and I thought that tool couldn't get any
worse. So only include the latest version of android_m2repository*.zip in
the temp cache.
For running in QEMU/KVM guests like on jenkins.debian.net, this sets up the
whole process automatically. This only really covers the case where this
is running in a KVM guest, and the original case of running VirtualBox on
bare metal. It could be extended to cover more cases if someone wanted to.
The Android SDK by default includes some Google repositories of their
proprietary SDKs. We of course do not want that stuff ever. We also do
not need the emulator images since this process does not currently install
or run an emulator.
make metadata exceptions optional based on CLI flag
In many cases, there are times where metadata errors need to be ignored, or
at least not stop the command from running. For example, there will
inevitably be new metadata fields added, in which case a packaged version
of fdroidserver will throw errors on each one. This adds a standard -W
flag to customize the response: ignore, default, or error.
* by default, the errors are still errors
* `fdroid readmeta -W` will just print errors
* `fdroid readmeta -Wignore` will not even print errors
https://gitlab.com/fdroid/fdroidserver/issues/150
See merge request !164
buildserver fixes pre-KVM
This is a collection of little fixes to the buildserver setup that I did while implementing the QEMU+KVM support. Since I was running the whole process repeatedly, I had opportunity to find and fix little things like this.
See merge request !165
This is the last thing using Chef, which adds a lot of time to the time it
takes to fully provision the buildserver. This slows down development on
the things we are actually using, like running all builds on
jenkins.debian.net.
#210#165
This does not have the careful result rechecking that chef has, when it
installs each package in the list one at a time. So to help with failures
caused by a package failing to download, first try downloading all the
package, then run the install. The install pass will try to download any
missing packages.
Really, this should use ansible or perhaps chef again since those include
lots of tricks around this stuff.
This forces the release channel and version of chef-solo to install on the
guest VM. I was getting really massive, odd stacktraces without specifying
this, and chef is only used for Kivy now anyway.
The technique where /var/cache/apt is mounted as a shared folder conflicts
with vagrant-cachier's workings. Therefore, ignore vagrant-cachier if the
user selects ./makebuildserver's custom apt cache. The shared folder way
has the advantage for CI builds of storing the cache outside of
VAGRANT_HOME, which is set to be in the git project. That gets wiped by
`git clean -fdx` on each CI build.
In many cases, there are times where metadata errors need to be ignored, or
at least not stop the command from running. For example, there will
inevitably be new metadata fields added, in which case a packaged version
of fdroidserver will throw errors on each one. This adds a standard -W
flag to customize the response: ignore, default, or error.
* by default, the errors are still errors
* `fdroid readmeta -W` will just print errors
* `fdroid readmeta -Wignore` will not even print errors
https://gitlab.com/fdroid/fdroidserver/issues/150
urllib.parse.urljoin() will strip off the last path segment before joining
if that last path segment does not end with /. That's a "feature" to make
it easy to replace file names. Here it was stripping off the essential
'fdroid' segment, making URLs like:
https://foo.com/repo
when they should be
https://foo.com/fdroid/repo
