buildserver fixes on the way to KVM
Here are a couple of relatively basic fixes I found while working on the KVM support (merge request coming soon).
See merge request !174
No point in running any other code if the script is just going to bail
out with an error. This assumes that ./makebuildserver is only ever
run from a git clone of fdroidserver.git.
Reproducible via `fdroid checkupdates --auto subreddit.android.appstore`
at fdroiddata HEAD (e76449ab).
WARNING: ...subreddit.android.appstore : Couldn't find package ID
CRITICAL: Unknown exception found!
Traceback (most recent call last):
File "/home/mvdan/.bin/fdroid", line 147, in <module>
main()
File "/home/mvdan/.bin/fdroid", line 124, in main
mod.main()
File "/home/mvdan/git/fsr/fdroidserver/checkupdates.py", line 571, in main
checkupdates_app(app)
File "/home/mvdan/git/fsr/fdroidserver/checkupdates.py", line 469, in checkupdates_app
if int(build.vercode) >= int(app.CurrentVersionCode):
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
VirtualBox runs as the same user as `fdroid`, so the cache does not need to
be accessible by the world. On libvirt, libvirtd runs the VMs as its own
user, so in that case, the cache dirs must have permissions to let that
user access them.
buildserver running in qemu/kvm to support KVM on KVM
jenkins.debian.net runs in QEMU/KVM instances, so in order to run the F-Droid buildserver there, it needs to work inside of a KVM guest. The best way I found to do that is to create QEMU/KVM instances via KVM's "nested" virtualization support. This collection of commits enables using QEMU/KVM as the buildserver when `./makebuildserver` detects that it is running inside of KVM. Otherwise, the old behavior is default: running in VirtualBox.
I have run these tests inside of ubuntu/16.04 on bare metal, which uses VirtualBox, and ubuntu/16.04 KVM guest, which uses QEMU/KVM. It'll also run on the Guardian Project jenkins box, which is Debian/jessie.
@mvdan @CiaranG @krt
See merge request !168
I think the `android update sdk` tool is installing all of the m2 files
that are present in the temp cache, and it seems to do it in order of
newest to oldest. Well done, and I thought that tool couldn't get any
worse. So only include the latest version of android_m2repository*.zip in
the temp cache.
For running in QEMU/KVM guests like on jenkins.debian.net, this sets up the
whole process automatically. This only really covers the case where this
is running in a KVM guest, and the original case of running VirtualBox on
bare metal. It could be extended to cover more cases if someone wanted to.
The Android SDK by default includes some Google repositories of their
proprietary SDKs. We of course do not want that stuff ever. We also do
not need the emulator images since this process does not currently install
or run an emulator.
make metadata exceptions optional based on CLI flag
In many cases, there are times where metadata errors need to be ignored, or
at least not stop the command from running. For example, there will
inevitably be new metadata fields added, in which case a packaged version
of fdroidserver will throw errors on each one. This adds a standard -W
flag to customize the response: ignore, default, or error.
* by default, the errors are still errors
* `fdroid readmeta -W` will just print errors
* `fdroid readmeta -Wignore` will not even print errors
https://gitlab.com/fdroid/fdroidserver/issues/150
See merge request !164
buildserver fixes pre-KVM
This is a collection of little fixes to the buildserver setup that I did while implementing the QEMU+KVM support. Since I was running the whole process repeatedly, I had opportunity to find and fix little things like this.
See merge request !165
This is the last thing using Chef, which adds a lot of time to the time it
takes to fully provision the buildserver. This slows down development on
the things we are actually using, like running all builds on
jenkins.debian.net.
#210#165
This does not have the careful result rechecking that chef has, when it
installs each package in the list one at a time. So to help with failures
caused by a package failing to download, first try downloading all the
package, then run the install. The install pass will try to download any
missing packages.
Really, this should use ansible or perhaps chef again since those include
lots of tricks around this stuff.
This forces the release channel and version of chef-solo to install on the
guest VM. I was getting really massive, odd stacktraces without specifying
this, and chef is only used for Kivy now anyway.
The technique where /var/cache/apt is mounted as a shared folder conflicts
with vagrant-cachier's workings. Therefore, ignore vagrant-cachier if the
user selects ./makebuildserver's custom apt cache. The shared folder way
has the advantage for CI builds of storing the cache outside of
VAGRANT_HOME, which is set to be in the git project. That gets wiped by
`git clean -fdx` on each CI build.
In many cases, there are times where metadata errors need to be ignored, or
at least not stop the command from running. For example, there will
inevitably be new metadata fields added, in which case a packaged version
of fdroidserver will throw errors on each one. This adds a standard -W
flag to customize the response: ignore, default, or error.
* by default, the errors are still errors
* `fdroid readmeta -W` will just print errors
* `fdroid readmeta -Wignore` will not even print errors
https://gitlab.com/fdroid/fdroidserver/issues/150
