For people using slow, expensive, and/or flaky internet, liberal use of
caching can make a huge difference. The restricted environment of the
gpjenkins box has been a good test environment for this (Tor-only,
whitelist of allowed IPs to visit, home internet connection).
By running the whole program in a main() function, it can be wrapped in
try/finally in order to stop the background display thread. This is also
done in ./fdroid, its standard practice for Python CLI utilities.
I ran into some annoying issues with UTF-8 output in the vagrant logs, and
it was hard to solve. So I switched to using python-vagrant, which handles
it all for us. Its been around since 2012, has a number of contributors,
and is still actively maintained, so it seems like a good bet. I also
packaged it for Debian, including a backport in jessie-backports.
On Debian/jessie, do `apt-get install python3-vagrant/jessie-backports`
No point in running any other code if the script is just going to bail
out with an error. This assumes that ./makebuildserver is only ever
run from a git clone of fdroidserver.git.
VirtualBox runs as the same user as `fdroid`, so the cache does not need to
be accessible by the world. On libvirt, libvirtd runs the VMs as its own
user, so in that case, the cache dirs must have permissions to let that
user access them.
buildserver running in qemu/kvm to support KVM on KVM
jenkins.debian.net runs in QEMU/KVM instances, so in order to run the F-Droid buildserver there, it needs to work inside of a KVM guest. The best way I found to do that is to create QEMU/KVM instances via KVM's "nested" virtualization support. This collection of commits enables using QEMU/KVM as the buildserver when `./makebuildserver` detects that it is running inside of KVM. Otherwise, the old behavior is default: running in VirtualBox.
I have run these tests inside of ubuntu/16.04 on bare metal, which uses VirtualBox, and ubuntu/16.04 KVM guest, which uses QEMU/KVM. It'll also run on the Guardian Project jenkins box, which is Debian/jessie.
@mvdan @CiaranG @krt
See merge request !168
For running in QEMU/KVM guests like on jenkins.debian.net, this sets up the
whole process automatically. This only really covers the case where this
is running in a KVM guest, and the original case of running VirtualBox on
bare metal. It could be extended to cover more cases if someone wanted to.
This is the last thing using Chef, which adds a lot of time to the time it
takes to fully provision the buildserver. This slows down development on
the things we are actually using, like running all builds on
jenkins.debian.net.
#210#165
This reverts commit 82d09560c6.
It doesn't work - the setup scripts are expecting a ".bin" file (which
is apparently a 7z archive), but what's actually got is a ".zip".
Conflicts:
buildserver/provision-android-ndk
Without this, running makebuildserver from a clean master results in the
following:
$ ./makebuildserver
Traceback (most recent call last):
File "./makebuildserver", line 74, in <module>
del(config['__builtins__']) # added by compile/exec
KeyError: '__builtins__'
This is because a clean checkout has no config, thus exec is never
actually ran.
Amusingly, the commit changed more than just this so a simple git revert
is not possible.
The problem is that the zip is replaced with each point release (24.0.1
at the time of writing) and there is no way to get a URL to a single,
non-changing version. Hence any caching or checksums are completely
worthless and will break every few weeks.
Vagrantfile is now committed and not changed between configurations. It is
configed by translating the python config file's dict to a YAML file, which
Vagrantfile now loads and uses. This makes it a lot easier for vagrant
users and python programmers to understand, and hopefully makes it easier
to maintain and test with.
Python can easily output dicts as YAML, and a Vagrantfile is a ruby script,
which can easily read YAML. Going this route means that Vagrantfile can
ultimately be committed to git, and the configuration will happen all via
Python dicts output as YAML. That makes it drastically easier to follow
the code, and to make modifications.
easy_install does not provide any download caching, while pip does. This
also moves the python module installing a shell script that takes python
packages as args. That will allow for future uses like allowing app
metadata to include pip modules that they need.
This makes it so there is only a single `apt-get install` command run,
instead of one command per-package like with the chef script. It also adds
`apt-get upgrade` to make sure that the base box is fully up-to-date.
This is part of the effort to remove moving parts from the whole build
server setup. Why wrap shell scripts in ruby and chef if we can just
directly run a shell script?
`fdroid build` handles setting the NDK env vars since the NDK version can
change depending on the app being build. Unlike ANDROID_HOME, there is no
single global NDK location. The NDK installs are all versioned.
The progress bar updating at each 1k chunk spams the terminal to such an
extent it stops responding to keypresses (at least with a fast
connection). It also seems extremely inefficient to be writing the file
in 1k chunks. This makes it 64k, which is more reasonable but quite
probably still too small.
Prevent build processes from modifying the cache, it is only needed
during provisioning anyway. A malicious build could still use sudo to
change the cache, but this is more to prevent mistaken modifications.
This was not using anything special from chef, so do it in a shell script
instead. This makes the script easier for the python/shell people, and
probably uses less memory, since chef is a memory hog. This might even
make the provision go faster since it uploads the whole script as a file to
the VM, then runs it there. I think chef sends each command via SSH.
`android update sdk --no-ui` is the standard command line tool for
installing the Android SDK. By symlinking into the $ANDROID_HOME/temp dir,
the cached files can still be used. This converts the chef recipe to a
vagrant shell provisioning script since it was all bash anyway.
Some file names no longer officially have a -linux in them, so those were
changed to keep the cache working with the default filename.
bash provides a standard file location for a script to be run when the
shell starts: /etc/profile.d/ This converts the scattered bits of code for
making ~/.bsenv into a single provisioning script to generate
/etc/profile.d/bsenv.sh, which gets automatically executed when bash starts
It will make it a lot easier to manage the cache if we use the original
file names, which often include the file version. This also changes the
download process to be resumable if there is a partial file in the cache,
and switches from calling wget on the command line to using the python libs
'requests' and 'clint' to provide a similar experience. While its not so
important for this particular bit of code to use those libraries, I think
those two will allow us to provide a better user experience throughout the
whole of fdroidserver.
In this case, it is already doing special tricks fetching the file size
from the server before trying to download it. I suppose this code could
instead check if the file exists, and if so, check the hash sum. I think
that would be slower for most people since checking the hash on large files
takes a noticeable about of time, while a HTTP HEAD request is pretty tiny.
Install version 25 now. Also use the smaller tools zip. While at it,
also remove the tools re-install - it's not worth it, as long as we keep
the initial tools zip up to date.
Fixes a couple errors like:
File "./makebuildserver", line 30, in vagrant
out += line
TypeError: Can't convert 'bytes' object to str implicitly
If universal_newlines=False, the default, then Popen will return bytes if
the newlines in the data do not match the system's newlines. Setting it to
true enables auto-conversion, and then guarantees that the data is always
str.
"If universal_newlines is True, the file objects stdin, stdout and stderr
are opened as text streams in universal newlines mode, as described above
in Frequently Used Arguments, otherwise they are opened as binary streams."
https://docs.python.org/3/library/subprocess.html#subprocess.Popen
makebuildserver: support running VirtualBox in a VM
For debian.jenkins.net, our test environment is a kvm instance that does not expose the hardware virtualization instructions. So this auto-detects whether the current machine, virtual or not, supports the hardware virtualization. If not, it uses VirtualBox's software emulator, which should run everywhere, even in a kvm instance.
See merge request !108
This checks the local CPU settings to see if it has the right virtualizing
CPU instructions, and only uses VirtualBox hardware virtualization on
setups where it is available. VMs instances usually do not provide these
instructions. Where the CPU features are not available, it will use VB's
software virtualization.
4 gigs is still a common amount of RAM these days for laptops, if the VM
takes almost all of that, it makes the machine drag to almost a halt. Most
apps build fine in 1gig of RAM, indeed that's the default for most CI
instances, like travis-ci and gitlab-ci.
On slow machines or VMs like the Debian jenkins box, the VM boot timeout
needs to be a lot longer, otherwise vagrant times out before setting up
the VM.
Setting it in the config file was not working, and right now, all of the
options are in the config file and not as command line flags, so remove
--debian-mirror to keep that consistent.
This also provides a config option to override that default. ~/.cache is
a standard location on GNU/Linux machines for cached content. It is also
good to have the cache outside of the git repo in case `git clean -fdx` is
run, which would delete all files in the directory that are not part of the
git repo, including buildserver/cache/
This makes it so that ./makebuildserver will run without any config file,
using the defaults that are embedded in the script itself. This is like
how `fdroid` works.
This host automatically detects which is the closest mirror, then uses that
one. It does so dynamically, so it'll work on machines that move too. Now
that we are pushing more people to run F-Droid build servers, the defaults
should take those use cases into account.
This keeps the numbers of names down to a minimum, and since the config
is placed right next to the script, this keeps tab completion working
nicely when the config file is in place.
The old file name is still supported.
On some setups, using a custom apt mirror is essential, so this adds a
command line flag to override the default one:
http://ftp.uk.debian.org/debian/
For example, someone who runs a local mirror for offline and low bandwidth
situations.
This uses a % rather than a .format() to avoid escaping { and }, which have
meaning when using .format().
config.vm.box_url can be a list/tuple of URLs, which is useful to specific
a locally cached copy. This is needed on slow connections, so that if it
fails, the download of jessie32.box does not have to start from the
beginning of the file again.
Add optional support for vagrant-cachier plugin
Building the basebox is excruciating for people on slow connections. I'm particularly sensitive to this after living in Central America for awhile :)
This won't affect anyone who hasn't installed the plugin. For those who do, it creates a persistent shared folder for each box (ie. testing23.box) and detects directories to cache between VM builds (apt, gems, pip, chef cache, etc.)
(The only downside is that, for those following server setup does who are not aware what vagrant-cachier does, it might be unexpected that artifacts persist between vagrant destroys.)
See merge request !25
Update Vagrantfile and docs to clarify v1.4.3 is ok
Saw in the server docs that we were recommending 1.3.x and saying 1.4.x was broken. I've confirmed that 1.4.x works, and updated things accordingly. Higher version might work, but figured minimal change to build stuff was best :)
See merge request !24
