This is the last thing using Chef, which adds a lot of time to the time it
takes to fully provision the buildserver. This slows down development on
the things we are actually using, like running all builds on
jenkins.debian.net.
#210#165
This reverts commit 82d09560c6.
It doesn't work - the setup scripts are expecting a ".bin" file (which
is apparently a 7z archive), but what's actually got is a ".zip".
Conflicts:
buildserver/provision-android-ndk
Without this, running makebuildserver from a clean master results in the
following:
$ ./makebuildserver
Traceback (most recent call last):
File "./makebuildserver", line 74, in <module>
del(config['__builtins__']) # added by compile/exec
KeyError: '__builtins__'
This is because a clean checkout has no config, thus exec is never
actually ran.
Amusingly, the commit changed more than just this so a simple git revert
is not possible.
The problem is that the zip is replaced with each point release (24.0.1
at the time of writing) and there is no way to get a URL to a single,
non-changing version. Hence any caching or checksums are completely
worthless and will break every few weeks.
Vagrantfile is now committed and not changed between configurations. It is
configed by translating the python config file's dict to a YAML file, which
Vagrantfile now loads and uses. This makes it a lot easier for vagrant
users and python programmers to understand, and hopefully makes it easier
to maintain and test with.
Python can easily output dicts as YAML, and a Vagrantfile is a ruby script,
which can easily read YAML. Going this route means that Vagrantfile can
ultimately be committed to git, and the configuration will happen all via
Python dicts output as YAML. That makes it drastically easier to follow
the code, and to make modifications.
easy_install does not provide any download caching, while pip does. This
also moves the python module installing a shell script that takes python
packages as args. That will allow for future uses like allowing app
metadata to include pip modules that they need.
This makes it so there is only a single `apt-get install` command run,
instead of one command per-package like with the chef script. It also adds
`apt-get upgrade` to make sure that the base box is fully up-to-date.
This is part of the effort to remove moving parts from the whole build
server setup. Why wrap shell scripts in ruby and chef if we can just
directly run a shell script?
`fdroid build` handles setting the NDK env vars since the NDK version can
change depending on the app being build. Unlike ANDROID_HOME, there is no
single global NDK location. The NDK installs are all versioned.
The progress bar updating at each 1k chunk spams the terminal to such an
extent it stops responding to keypresses (at least with a fast
connection). It also seems extremely inefficient to be writing the file
in 1k chunks. This makes it 64k, which is more reasonable but quite
probably still too small.
Prevent build processes from modifying the cache, it is only needed
during provisioning anyway. A malicious build could still use sudo to
change the cache, but this is more to prevent mistaken modifications.
This was not using anything special from chef, so do it in a shell script
instead. This makes the script easier for the python/shell people, and
probably uses less memory, since chef is a memory hog. This might even
make the provision go faster since it uploads the whole script as a file to
the VM, then runs it there. I think chef sends each command via SSH.
`android update sdk --no-ui` is the standard command line tool for
installing the Android SDK. By symlinking into the $ANDROID_HOME/temp dir,
the cached files can still be used. This converts the chef recipe to a
vagrant shell provisioning script since it was all bash anyway.
Some file names no longer officially have a -linux in them, so those were
changed to keep the cache working with the default filename.
bash provides a standard file location for a script to be run when the
shell starts: /etc/profile.d/ This converts the scattered bits of code for
making ~/.bsenv into a single provisioning script to generate
/etc/profile.d/bsenv.sh, which gets automatically executed when bash starts
It will make it a lot easier to manage the cache if we use the original
file names, which often include the file version. This also changes the
download process to be resumable if there is a partial file in the cache,
and switches from calling wget on the command line to using the python libs
'requests' and 'clint' to provide a similar experience. While its not so
important for this particular bit of code to use those libraries, I think
those two will allow us to provide a better user experience throughout the
whole of fdroidserver.
In this case, it is already doing special tricks fetching the file size
from the server before trying to download it. I suppose this code could
instead check if the file exists, and if so, check the hash sum. I think
that would be slower for most people since checking the hash on large files
takes a noticeable about of time, while a HTTP HEAD request is pretty tiny.
Install version 25 now. Also use the smaller tools zip. While at it,
also remove the tools re-install - it's not worth it, as long as we keep
the initial tools zip up to date.
Fixes a couple errors like:
File "./makebuildserver", line 30, in vagrant
out += line
TypeError: Can't convert 'bytes' object to str implicitly
If universal_newlines=False, the default, then Popen will return bytes if
the newlines in the data do not match the system's newlines. Setting it to
true enables auto-conversion, and then guarantees that the data is always
str.
"If universal_newlines is True, the file objects stdin, stdout and stderr
are opened as text streams in universal newlines mode, as described above
in Frequently Used Arguments, otherwise they are opened as binary streams."
https://docs.python.org/3/library/subprocess.html#subprocess.Popen
makebuildserver: support running VirtualBox in a VM
For debian.jenkins.net, our test environment is a kvm instance that does not expose the hardware virtualization instructions. So this auto-detects whether the current machine, virtual or not, supports the hardware virtualization. If not, it uses VirtualBox's software emulator, which should run everywhere, even in a kvm instance.
See merge request !108
