Standalone Vagrantfile
@mvdan @CiaranG as a follow up on our work moving to a 64-bit build server VM, this moves the buildserver config to a standalone YAML file and commits a static _Vagrantfile_ to git. This makes it a lot easier to work with, especially for people who normally use git. The buildserver config is already a Python _dict_, and its trivial to export a _dict_ to a YAML file. _Vagrantfile_ is a Ruby script, where its trivial to load a YAML file. This moves some of the logic into _Vagrantfile_ itself. This means that someone can mess with _Yagrantfile.yaml_ and/or _Vagrantfile_ to work on the buildserver setup, without having to run `./makebuildserver`. Then once something is working, it can be ported to the current `./makebuildserver` setup that generates _Vagrantfile.yaml_.
This is important for working on getting this whole thing running in a KVM instance like jenkins.debian.net and elsewhere. From what I read, VirtualBox in KVM is only possible if VirtualBox is running in 32-bit mode, so that's a dead end for us. We need to be able to run the buildserver as KVM in KVM #190. This merge request doesn't get us there yet, but it makes the process a lot easier.
This also moves everything but Kivy to provisioning shell scripts, since the existing chef scripts were really just shell scripts wrapped in Chef wrapped in Vagrant wrapped in `./makebuildserver`.
This passes the gpjenkins CI build that creates the buildserver from scratch, then builds F-Droid and AdAway:
http://qssio5fppcrojdh3.onion:8080/job/fdroidserver-makebuildserver-eighthave/602/
I tried adding Amaze and Retrolambda as test apps for the buildserver too, we'll see how that goes:
http://qssio5fppcrojdh3.onion:8080/job/fdroidserver-makebuildserver-eighthave
See merge request !144
Vagrantfile is now committed and not changed between configurations. It is
configed by translating the python config file's dict to a YAML file, which
Vagrantfile now loads and uses. This makes it a lot easier for vagrant
users and python programmers to understand, and hopefully makes it easier
to maintain and test with.
Python can easily output dicts as YAML, and a Vagrantfile is a ruby script,
which can easily read YAML. Going this route means that Vagrantfile can
ultimately be committed to git, and the configuration will happen all via
Python dicts output as YAML. That makes it drastically easier to follow
the code, and to make modifications.
easy_install does not provide any download caching, while pip does. This
also moves the python module installing a shell script that takes python
packages as args. That will allow for future uses like allowing app
metadata to include pip modules that they need.
This makes it so there is only a single `apt-get install` command run,
instead of one command per-package like with the chef script. It also adds
`apt-get upgrade` to make sure that the base box is fully up-to-date.
This is part of the effort to remove moving parts from the whole build
server setup. Why wrap shell scripts in ruby and chef if we can just
directly run a shell script?
The other apps are too flaky on gpjenkins right now, and that's our
only box for running full buildserver tests. Once we get the
buildserver tests running on jenkins.debian.net, then we can add a
bunch more apps to the test script. gpjenkins is an extra locked down
box, so that's why the builds are flaky: gradle and maven downloads
regularly fail because they are blocked.
Note that the apt packages are split into two halves, because it takes
too long (on 64 bit!) to install them all. The sensible fix would be
to simply up the timeout on the package installation section, but this
is completely broken in chef.
Java8 and other fixes
This fixes the OSX travis-ci job for the Java8 update, and also two other fixes as described in the commit message. @CiaranG I think the _/etc/profile_ approach in 7b466b6266 gets us closer to the idea of having the `fdroid build` jobs run in the same environment as `vagrant ssh`.
See merge request !138
The ZIP format has no official encoding :-| so we have to do hacks. The
zipfile devs couldn't even sort this out:
https://bugs.python.org/issue10614closes#167
`fdroid update` should be able to handle any valid filename (hopefully
aapt doesn't barf on them). To handle that, the environment where the
shell commands are run in needs to have a UTF-8 locale set. If LANG is
not set, things default to ASCII and UTF-8 filenames fail.
This also renames test APK with lots of Unicode chars as a test case.
closes#167
This will make `vagrant ssh` and `fdroid build --server` be the same env,
so troubleshooting should be easier. !135 Here's what `man bash` says:
When bash is invoked as an interactive login shell, or as a
non-interactive shell with the --login option, it first reads and
executes commands from the file /etc/profile, if that file
exists. After reading that file, it looks for ~/.bash_profile,
~/.bash_login, and ~/.profile, in that order, and reads and
executes commands from the first one that exists and is readable.
The --noprofile option may be used when the shell is started to
inhibit this behavior. When a login shell exits, bash reads and
executes commands from the file ~/.bash_logout, if it exists.
This stops these errors:
fdroid/fdroidserver/fdroidserver/update.py:744: ResourceWarning: unclosed
file <_io.BufferedReader
name='repo/icons-320/info.guardianproject.urzip.100.png'>
fdroid/fdroidserver/fdroidserver/update.py:721: DeprecationWarning: The
'warn' function is deprecated, use 'warning' instead
all: switch to jdk8 as default
@eighthave this will probably require some action on the jenkins machine to replace jdk7 with jdk8.
Any thoughts @CiaranG?
See merge request !137
Also, remove jdk7 as it will become unused. We added jdk8 for
retrolambda, and now that we will use jdk8 as the default, jdk7 is
unnecessary as retrolambda can work fine with just jdk8.
This removes it from the buildserver, and the new CI image also only has
jdk8 from jessie-backports.
Fixes#185.
Some build fixes
This includes a couple of fixes discussed with @mvdan including removing the `ANDROID_NDK_HOME` env var from the buildserver and removing the default behavior to force the build-tools version.
See merge request !136
This replaces the current default behavior of always forcing the
build_tools version and allows the user to set build-tools forcing in
config.py.
closes#147
By checking first, this prevents a stacktrace when the duplicate metadata
file is not valid. For example, in the tests, the duplicate is just a zero
length file, which was causing a stacktrace.
`fdroid build` handles setting the NDK env vars since the NDK version can
change depending on the app being build. Unlike ANDROID_HOME, there is no
single global NDK location. The NDK installs are all versioned.
Fix buildserver, broken by e449d2f
Switching to a script /etc/profile.d to set up the environment is all
well and good, except that doesn't get run when you directly execute a
command directly via ssh, which means that the buildserver didn't work
at all (at least for anything that used gradle, or relied on the
environment variables.
This fix doesn't look very nice, but it works - it just forces the
appropriate script to run before build.py is executed on the server.
(Side note, I thought we had tests for this, how did it get past them?)
See merge request !135
Tone down makebuildserver terminal spamming
The progress bar updating at each 1k chunk spams the terminal to such an
extent it stops responding to keypresses (at least with a fast
connection). It also seems extremely inefficient to be writing the file
in 1k chunks. This makes it 64k, which is more reasonable but quite
probably still too small.
See merge request !134
Switching to a script /etc/profile.d to set up the environment is all
well and good, except that doesn't get run when you directly execute a
command directly via ssh, which means that the buildserver didn't work
at all (at least for anything that used gradle, or relied on the
environment variables.
This fix doesn't look very nice, but it works - it just forces the
appropriate script to run before build.py is executed on the server.
(Side note, I thought we had tests for this, how did it get past them?)
The progress bar updating at each 1k chunk spams the terminal to such an
extent it stops responding to keypresses (at least with a fast
connection). It also seems extremely inefficient to be writing the file
in 1k chunks. This makes it 64k, which is more reasonable but quite
probably still too small.
Adding support for DSA and ECDSA signatures in binaries
Before this patch only RSA signature files was supported when using the Binaries metadata directive as described in https://f-droid.org/wiki/page/Deterministic,_Reproducible_Builds.
This patch simply makes .DSA and .EC files also be recognised as signature files.
See merge request !133
switch SDK/NDK/gradle buildserver provisioning to shell scripts
This converts the SDK, NDK, and gradle chef recipes to vagrant provisioning shell scripts. Those recipes were just shell scripts anyway, forced into that ugly ruby/chef syntax. Hopefully this will make things easier to maintain since simple bash is easier for most devs than ruby/chef. Also, it might speed up the provisioning a little bit since the whole script is sent to the VM then executed, rather than sent line-by-line. Additionally, the SDK components are now installed using `android update sdk` so we do not need to duplicate Google's crazy kludges with that stuff.
See merge request !132
When the metadata changes, different things will be stored about each APK.
So invalidate the cached info parsed from APKs if the cache's metadata
version does not match the metadata version of the currently running tools.
