`pip install` is a relatively common thing for app builds to use in sudo=,
so it should be built into the buildserver. This then builds in the trust
relationship to pypi, since it comes from Debian rather than some random
pip download/install process.
This expands the gradle wrapper shell script used by the buildserver for
usage outside the buildserver environment. It also allows downloading
whitelisted versions of gradle if they are not yet deployed to the
buildserver by simply upsating the copy of fdroidserver (in contrast to
having to reprovision the whole buildserver).
We first move the buildserver/gradle shell script to the repo root
as gradlew-fdroid, as it's an fdroid specific gradle wrapper.
We also now sync it inside the build VM before each build.
We then add a list of whitelisted gradle distributions taken from the
makebuildserver script.
The script additionally now reads two env vars which tell it where to
expect installed versions of gradle and where it might store downloaded
gradle .zip files. Both of those are configurable from config.py. As the
first should normally just be a subdir of the second it's not exposed in
the example config.py but only used by the buildserver config.py.
Default config now uses this internal gradle wrapper but a path to a
custom wrapper or specific gradle distribution can still be set from
config.py.
Closesfdroid/fdroidserver#98
Ref: fdroid/fdroidserver#370
When a new minor version of an NDK is released, it replaces an older one,
e.g. r16 with r16b (see commit 6f295cb). But old NDK package remains in
the cache and provisioning script unpacks it too as it matches the mask.
Fix NDK provisioning to unzip only while-listed versions.
We should be replacing all our custom git shell commands with python3-git,
since it is a common library for doing that. It will receive a lot more
attention and maintenance than our code for doing it. For example, we
should not ever use shell=True, since that opens up a lot of security
risks.
The currently included Qt has known security issues and is outdated. This
can now be replaced by downloading and installing the Qt installer using
the sudo= build field. @relan's provisioner system will also replace this
once that's done. There are only two apps that currently use the Qt stuff:
* csd.qtproject.minesweeper
* org.openorienteering.mapper
We never allow git via SSH or password/key access, and right now, this
causes things to hang forever. This sets things up to fail quickly
with invalid ssh connections.
BatchMode=yes - passphrase/password querying will be disabled.
StrictHostKeyChecking=yes - never automatically prompt, or add host keys to
the ~/.ssh/known_hosts file, and refuse to connect to hosts whose host key
has changed.
Now that fdroidserver requires Python 3, there's no need to tune sshd for
an old version of Paramiko because Ubuntu 14.04 does not have
python3-paramiko package in its repositories:
https://packages.ubuntu.com/search?keywords=python3-paramiko
We can safely assume that Ubuntu 14.04 users have installed a more recent
version of Paramiko from PIP.
The old hash does not work anymore, at least for platform-27. Where this
new one comes from:
1. Download command line tools from
https://developer.android.com/studio/index.html#command-tools
2. Unzip the package
3. Run ./tools/bin/sdkmanager --licenses
4. Read and accept licenses
5. Find the hash in ./licenses/android-sdk-license
Google releases new API SDKs quite rarely (about 2 times a year), but when
they do this, many apps quickly start using it. Allow downloading new API
SDKs to avoid waiting for the next buildserver rebuild.
Now that the download file name and type seems to have stabilized, I
think we no longer need to manually specify each new added release in
this script to unpack.
closes#331
Google is making gradle automatically download Android SDK components that
are needed by the build, but not already present. We need to support that
since it would be a lot of work to fight it. Plus, since each build starts
from the fresh snapshot, it should not be such a big deal to let each
build install stuff during the process.
closes#268
The ever troublesome gpjenkins box needs to use HTTPS mirrors. Plus it
improves the security of the buildserver, since there have been CVEs that
HTTPS would protect against:
https://www.debian.org/security/2016/dsa-3733
Google is pushing gradle towards downloading all the SDK components that it
needs, rather than having a preconfigured SDK installed. The buildserver
strongly supports the old model, with added checksum checking even. We can
still support the old model by pre-configuring the SDK and locking it down
as root. This can then also support the new model by setting the file perms
so that new packages can be auto-installed, but they cannot overwrite any
packages that come pre-installed and pre-verified.
fdroiddata!2096
closes#247
gradle will now automatically download and install missing bits of the
Android SDK. While we prefer to have the SDK packages fully verified, we
should allow this behavior on the buildserver to ensure that builds work
even when the buildserver can't be updated. Since each build starts from a
clean snapshot, this auto-installed build-tools will only be used for the
single build, so it won't affect other apps.
The new ConstraintLayout library in Android Support has some new custom way
of handling the license. I suspect that they are going to use this new way
with all of the bits that gradle downloads. We also have to support it for
apps that use it, including soon fdroidclient.
fdroiddata!2094
ci-images!1
These should be reasonably backwards compatible, and there is already
automake1.11 for those that need a version that old. As for cmake, there
are five apps that seem to it:
com.amaze.filemanager
org.dolphinemu.dolphinemu
org.navitproject.navit
org.yabause.android
org.videolan.vlc
It looks like VLC is the only app that is currently building and using
cmake in the most recent releases. Some of them used to use cmake, but no
longer.
Sometimes, a build process requires newer versions of build tools than are
available in Debian/stable. Oftentimes, using the package straight from
Debian/testing works fine when a package is not available as a backport.
libtool 2.4.6 is needed for building VLC, so it is one example of this.
The preferences file sets up the apt "pinning" so that all updates are not
installed from testing, only the packages that are requested by adding
"/testing" after then package name.
closes#224
In order to install a package from jessie-backports, apt-get has to be told
to get it from there rather than the main archive. It will not use
jessie-backports by default even if it is added as an apt source.
closes#224
buildserver running in qemu/kvm to support KVM on KVM
jenkins.debian.net runs in QEMU/KVM instances, so in order to run the F-Droid buildserver there, it needs to work inside of a KVM guest. The best way I found to do that is to create QEMU/KVM instances via KVM's "nested" virtualization support. This collection of commits enables using QEMU/KVM as the buildserver when `./makebuildserver` detects that it is running inside of KVM. Otherwise, the old behavior is default: running in VirtualBox.
I have run these tests inside of ubuntu/16.04 on bare metal, which uses VirtualBox, and ubuntu/16.04 KVM guest, which uses QEMU/KVM. It'll also run on the Guardian Project jenkins box, which is Debian/jessie.
@mvdan @CiaranG @krt
See merge request !168
I think the `android update sdk` tool is installing all of the m2 files
that are present in the temp cache, and it seems to do it in order of
newest to oldest. Well done, and I thought that tool couldn't get any
worse. So only include the latest version of android_m2repository*.zip in
the temp cache.
The Android SDK by default includes some Google repositories of their
proprietary SDKs. We of course do not want that stuff ever. We also do
not need the emulator images since this process does not currently install
or run an emulator.
This is the last thing using Chef, which adds a lot of time to the time it
takes to fully provision the buildserver. This slows down development on
the things we are actually using, like running all builds on
jenkins.debian.net.
#210#165
This does not have the careful result rechecking that chef has, when it
installs each package in the list one at a time. So to help with failures
caused by a package failing to download, first try downloading all the
package, then run the install. The install pass will try to download any
missing packages.
Really, this should use ansible or perhaps chef again since those include
lots of tricks around this stuff.
This forces the release channel and version of chef-solo to install on the
guest VM. I was getting really massive, odd stacktraces without specifying
this, and chef is only used for Kivy now anyway.
The technique where /var/cache/apt is mounted as a shared folder conflicts
with vagrant-cachier's workings. Therefore, ignore vagrant-cachier if the
user selects ./makebuildserver's custom apt cache. The shared folder way
has the advantage for CI builds of storing the cache outside of
VAGRANT_HOME, which is set to be in the git project. That gets wiped by
`git clean -fdx` on each CI build.
Add ndk r11c
NDK11 is required by some apps (some versions of VLC) which will not build with r10e or r12b. As always, please test this before merging: I havent added a NDK before.
See merge request !155
This reverts commit 82d09560c6.
It doesn't work - the setup scripts are expecting a ".bin" file (which
is apparently a 7z archive), but what's actually got is a ".zip".
Conflicts:
buildserver/provision-android-ndk
Vagrantfile is now committed and not changed between configurations. It is
configed by translating the python config file's dict to a YAML file, which
Vagrantfile now loads and uses. This makes it a lot easier for vagrant
users and python programmers to understand, and hopefully makes it easier
to maintain and test with.
Python can easily output dicts as YAML, and a Vagrantfile is a ruby script,
which can easily read YAML. Going this route means that Vagrantfile can
ultimately be committed to git, and the configuration will happen all via
Python dicts output as YAML. That makes it drastically easier to follow
the code, and to make modifications.
easy_install does not provide any download caching, while pip does. This
also moves the python module installing a shell script that takes python
packages as args. That will allow for future uses like allowing app
metadata to include pip modules that they need.
This makes it so there is only a single `apt-get install` command run,
instead of one command per-package like with the chef script. It also adds
`apt-get upgrade` to make sure that the base box is fully up-to-date.
This is part of the effort to remove moving parts from the whole build
server setup. Why wrap shell scripts in ruby and chef if we can just
directly run a shell script?
Note that the apt packages are split into two halves, because it takes
too long (on 64 bit!) to install them all. The sensible fix would be
to simply up the timeout on the package installation section, but this
is completely broken in chef.
Also, remove jdk7 as it will become unused. We added jdk8 for
retrolambda, and now that we will use jdk8 as the default, jdk7 is
unnecessary as retrolambda can work fine with just jdk8.
This removes it from the buildserver, and the new CI image also only has
jdk8 from jessie-backports.
Fixes#185.
`fdroid build` handles setting the NDK env vars since the NDK version can
change depending on the app being build. Unlike ANDROID_HOME, there is no
single global NDK location. The NDK installs are all versioned.
This was not using anything special from chef, so do it in a shell script
instead. This makes the script easier for the python/shell people, and
probably uses less memory, since chef is a memory hog. This might even
make the provision go faster since it uploads the whole script as a file to
the VM, then runs it there. I think chef sends each command via SSH.
`android update sdk --no-ui` is the standard command line tool for
installing the Android SDK. By symlinking into the $ANDROID_HOME/temp dir,
the cached files can still be used. This converts the chef recipe to a
vagrant shell provisioning script since it was all bash anyway.
Some file names no longer officially have a -linux in them, so those were
changed to keep the cache working with the default filename.
bash provides a standard file location for a script to be run when the
shell starts: /etc/profile.d/ This converts the scattered bits of code for
making ~/.bsenv into a single provisioning script to generate
/etc/profile.d/bsenv.sh, which gets automatically executed when bash starts
Apparently, ruby is quite a memory hog when it forks. I've been getting
errors like this with ./makebuildserver:
Errno::ENOMEM - script[add_btools_17] (android-sdk::default line 72) had an
error: Errno::ENOMEM: Cannot allocate memory - fork(2)
So instead of looping in ruby and forking for each loop, handle the looping
in the bash script, so ruby is just calling a single bash script.
It will make it a lot easier to manage the cache if we use the original
file names, which often include the file version. This also changes the
download process to be resumable if there is a partial file in the cache,
and switches from calling wget on the command line to using the python libs
'requests' and 'clint' to provide a similar experience. While its not so
important for this particular bit of code to use those libraries, I think
those two will allow us to provide a better user experience throughout the
whole of fdroidserver.
In this case, it is already doing special tricks fetching the file size
from the server before trying to download it. I suppose this code could
instead check if the file exists, and if so, check the hash sum. I think
that would be slower for most people since checking the hash on large files
takes a noticeable about of time, while a HTTP HEAD request is pretty tiny.
Install version 25 now. Also use the smaller tools zip. While at it,
also remove the tools re-install - it's not worth it, as long as we keep
the initial tools zip up to date.
