Without this, running makebuildserver from a clean master results in the
following:
$ ./makebuildserver
Traceback (most recent call last):
File "./makebuildserver", line 74, in <module>
del(config['__builtins__']) # added by compile/exec
KeyError: '__builtins__'
This is because a clean checkout has no config, thus exec is never
actually ran.
Amusingly, the commit changed more than just this so a simple git revert
is not possible.
The problem is that the zip is replaced with each point release (24.0.1
at the time of writing) and there is no way to get a URL to a single,
non-changing version. Hence any caching or checksums are completely
worthless and will break every few weeks.
Currently, if buildjni is not used but ndk is set to an invalid value,
the build would start but $ANDROID_NDK would be empty. This is happening
in VLC, which results in very confusing errors.
If a build uses a ndk= value that is not set up, such as r11/r12 which
we do not have yet, it should error with "NDK version could not be
found". It does with this change.
Pass a date from APK to KnownApks.recordapk()
... if --use-date-from-apks option is used.
Essentially, it just expands influence of `--use-date-from-apks` option to `stats/known_apks.txt`.
See merge request !141
Properly print warnings to stderr. Also, use : instead of 'echo' as a
fallback as the latter spits out garbage to stdout.
Examples without pep8 installed.
Before:
$ ./hooks/pre-commit
./hooks/pre-commit: line 97: WARNING:: command not found
ERROR: pep8 tests failed!
After:
$ ./hooks/pre-commit
WARNING: pep8 is not installed, using dummy placeholder!
support APK Extension OBB files
Google Play specifies OBB aka "APK Extension" files for apps that need more than 100 MBs, which is the Play APK size limit. They also provide a mechanism to deliver large data blobs that do not need to be part of the APK. For example, a game's assets do not need to change often, so they can be shipped as an OBB, then APK updates do not need to include all those assets for each update.
https://developer.android.com/google/play/expansion-files.html
See merge request !143
OBB files provide a mechanism to deliver large data blobs that do not need
to be part of the APK. For example, a game's assets do not need to change
often, so they can be shipped as an OBB, then APK updates do not need to
include all those assets for each update.
https://developer.android.com/google/play/expansion-files.html
Standalone Vagrantfile
@mvdan @CiaranG as a follow up on our work moving to a 64-bit build server VM, this moves the buildserver config to a standalone YAML file and commits a static _Vagrantfile_ to git. This makes it a lot easier to work with, especially for people who normally use git. The buildserver config is already a Python _dict_, and its trivial to export a _dict_ to a YAML file. _Vagrantfile_ is a Ruby script, where its trivial to load a YAML file. This moves some of the logic into _Vagrantfile_ itself. This means that someone can mess with _Yagrantfile.yaml_ and/or _Vagrantfile_ to work on the buildserver setup, without having to run `./makebuildserver`. Then once something is working, it can be ported to the current `./makebuildserver` setup that generates _Vagrantfile.yaml_.
This is important for working on getting this whole thing running in a KVM instance like jenkins.debian.net and elsewhere. From what I read, VirtualBox in KVM is only possible if VirtualBox is running in 32-bit mode, so that's a dead end for us. We need to be able to run the buildserver as KVM in KVM #190. This merge request doesn't get us there yet, but it makes the process a lot easier.
This also moves everything but Kivy to provisioning shell scripts, since the existing chef scripts were really just shell scripts wrapped in Chef wrapped in Vagrant wrapped in `./makebuildserver`.
This passes the gpjenkins CI build that creates the buildserver from scratch, then builds F-Droid and AdAway:
http://qssio5fppcrojdh3.onion:8080/job/fdroidserver-makebuildserver-eighthave/602/
I tried adding Amaze and Retrolambda as test apps for the buildserver too, we'll see how that goes:
http://qssio5fppcrojdh3.onion:8080/job/fdroidserver-makebuildserver-eighthave
See merge request !144
Vagrantfile is now committed and not changed between configurations. It is
configed by translating the python config file's dict to a YAML file, which
Vagrantfile now loads and uses. This makes it a lot easier for vagrant
users and python programmers to understand, and hopefully makes it easier
to maintain and test with.
Python can easily output dicts as YAML, and a Vagrantfile is a ruby script,
which can easily read YAML. Going this route means that Vagrantfile can
ultimately be committed to git, and the configuration will happen all via
Python dicts output as YAML. That makes it drastically easier to follow
the code, and to make modifications.
easy_install does not provide any download caching, while pip does. This
also moves the python module installing a shell script that takes python
packages as args. That will allow for future uses like allowing app
metadata to include pip modules that they need.
This makes it so there is only a single `apt-get install` command run,
instead of one command per-package like with the chef script. It also adds
`apt-get upgrade` to make sure that the base box is fully up-to-date.
This is part of the effort to remove moving parts from the whole build
server setup. Why wrap shell scripts in ruby and chef if we can just
directly run a shell script?
The other apps are too flaky on gpjenkins right now, and that's our
only box for running full buildserver tests. Once we get the
buildserver tests running on jenkins.debian.net, then we can add a
bunch more apps to the test script. gpjenkins is an extra locked down
box, so that's why the builds are flaky: gradle and maven downloads
regularly fail because they are blocked.
Note that the apt packages are split into two halves, because it takes
too long (on 64 bit!) to install them all. The sensible fix would be
to simply up the timeout on the package installation section, but this
is completely broken in chef.
Java8 and other fixes
This fixes the OSX travis-ci job for the Java8 update, and also two other fixes as described in the commit message. @CiaranG I think the _/etc/profile_ approach in 7b466b6266 gets us closer to the idea of having the `fdroid build` jobs run in the same environment as `vagrant ssh`.
See merge request !138
The ZIP format has no official encoding :-| so we have to do hacks. The
zipfile devs couldn't even sort this out:
https://bugs.python.org/issue10614closes#167
`fdroid update` should be able to handle any valid filename (hopefully
aapt doesn't barf on them). To handle that, the environment where the
shell commands are run in needs to have a UTF-8 locale set. If LANG is
not set, things default to ASCII and UTF-8 filenames fail.
This also renames test APK with lots of Unicode chars as a test case.
closes#167
This will make `vagrant ssh` and `fdroid build --server` be the same env,
so troubleshooting should be easier. !135 Here's what `man bash` says:
When bash is invoked as an interactive login shell, or as a
non-interactive shell with the --login option, it first reads and
executes commands from the file /etc/profile, if that file
exists. After reading that file, it looks for ~/.bash_profile,
~/.bash_login, and ~/.profile, in that order, and reads and
executes commands from the first one that exists and is readable.
The --noprofile option may be used when the shell is started to
inhibit this behavior. When a login shell exits, bash reads and
executes commands from the file ~/.bash_logout, if it exists.
This stops these errors:
fdroid/fdroidserver/fdroidserver/update.py:744: ResourceWarning: unclosed
file <_io.BufferedReader
name='repo/icons-320/info.guardianproject.urzip.100.png'>
fdroid/fdroidserver/fdroidserver/update.py:721: DeprecationWarning: The
'warn' function is deprecated, use 'warning' instead
