The new policy is to move APKs with invalid signatures to the archive,
and only add those APKs to the archive's index if they have valid MD5
signatures.
closes#323closes#292
For people using slow, expensive, and/or flaky internet, liberal use of
caching can make a huge difference. The restricted environment of the
gpjenkins box has been a good test environment for this (Tor-only,
whitelist of allowed IPs to visit, home internet connection).
This syncs up the field names between the fdroiddata .yml files, the keys
used in the implementation in fdroidserver, the index data format, and the
final data structures in fdroidclient. This makes it easier for devs to
follow, and makes the Jackson parsing library automatically handle
converting the data from the index file to Java instances.
This bumps the metadata version since the apkcache will have to be
discarded.
Here are the name changes:
* apkname --> apkName
* id --> packageName
* sha256 --> hash
* version --> versionName
* versioncode --> versionCode
tests/repo/index.xml was changed only to bump the metadata version
from 17 to 18.
Using datetime instances as the internal format makes it much easier to
convert between the formats needed for index.xml and index-v1. apkcache
still uses time tuples and known_apks.txt still uses the ISO date.
A Binary Transparency Log is a append only log of all binaries published by
a repo. This is useful for people to find whether the binary they have
matches what F-Droid has published, and also makes it more difficult for
the published history to be changed without notice, or for a server to give
specific users custom malware binaries.
https://www.eff.org/deeplinks/2014/02/open-letter-to-tech-companies
The original index.xml format needs to stay around for backwards
compatibility, but we shouldn't touch it anymore once the new format is in
place. This is a test to make sure `fdroid update` can still generate the
correct XML.
install_list and uninstall_list should be tuples or lists in order to
ensure that the order is preserved.
These tests also check that the added and lastupdated dates are
working correct, based on the dates in tests/stats/known_apks.txt. I
could see no useful way to test the timestamp, it is just hardcoded
using a regexp search-and-replace. Running these tests manually might
require deleting tmp/apkcache.
JSON and YAML are very closely related, so supporting both of them is
basically almost no extra work. Both are also closely related to how
Python works with dicts and pickles. XML is a very different beast, and its
not popular for this kind of thing anyway, so just purge it.
This makes sure there is a GPG signature on any file that is included in
the repo, including APKs, OBB, source tarballs, media files, OTA update
ZIPs, etc. Having a GPG signature is more important on non-APK files since
they mostly do not have any signature mechanism of their own.
This also adds basic tests of adding non-APK/OBB files to a repo with
`fdroid update`.
closes#232
For running in QEMU/KVM guests like on jenkins.debian.net, this sets up the
whole process automatically. This only really covers the case where this
is running in a KVM guest, and the original case of running VirtualBox on
bare metal. It could be extended to cover more cases if someone wanted to.
It is now possible for the server operator to specify lists of apps that
must be installed or deleted on the client (aka "push installs). If
the user has opted in, or the device is already setup to respond to
these requests, then fdroidclient will automatically install/delete
the packageNames listed. This is protected by the same signing key
as the app index metadata.
It generates single XML elements with the data set in the attributes. This
keeps the XML compact and easily extensible, e.g. for adding versionCode,
signingKey, etc as attributes:
<install packageName="com.fsck.k9"/>
<install packageName="at.bitfire.davdroid"/>
<delete packageName="com.facebook.orca"/>
Copyright: 2016 Blue Jay Wireless
Signed-off-by: Hans-Christoph Steiner <hans@eds.org>
closes#177
This reverts commit 82d09560c6.
It doesn't work - the setup scripts are expecting a ".bin" file (which
is apparently a 7z archive), but what's actually got is a ".zip".
Conflicts:
buildserver/provision-android-ndk
Also, remove jdk7 as it will become unused. We added jdk8 for
retrolambda, and now that we will use jdk8 as the default, jdk7 is
unnecessary as retrolambda can work fine with just jdk8.
This removes it from the buildserver, and the new CI image also only has
jdk8 from jessie-backports.
Fixes#185.
This replaces the current default behavior of always forcing the
build_tools version and allows the user to set build-tools forcing in
config.py.
closes#147
serverwebroot has long supported uploading to multiple servers, this bit of
metadata communicates those official mirrors to the client so that it can
automatically do something useful with that information.
closes#14https://gitlab.com/fdroid/fdroidserver/issues/14
4 gigs is still a common amount of RAM these days for laptops, if the VM
takes almost all of that, it makes the machine drag to almost a halt. Most
apps build fine in 1gig of RAM, indeed that's the default for most CI
instances, like travis-ci and gitlab-ci.
On slow machines or VMs like the Debian jenkins box, the VM boot timeout
needs to be a lot longer, otherwise vagrant times out before setting up
the VM.
This also provides a config option to override that default. ~/.cache is
a standard location on GNU/Linux machines for cached content. It is also
good to have the cache outside of the git repo in case `git clean -fdx` is
run, which would delete all files in the directory that are not part of the
git repo, including buildserver/cache/
This makes it so that ./makebuildserver will run without any config file,
using the defaults that are embedded in the script itself. This is like
how `fdroid` works.
This host automatically detects which is the closest mirror, then uses that
one. It does so dynamically, so it'll work on machines that move too. Now
that we are pushing more people to run F-Droid build servers, the defaults
should take those use cases into account.
This keeps the numbers of names down to a minimum, and since the config
is placed right next to the script, this keeps tab completion working
nicely when the config file is in place.
The old file name is still supported.
For a bit repo like f-droid.org, it makes sense to standardize on a single
format for metadata files. This adds support for enforcing a single data
format, or a reduced set of data formats. So f-droid.org would run like
this if it changed to YAML:
accepted_formats = ['txt', 'yaml']
Then once everything was converted to YAML, it could look like this:
accepted_formats = ['yaml']
config option to enable per-app repos for nightly builds
For Guardian Project, we've been running an fdroid repo for the nightly builds for each of our apps: https://dev.guardianproject.info/debug This is built using a big, hacked up shell script: [update-debug-fdroid-repo](https://github.com/guardianproject/fdroid-repo-tools/blob/master/update-debug-fdroid-repo). It has proven very useful to us to be able to subscribe to the nightly build for a single app, so this the first step of porting that horrid shell script to `fdroidserver`.
This also helps make the fdroidserver tool suite the single set of tools for all types of builds and releases. That will hopefully drive more free software developers to make f-droid.org the core channel for official releases.
See merge request !66
config.vm.box_url can be a list/tuple of URLs, which is useful to specific
a locally cached copy. This is needed on slow connections, so that if it
fails, the download of jessie32.box does not have to start from the
beginning of the file again.
For devs that want to build and distribute nightly builds of their apps
using the fdroid tools. The core idea here is to make the fdroidserver
tool suite the single set of tools for all types of builds and releases.
That will hopefully drive more free software developers to make f-droid.org
an core channel for official releases.
Update documentation re: testing32 image
Use different subsections for the Debian base box setup (which can be bypassed using the prebuilt image) and the F-Droid / SDK setup (which cannot be bypassed).
Side note: would it be possible to offer a torrent for testing32.box? The direct download is taking over 2 hours on a 50Mbps link.
See merge request !32
By not setting sdk_path when /usr/bin/aapt is found, sdk_path then defaults
to $ANDROID_HOME when its used. Since in this case, aapt will be used from
the system path, using aapt entirely ignores sdk_path. If the user runs
`fdroid build` in this setup, sdk_path will be $ANDROID_HOME, so it should
check the env vars for it, but maybe that doesn't actually work like that
yet.
I wrote up the feature to automatically generate symlinks with a constant name
that points to the current release version. I have it on by default, with a
*config.py* option to turn it off. There is also an option to set where the
symlink name comes from which defaults to app['Name'] i.e. F-Droid.apk, but
can easily be set to app['id'], i.e. _org.fdroid.fdroid.apk_. I think the best
place for the symlinks is in the root of the repo, so like
https://f-droid.org/F-Droid.apk or https://guardianproject.info/fdroid/ChatSecure.apk
For the case of the current FDroid static link https://f-droid.org/FDroid.apk
it can just be a symlink to the generated one (https://f-droid.org/F-Droid.apk
or https://f-droid.org/org.fdroid.fdroid.apk). Right now, this feature is all
or nothing, meaning it generates symlinks for all apps in the repo, or none. I
can’t think of any problems that this might cause since its only symlinks, so
the amount of disk space is tiny. Also, I think it would be useful for having
an easy “Download this app” button on each app’s page on the “Browse” view. As
long as this button is less prominent than the “Download F-Droid” button, and
it is clear that it is better to use the FDroid app than doing direct
downloads. For the f-droid.org repo, the symlinks should probably be based on
app['id'] to prevent name conflicts.
more info here:
https://f-droid.org/forums/topic/static-urls-to-current-version-of-each-app/
This gives us flexibility in how the blocks of text can be formatted in
config.py, but also provides a more useful format for displaying since the
client can decide where to wrap the text.
To support a fully offline build/signing machine, there is the "local copy
dir". The repo is generated on the offline machine and then copied to a
local dir where a thumb drive or SD Card is mounted. Then on the online
machine, using `fdroid server update --sync-from-local-copy-dir` allows
the whole server update process to happen in a single command:
0. read config.py on online machine's repo
1. rsync from the local_copy_dir to the current dir
2. copy to serverwebroot, awsbucket, etc.
This allows a dir to be specified in config.py that `fdroid server update`
will automatically rsync the repo to. The idea is that the path would
point to an SD card on a fully offline machine that serves as the secure
repo signing machine.
This allows the SSH key used to sync with the server to be specified via
the config.py or the command line. I need it for running automated tests
and setups.
apache-libcloud enables uploading to basically any cloud storage service.
This is the first implementation that allows `fdroid server` to push a repo
up to a AWS S3 'bucket'. Supporting other cloud storage services should
mostly be a matter of finding the libcloud "Provider" and setting the
access creditials.
fixes#3137https://dev.guardianproject.info/issues/3137
Having serverwebroot optional in `fdroid server` means that it can support
multiple methods of hosting, like cloud storage services. `fdroid server`
can also then support multiple repo hosting options at the same time.
This assumes that the smartcard is already setup with a signing key. init
does not generate a key on the smartcard, and skips genkey() if things are
configured to use a smartcard.
This also does not touch APK signing because that is a much more elaborate
question, since each app is signed by its own key.
These options are needed to configure Java's keytool and jarsigner to use
a Hardware Security Module aka HSM aka smartcard. The defaults provided
are meant to make things work as easily as possible.
This also makes the file layout in git basically the same as the installed
file layout, using an examples/ dir. I'm not sure if config.buildserver.py
is an example conf file, or a conf file that is actually in use, so I did
not move it.
