rsync improvements for fdroid server update
This is a couple of improvements to how `fdroid server update` uses `rsync`.
(also, please remove the branch when accepting any of my merge requests)
This allows the SSH key used to sync with the server to be specified via
the config.py or the command line. I need it for running automated tests
and setups.
rsync uses the modification time and size of the file when deciding whether
to update a file. These are relatively easy to control in malicious code,
so instead make rsync use a full MD5 checksum when decided whether the
index needs to be updated. I suppose we could add an option to use
checksum checking on all files, but since the signed repo already provides
a checksum check, it seems not worth the added load on the process.
Also, renamed 'index' to 'indexxml' to make it clear what is the XML and
what is the JAR.
Test updates and related bug fixes
I just set up some big tests of generating repos based on feeding as many random APKs into `fdroid update` as possible. On our jenkins server, the tests copy all of the APKs that the jenkins server has generated and builds a repo from them. This process caught lots of little glitches in the whole process. While these little glitches are usually caused by problematic APKs, `fdroid update` should handle them gracefully. Hopefully this set of fixes accomplishes that.
In this case, ANDROID_HOME is set to a fake, non-working version that will
be detected by fdroid as an Android SDK install. It should use the path
set by --android-home over the one in ANDROID_HOME, therefore if it uses
the one in ANDROID_HOME, it won't work because it is a fake one. Only
--android-home provides a working one.
It is not necessarily a good idea to try to distribute system APKs via
FDroid, but `fdroid update` should just ignore APKs it cannot handle rather
than die and prevent a repo from being fully created. This is necessary to
handle the automatic creation of repos, like for debug builds from a
Jenkins server.
Using this example app which does not have <uses-sdk>:
https://android.googlesource.com/platform/development/+log/master/samples/ApiDemos/assets/HelloActivity.apk
aapt then returns "sdkVersion:'IceCreamSandwich'". minSdkVersion is only
ever supposed to be an integer, so this is a bizarre APK. It is included
only as a binary in the git repo for Android sample code. But who knows
what else is out there, so report and error and carry on with the update
process.
Some APKs can be corrupt or some system APKs do not have all the normal
info. Instead of quitting, `fdroid update` skips the non-parsable APKs and
optionally deletes them if --delete-unknown is specified.
FDroidPopen outputs by default, this should be controlled by the --verbose
flag so that most of the time, only meaningful messages are shown like
errors and such. For command output that should be shown everytime,
output=True can be set.
This lets people easily set whatever dir they want, while letting jenkins
search through its whole workspace for any APKs that have been built. Also,
only include the latest version of a given packageName+versionCode.
Yes, this includes a binary file, but it is only for the tests, and it is
free software since I wrote it. The source is here:
https://github.com/eighthave/urzip
Previously, `fdroid update -c` would only create the new metadata, but
would not add the new apps/apks to the repo. That required a second run of
`fdroid update`. This has been fixes, so this test makes sure it stays
fixed, in a very generic way.
Make sure that fdroid can find aapt in the current config, otherwise exit
with an error. Some users don't have build_tools set, and their SDK does
not include the build-tools in the default versioned dir, so this should
warn them of what is wrong.
Since the new metadata created by -c is added after the existing metadata
was already parsed, `fdroid update -c` was not doing a complete update of
the repo. This moves the metadata creation as early as possible, then
reruns the metadata parsing if new metadata was created.
refs #12https://gitlab.com/fdroid/fdroidserver/issues/12
This adds the option --delete-unknown for the current default behavior of
`fdroid update`: to delete any unknown APKs. Instead, it just outputs a
warning about the unknown APKs and suggests -c for adding it.
Fixes#8https://gitlab.com/fdroid/fdroidserver/issues/8
