This lets index-v1 be parsed directly into class instances because the
field/instance var names match exactly. The original index v0 element
must retain the 'lastupdated' name for backwards compatibility.
If a group of items are enclosed in {}, then that will be a Python set,
which does not preserve order. To preserve order, the data must be either
a tuple () or list [].
In the future, we should have better internal datatypes for this stuff,
i.e. instead of gradle: ['yes'] for True, actually use a boolean. For now,
make the YAML and JSON metadata produce the same internal data as .txt.
Like with the App class in the commit before, this makes it a lot
easier to work with this data when converting between the internal
formats and external formats like YAML, JSON, MsgPack, protobuf, etc.
The one unfortunate thing here is Build.update. It becomes
dict.update(), which is a method not an attribute.
build.get('update') or build['update'] could be used, but that would
be oddly inconsistent. So instead the field is renamed to
'androidupdate', except for in the .txt v0 metadata files. This better
describes what field does anyway, since it runs `android update`.
Build.update is only referenced in two places right next to each other
for the ant builds, so this change still seems worthwhile.
Python is heavily based on its core data types, and dict is one of the more
important ones. Even classes are basically a wrapper around a dict. This
converts metadata.App to be a subclass of dict so it can behave like a dict
when being dumped and loaded. This makes its drastically easier to use
different data formats for build metadata and for sending data to the
client. This approach will ultimately mean we no longer have to maintain
custom parsing and dumping code.
This also means then that the YAML/JSON field names will not have spaces in
them, and they will match exactly what it used as the dict keys once the
data is parsed, as well as matching exactly the instance attribute names:
* CurrentVersion: 1.2.6
* app['CurrentVersion'] == '1.2.6'
* app.CurrentVersion == '1.2.6'
Inspired by:
https://goodcode.io/articles/python-dict-object/
This makes it a lot easier to work with Build instances with parsing and
dumping libraries, since they expect only core Python types (dict, list,
tuple, str, etc)
ad2b9b99c2 put this in the wrong place, it
was running it on the buildserver host rather than in the buildserver VM
itself, where the builds actually run.
refs #148
This is to clean up the code for the introduction of a new index format.
This also makes it so that only repo/categories.txt is generated, and not
archive/categories.txt.
Currently, the only thing that I can find that reads categories.txt is the
wordpress plugin, e.g. wp-fdroid. And it only reads repo/categories.txt
not archive/categories.txt.
* New command `dscanner`, enables one to scan signed APKs with Drozer
* Drozer is a dynamic vulnerability scanner for Android
* Drozer runs in a emulator or on-device, this new `dscanner` command...
* starts a docker image with Drozer and the Android Emulator pre-installed,
* loads the signed APK into the emulator
* activates Drozer automated tests for the APK
* gathers the report output and places it next to the original APK
* The Drozer docker image can be:
* cached locally for re-use (just don't run --clean*)
* retrieved from dockerhub.com for more efficient runtime
* or be built from scratch (in the new "./docker" directory)
* New "Vulnerability Scanning" documentation section (run gendocs.sh)
JSON and YAML are very closely related, so supporting both of them is
basically almost no extra work. Both are also closely related to how
Python works with dicts and pickles. XML is a very different beast, and its
not popular for this kind of thing anyway, so just purge it.
This scans all APKs for old versions of OpenSSL libraries that are known to
be vulnerable to issues, or fully unsupported.
This really should be implemented as a per-APK AntiFeature, so that it can
apply to any version that is vulnerable. Since AntiFeatures are currently
only per-App, this instead sets the AntiFeature only if the latest APK is
vulnerable.
Google also enforces this:
https://support.google.com/faqs/answer/6376725?hl=en
apk['antiFeatures'] has the first letter small, since all build fields
start with a lowercase letter. app.AntiFeatures has the first
uppercase since all App fields are that way.
Since `fdroid update` parses the output of aapt, and since aapt's output
format changes in non-compatible ways, test to make sure that the version
of aapt is new enough to prevent mystery stacktraces. This only prints a
warning since in many cases, the old version will work just fine.
When a git repo has a .fdroid.yml file in it, and `fdroid build` is run
directly in that git repo, then this uses the file path as the remote for
the git repo in build/appid that is actually built. That makes it possible
to run builds of commits that are only local, and makes things a whole lot
faster.
This allows a source repo to include a complete metadata file so that it
can be built directly in place using `fdroid build`. If that app is then
included in fdroiddata, it will first load the source repo type and URL
from fdroiddata, then read .fdroid.yml if it exists, then include the rest
of the metadata as specified in fdroiddata, so that fdroiddata has
precedence over the metadata in the source code.
This lets `fdroid build` apps without having a whole fdroiddata setup, but
instead just directly in place in the source code. This also lets devs
optionallu maintain the fdroid metadata as part of their app, rather than
in fdroiddata without loosing any control. This should make it easier to
spread around the maintenance load.
Something like `gradle: yes` in YAML will be parsed as a boolean, since
'yes' is officially defined as a boolean true in YAML. For metadata fields
that need to be lists, this needs to be converted. Same goes for a single
string like `gradle: customFlavor`.
This includes more info to help track down problems with reproducible
builds, like the specific version being built, and which exact versions of
the Android SDK and NDK were used.
Any variation in the Android tools used to build an APK can cause the build
to be unreproducible. To help troubleshoot these times, this posts the
installed versions of the Android SDK and NDK components to the lastbuild
log, for the long term record.
refs #148
This makes sure there is a GPG signature on any file that is included in
the repo, including APKs, OBB, source tarballs, media files, OTA update
ZIPs, etc. Having a GPG signature is more important on non-APK files since
they mostly do not have any signature mechanism of their own.
This also adds basic tests of adding non-APK/OBB files to a repo with
`fdroid update`.
closes#232
This makes it so that the final build product can be specified in output=
and it'll work no matter if its an APK or not. This was developed around
the case of building the OTA update.zip for the Privileged Extension. It
should work for any build process in theory but it has not yet been tested.
https://gitlab.com/fdroid/privileged-extension/issues/9
The versionName is defined as a string or string resource that can be any
arbitrary data. fdroid should not second guess the developer here, and
should just use the versionName unmodified. For anything that needs to
compare different versions of apps, versionCode should always be used since
that's what Android uses.
https://developer.android.com/guide/topics/manifest/manifest-element.html#vname
Support media files
This is the first step in supporting adding artibitrary files to fdroid repos, targeted add supporting media files and flashable _update.zip_ files like used to install the Privileged Extension. This reuses the existing metadata fields to keep compatibilty with older versions, but that means that lots of names are confusingly named since they refer to aspects of an APK rather than a generic file. We can address that later when we refactor the whole index metadata.
See merge request !177
This adds the most basic level of support for including arbitrary files in
an F-Droid repository. This is useful for things like including videos,
ebooks, update.zip files for ROM updates, and more. The aim is to have
this as generic as possible to keep it flexible for unforeseen uses.
Code-wise, this is really just a first effort. This area of code has not
been touched in a very long time, and the repo parsing is done in a giant
function that is not easy to break apart. It should be broken up to more
cleanly support arbitrary files.
Also remove the TODO line, we've decided to keep the old permission
format for now, at least until there is a major overhaul of the index
data format. And the issue tracker the proper place for TODOs.
