This is to clean up the code for the introduction of a new index format.
This also makes it so that only repo/categories.txt is generated, and not
archive/categories.txt.
Currently, the only thing that I can find that reads categories.txt is the
wordpress plugin, e.g. wp-fdroid. And it only reads repo/categories.txt
not archive/categories.txt.
Sometimes, a build process requires newer versions of build tools than are
available in Debian/stable. Oftentimes, using the package straight from
Debian/testing works fine when a package is not available as a backport.
libtool 2.4.6 is needed for building VLC, so it is one example of this.
The preferences file sets up the apt "pinning" so that all updates are not
installed from testing, only the packages that are requested by adding
"/testing" after then package name.
closes#224
In order to install a package from jessie-backports, apt-get has to be told
to get it from there rather than the main archive. It will not use
jessie-backports by default even if it is added as an apt source.
closes#224
* New command `dscanner`, enables one to scan signed APKs with Drozer
* Drozer is a dynamic vulnerability scanner for Android
* Drozer runs in a emulator or on-device, this new `dscanner` command...
* starts a docker image with Drozer and the Android Emulator pre-installed,
* loads the signed APK into the emulator
* activates Drozer automated tests for the APK
* gathers the report output and places it next to the original APK
* The Drozer docker image can be:
* cached locally for re-use (just don't run --clean*)
* retrieved from dockerhub.com for more efficient runtime
* or be built from scratch (in the new "./docker" directory)
* New "Vulnerability Scanning" documentation section (run gendocs.sh)
add KnownVuln anti-feature, and other fixes
This is a bit of a random collection of things that I have added in the process of working on the drozer/scanner support, YAML support, etc. The only new things are:
* adding new AntiFeature for security issues: `KnownVuln`
* removing broken, incomplete XML metadata support
Everything else included are just code improvements. This also includes the first check for `KnownVuln`, which is a scanner to check whether custom OpenSSL binaries in apps are not old with known vulnerabilities. `KnownVuln` will then ultimately be used for things like the drozer scanner !187
See merge request !189
When making code changes related to the metadata parsing, it is useful to
see how the internal format has changed by seeing the differences in the
dump files. Those files are currently in the binary .pickle format. This
just straight converts them to YAML, which is a text format, so that normal
diff tools work to see changes.
The dump files are named .yaml instead of .yml since .yml is used for hand-
edited YAML files for fdroiddata/metadata, while these dump files here are
a human readable form of a Python pickle.
JSON and YAML are very closely related, so supporting both of them is
basically almost no extra work. Both are also closely related to how
Python works with dicts and pickles. XML is a very different beast, and its
not popular for this kind of thing anyway, so just purge it.
This scans all APKs for old versions of OpenSSL libraries that are known to
be vulnerable to issues, or fully unsupported.
This really should be implemented as a per-APK AntiFeature, so that it can
apply to any version that is vulnerable. Since AntiFeatures are currently
only per-App, this instead sets the AntiFeature only if the latest APK is
vulnerable.
Google also enforces this:
https://support.google.com/faqs/answer/6376725?hl=en
apk['antiFeatures'] has the first letter small, since all build fields
start with a lowercase letter. app.AntiFeatures has the first
uppercase since all App fields are that way.
Since `fdroid update` parses the output of aapt, and since aapt's output
format changes in non-compatible ways, test to make sure that the version
of aapt is new enough to prevent mystery stacktraces. This only prints a
warning since in many cases, the old version will work just fine.
builds straight from source repo using .fdroid.yml
The overarching theme of the merge request is allowing _.fdroid.yml_ to be included in an app's source repo, then letting `fdroid build` build the app straight out of the git repo without requiring a setup like _fdroiddata_ (e.g. _config.py_, _metadata/packagename.txt_, etc.). _fdroiddata_ repos can then include source repos with a _.fdroid.yml_ by having _metadata/packagename.txt_ that includes just:
```
Repo Type:git
Repo:https://gitlab.com/upstream/app.git
```
Any other metadata fields that are included in _metadata/packagename.txt_ will override what is in _.fdroid.yml_, giving the repo manager the final say about what is included in their repo. This setup provides a number of benefits:
* CI systems like jenkins, travis, gitlab-ci can build from _.fdroid.yml_
* very easy to start building apps using `fdroid build`, no separate repo needed
* some maintenance can be offloaded to the upstream dev
See merge request !184
When a git repo has a .fdroid.yml file in it, and `fdroid build` is run
directly in that git repo, then this uses the file path as the remote for
the git repo in build/appid that is actually built. That makes it possible
to run builds of commits that are only local, and makes things a whole lot
faster.
This allows a source repo to include a complete metadata file so that it
can be built directly in place using `fdroid build`. If that app is then
included in fdroiddata, it will first load the source repo type and URL
from fdroiddata, then read .fdroid.yml if it exists, then include the rest
of the metadata as specified in fdroiddata, so that fdroiddata has
precedence over the metadata in the source code.
This lets `fdroid build` apps without having a whole fdroiddata setup, but
instead just directly in place in the source code. This also lets devs
optionallu maintain the fdroid metadata as part of their app, rather than
in fdroiddata without loosing any control. This should make it easier to
spread around the maintenance load.
Something like `gradle: yes` in YAML will be parsed as a boolean, since
'yes' is officially defined as a boolean true in YAML. For metadata fields
that need to be lists, this needs to be converted. Same goes for a single
string like `gradle: customFlavor`.
This includes more info to help track down problems with reproducible
builds, like the specific version being built, and which exact versions of
the Android SDK and NDK were used.
Any variation in the Android tools used to build an APK can cause the build
to be unreproducible. To help troubleshoot these times, this posts the
installed versions of the Android SDK and NDK components to the lastbuild
log, for the long term record.
refs #148
