This is some very messy logic built up since 2010. This will all go away
once we have a python3 version of androguard available.
The removed imports and `dir(APK)` is to silence pyflakes
closes#303
This is a little omission. keys that are used in metadata/*.yml all start
with an UpperCase letter, but in fdroidserver, index-v1.json, and
fdroidclient, it is all camelCase with lowercase first letter. The keys
from the 'localized' section are currently never in metadata/*.yml, so
these keys never get downcase. This change will break fdroidclient
versions that do not also have this change, but since we're in alpha, that
should be fine.
If support for a 'localized' section is added to metadata/*.yml, then the
keys there should probably be UpperCase CamelCase to match the other keys.
Fastlane Supply, Triple-T Gradle Play Publisher, and many app stores
include the possibility to specify a website for the author, as distinct
from the website for the app.
closes#204
This uses the "What's New" entry for the CurrentVersionCode and includes it
as the current WhatsNew metadata for the App class.
Things like fastlane supply and Google Play support a "What's New" entry
per-APK, but fdroidclient does not current use anything but the current
version of this data. Right now, it seems we probably only want to have
the latest WhatsNew in the index to save space.
In theory, we could make the WhatsNew data structure follow the structure
of fastlane/Play, but that would quite a bit of complexity for something
that might never be used.
fdroidclient#910
This option was not hooked up at all, and does not make sense as a command
line argument. It should just be a config.py item. In that case, the
presence of config.py marks the current dir as a repo, so there is no
longer a need to test for a dir called repo/ as a safety. This makes the
setup easier, since sync_from_localcopy() now creates repo/ for the user.
Since `fdroid server update` is the place where all uploads to servers
happens, it makes sense to also handle the git push for the binary
transparency log here instead of `fdroid btlog`
Google has their own utility for verifying APK signatures on a desktop
machine since Java's jarsigner is bad for the task. For example, it
acts as if an unsigned APK validates. And to check whether an APK is
unsigned using jarsigner is difficult.
apksigner also does the v2 signatures, so it will have to be used
eventually anyway. It is already in Debian/stretch and can be
available in jessie-backports if need be.
https://android.googlesource.com/platform/tools/apksighttps://packages.debian.org/apksigner
The ZIP format allows multiple entries with the exact same filename, and on
top of that, it does not allow deleting or updating entries. To make the
`fdroid verify` procedure failsafe, it needs to create a new temporary APK
that is made up on the contents of the "unsigned APK" and the signature
from the "signed APK". Since it would be possible to give a signed APK as
in the unsigned one's position, `fdroid verify` was not able to update the
signature since it was just adding the new signature to the end of the ZIP
file. When reading a ZIP, the first entry is used.
This is a bit different than index.jar: instead of their being index.xml
and index_unsigned.jar, the presense of index-v1.json means that there is
unsigned data. That file is then stuck into a jar and signed by the
signing process. index-v1.json is never published to the repo. It is
included in the binary transparency log, if that is enabled.
This just makes it easier for people writing build recipes. Rewriting will
output a list of strings as well.
The test index.xml and categories.txt are updated to include the new number
categories, and the changed CurrentVersionCode to 2147483647 (MAX_VALUE)
This syncs up the field names between the fdroiddata .yml files, the keys
used in the implementation in fdroidserver, the index data format, and the
final data structures in fdroidclient. This makes it easier for devs to
follow, and makes the Jackson parsing library automatically handle
converting the data from the index file to Java instances.
This bumps the metadata version since the apkcache will have to be
discarded.
Here are the name changes:
* apkname --> apkName
* id --> packageName
* sha256 --> hash
* version --> versionName
* versioncode --> versionCode
tests/repo/index.xml was changed only to bump the metadata version
from 17 to 18.
Python encode/decode libs work directly with dicts, so the internal dict
can just be passed directly to any of these libs (pyyaml, pyjson, msgpack,
simplejson, etc). This still generates the exact same index.xml as before.
This converts the internal format for the repo timestamp to a datetime
instance, which can be easily converted to UNIX time in seconds for XML
and UNIX time in milliseconds for the new index formats. UNIX time in
milliseconds is directly serialized into a java.util.Date instance by
Jackson.
Since it is now possible to build and include arbitrary files, like OTA
update ZIP files, the update procedure needs to look for non-APK files that
match the packageName_versionCode pattern of fdroid-generated files.
!193
admin#14
privileged-extension#9
This lets index-v1 be parsed directly into class instances because the
field/instance var names match exactly. The original index v0 element
must retain the 'lastupdated' name for backwards compatibility.
If a group of items are enclosed in {}, then that will be a Python set,
which does not preserve order. To preserve order, the data must be either
a tuple () or list [].
Since https://gitlab.com/fdroid/ci-test-app is a separate git repo, things
with incompatible changes could get out of sync. Therefore, this test
should specify which git commit is runs against.
For example, the .fdroid.yml file is still a moving target. Just now, the
keys had the spaces removed as part of this MR.
In the future, we should have better internal datatypes for this stuff,
i.e. instead of gradle: ['yes'] for True, actually use a boolean. For now,
make the YAML and JSON metadata produce the same internal data as .txt.
This requires manually running it. I suppose it would be possible to
include a snapshot of the dumped internal representation for each release,
then make the tests run automatically against that. Right now, the dump is
17megs of YAML. Seems large to include in this git repo.
Like with the App class in the commit before, this makes it a lot
easier to work with this data when converting between the internal
formats and external formats like YAML, JSON, MsgPack, protobuf, etc.
The one unfortunate thing here is Build.update. It becomes
dict.update(), which is a method not an attribute.
build.get('update') or build['update'] could be used, but that would
be oddly inconsistent. So instead the field is renamed to
'androidupdate', except for in the .txt v0 metadata files. This better
describes what field does anyway, since it runs `android update`.
Build.update is only referenced in two places right next to each other
for the ant builds, so this change still seems worthwhile.
Python is heavily based on its core data types, and dict is one of the more
important ones. Even classes are basically a wrapper around a dict. This
converts metadata.App to be a subclass of dict so it can behave like a dict
when being dumped and loaded. This makes its drastically easier to use
different data formats for build metadata and for sending data to the
client. This approach will ultimately mean we no longer have to maintain
custom parsing and dumping code.
This also means then that the YAML/JSON field names will not have spaces in
them, and they will match exactly what it used as the dict keys once the
data is parsed, as well as matching exactly the instance attribute names:
* CurrentVersion: 1.2.6
* app['CurrentVersion'] == '1.2.6'
* app.CurrentVersion == '1.2.6'
Inspired by:
https://goodcode.io/articles/python-dict-object/
The original index.xml format needs to stay around for backwards
compatibility, but we shouldn't touch it anymore once the new format is in
place. This is a test to make sure `fdroid update` can still generate the
correct XML.
install_list and uninstall_list should be tuples or lists in order to
ensure that the order is preserved.
These tests also check that the added and lastupdated dates are
working correct, based on the dates in tests/stats/known_apks.txt. I
could see no useful way to test the timestamp, it is just hardcoded
using a regexp search-and-replace. Running these tests manually might
require deleting tmp/apkcache.
When making code changes related to the metadata parsing, it is useful to
see how the internal format has changed by seeing the differences in the
dump files. Those files are currently in the binary .pickle format. This
just straight converts them to YAML, which is a text format, so that normal
diff tools work to see changes.
The dump files are named .yaml instead of .yml since .yml is used for hand-
edited YAML files for fdroiddata/metadata, while these dump files here are
a human readable form of a Python pickle.
JSON and YAML are very closely related, so supporting both of them is
basically almost no extra work. Both are also closely related to how
Python works with dicts and pickles. XML is a very different beast, and its
not popular for this kind of thing anyway, so just purge it.
This allows a source repo to include a complete metadata file so that it
can be built directly in place using `fdroid build`. If that app is then
included in fdroiddata, it will first load the source repo type and URL
from fdroiddata, then read .fdroid.yml if it exists, then include the rest
of the metadata as specified in fdroiddata, so that fdroiddata has
precedence over the metadata in the source code.
This lets `fdroid build` apps without having a whole fdroiddata setup, but
instead just directly in place in the source code. This also lets devs
optionallu maintain the fdroid metadata as part of their app, rather than
in fdroiddata without loosing any control. This should make it easier to
spread around the maintenance load.
