In the future, we should have better internal datatypes for this stuff,
i.e. instead of gradle: ['yes'] for True, actually use a boolean. For now,
make the YAML and JSON metadata produce the same internal data as .txt.
Like with the App class in the commit before, this makes it a lot
easier to work with this data when converting between the internal
formats and external formats like YAML, JSON, MsgPack, protobuf, etc.
The one unfortunate thing here is Build.update. It becomes
dict.update(), which is a method not an attribute.
build.get('update') or build['update'] could be used, but that would
be oddly inconsistent. So instead the field is renamed to
'androidupdate', except for in the .txt v0 metadata files. This better
describes what field does anyway, since it runs `android update`.
Build.update is only referenced in two places right next to each other
for the ant builds, so this change still seems worthwhile.
Python is heavily based on its core data types, and dict is one of the more
important ones. Even classes are basically a wrapper around a dict. This
converts metadata.App to be a subclass of dict so it can behave like a dict
when being dumped and loaded. This makes its drastically easier to use
different data formats for build metadata and for sending data to the
client. This approach will ultimately mean we no longer have to maintain
custom parsing and dumping code.
This also means then that the YAML/JSON field names will not have spaces in
them, and they will match exactly what it used as the dict keys once the
data is parsed, as well as matching exactly the instance attribute names:
* CurrentVersion: 1.2.6
* app['CurrentVersion'] == '1.2.6'
* app.CurrentVersion == '1.2.6'
Inspired by:
https://goodcode.io/articles/python-dict-object/
When making code changes related to the metadata parsing, it is useful to
see how the internal format has changed by seeing the differences in the
dump files. Those files are currently in the binary .pickle format. This
just straight converts them to YAML, which is a text format, so that normal
diff tools work to see changes.
The dump files are named .yaml instead of .yml since .yml is used for hand-
edited YAML files for fdroiddata/metadata, while these dump files here are
a human readable form of a Python pickle.
JSON and YAML are very closely related, so supporting both of them is
basically almost no extra work. Both are also closely related to how
Python works with dicts and pickles. XML is a very different beast, and its
not popular for this kind of thing anyway, so just purge it.
This allows a source repo to include a complete metadata file so that it
can be built directly in place using `fdroid build`. If that app is then
included in fdroiddata, it will first load the source repo type and URL
from fdroiddata, then read .fdroid.yml if it exists, then include the rest
of the metadata as specified in fdroiddata, so that fdroiddata has
precedence over the metadata in the source code.
This lets `fdroid build` apps without having a whole fdroiddata setup, but
instead just directly in place in the source code. This also lets devs
optionallu maintain the fdroid metadata as part of their app, rather than
in fdroiddata without loosing any control. This should make it easier to
spread around the maintenance load.
Something like `gradle: yes` in YAML will be parsed as a boolean, since
'yes' is officially defined as a boolean true in YAML. For metadata fields
that need to be lists, this needs to be converted. Same goes for a single
string like `gradle: customFlavor`.
Though the YAML people recommend .yaml for the file extension, in Android
land it seems clear that .yml has won out:
* .travis.yml
* .gitlab-ci.yml
* .circle.yml
* Ansible main.yml
Only keep lists in metadata files in the json format, since they don't
support multiline strings that are readable.
This makes the internal code easier, and a bit faster.
This simplifies usage, goes from
build['flag']
to
build.flag
Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.
As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.
While at it, unify "build", "thisbuild", "info", "thisinfo", etc into
just "build".
This simplifies usage, goes from
app['Foo']
to
app.Foo
Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.
As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.
In order to prevent confusion caused by multiple metadata files for a given
app, fdroid will exit with an error if it finds any app metadata file with
the same package ID as one that has already been parsed.
YAML is a format that is quite similar to the .txt format, but is a
widespread standard that has editing modes in popular editors. It is also
easily parsable in python.
The .pickle for testing is a lightly edited version of the real metadata
for org.videolan.vlc:
* comments were removed
This puts all of the needed post parsing checks on the metadata into a
single method that is used by all parsing methods (.txt, JSON, XML, YAML).
This provides the single place to normalize the internal representation of
the metadata.
It would be good to also change the internal representation to use more Python
bool/int types so that less post parsing is needed for JSON, XML, and YAML.
The SMSSecure test .pickle was changed to account for the use of lstrip()
and rstrip() on all 'script' types.
This also changes the example JSON to use ints for versionCodes
While the current text metadata format is good for human readability and
editability, it is difficult to produce and parse using code. XML is a
widespread standard format for easy automatic parsing and creating, while
having decent human readability.
The .pickle for testing is a lightly edited version of the real metadata
for net.osmand.plus:
* comments were removed
* "NonFreeNet" was added as an AntiFeature
The AntiFeatures metadata is a comma-separated list of tags, like
Categories, so it should also be stored internally as a list. This makes
parsing XML and JSON easier.
The test cases' .pickle files look like they change a lot, but they really
don't, its only the change of default AntiFeatures value from None to []
The .pickle was created by dumping the output from parsing the current .txt
metadata for org.adaway. The JSON started from that pickle dump, but was
then hand edited to be more proper JSON, e.g. using boolean values.
This is a test to cover future modifications of the .txt metadata parsing.
The pickle file was generated by just dumping the current parsed metadata,
so this test will always succeed if the parsing is not changed.
