fdroid --version and shared tests between gitlab and jenkins
This adds a `fdroid --version` flag for people to easily see the exact version of fdroidserver in use. It'll also report the version using `git describe` when running from git.
The other moves the extended tests out of `./jenkins-build` into a common script for both gitlab-ci and jenkins.
See merge request !70
The tests use a little hack in order to cleanly import the fdroidserver
package locally like a regular package. pep8 doesn't see that, so this
changes the pep8 to skip E402 on *.TestCase
the `android` utility is pretty stupid, it doesn't really cache the
package index info. So each time it is run, it tries to fetch the
indexes from the network. This combines all android package installs
to a single command to make things run quicker.
This will report the version embedded in the module if it is installed, and
will report `git describe` if being run from git. If someone installs from
git using pip, this will probably report the version in setup.py, which
will be wrong. But that is not a documented install method, and I haven't
heard of anyone using it. The recommended way is to run straight from git.
Support XML, JSON, and YAML for metadata
Add support for app metadata files in JSON, XML, and YAML data formats. All of the formats use the exact same metadata tags, so there is no translation layer needed. They all just parse the data into the same internal data format: Python dicts. Supporting these standard formats will make it much easier for people to write recipes since they can choose a data format that they are familiar with. It also makes it much easier to generate metadata programmatically, since there are good libraries for working with all three formats in basically every language (unlike FDroid's .txt format).
Here are the same tags in .txt, JSON, XML, and YAML:
Source Code:https://github.com/SMSSecure/SMSSecure
"Source Code": "https://github.com/SMSSecure/SMSSecure",
<string name="Source Code">https://github.com/SMSSecure/SMSSecure</string>
Source Code: https://github.com/SMSSecure/SMSSecure
Looking for comments, suggestions, flames, etc. from @CiaranG, @mvdan, and everyone else.
See merge request !57
For a bit repo like f-droid.org, it makes sense to standardize on a single
format for metadata files. This adds support for enforcing a single data
format, or a reduced set of data formats. So f-droid.org would run like
this if it changed to YAML:
accepted_formats = ['txt', 'yaml']
Then once everything was converted to YAML, it could look like this:
accepted_formats = ['yaml']
In order to prevent confusion caused by multiple metadata files for a given
app, fdroid will exit with an error if it finds any app metadata file with
the same package ID as one that has already been parsed.
This puts process of setting up the defaults for the internal dict
that represents a parsed app into a single method that is reused for all
metadata formats.
YAML is a format that is quite similar to the .txt format, but is a
widespread standard that has editing modes in popular editors. It is also
easily parsable in python.
The .pickle for testing is a lightly edited version of the real metadata
for org.videolan.vlc:
* comments were removed
This puts all of the needed post parsing checks on the metadata into a
single method that is used by all parsing methods (.txt, JSON, XML, YAML).
This provides the single place to normalize the internal representation of
the metadata.
It would be good to also change the internal representation to use more Python
bool/int types so that less post parsing is needed for JSON, XML, and YAML.
The SMSSecure test .pickle was changed to account for the use of lstrip()
and rstrip() on all 'script' types.
This also changes the example JSON to use ints for versionCodes
No need for a specific metadata type for 'Requires Root'. Just use 'bool',
then convert on output. This keeps the internal representation consistent
since all boolean-ish values will be Python bools. This in turn makes
parsing metadata in various formats easier. This also makes setting
booleans more tolerant since all these values are valid:
Yes yes No no True true False false
While the current text metadata format is good for human readability and
editability, it is difficult to produce and parse using code. XML is a
widespread standard format for easy automatic parsing and creating, while
having decent human readability.
The .pickle for testing is a lightly edited version of the real metadata
for net.osmand.plus:
* comments were removed
* "NonFreeNet" was added as an AntiFeature
The AntiFeatures metadata is a comma-separated list of tags, like
Categories, so it should also be stored internally as a list. This makes
parsing XML and JSON easier.
The test cases' .pickle files look like they change a lot, but they really
don't, its only the change of default AntiFeatures value from None to []
The only data produced by parsing metadata that was stored as a tuple was
the comments' key/values. Everywhere else, lists are used. This changes
the comments to also only use lists to keep the data consistent internally.
That makes it vastly easier to produce the exact same metadata dict when
parsing .txt, JSON, XML, etc.
The .pickle was created by dumping the output from parsing the current .txt
metadata for org.adaway. The JSON started from that pickle dump, but was
then hand edited to be more proper JSON, e.g. using boolean values.
fdroidserver currently uses plain python 2.x strings rather than unicode
strings, so the parsed JSON should produce the same format as the parsed
.txt metadata.
This is a test to cover future modifications of the .txt metadata parsing.
The pickle file was generated by just dumping the current parsed metadata,
so this test will always succeed if the parsing is not changed.
While the current text metadata format is good for human readability and
editability, it is difficult to produce and parse using code. JSON is a
widespread standard format for easy automatic parsing and creating, while
having decent human readability.
