.java .gradle and XML files all can use any encoding. Most code is ASCII,
but authors' names, etc. can easily be non-ASCII. UTF-8 is by far the most
common file encoding. While UTF-8 is the default encoding inside the code
in Python 3, it still has to deal with the real world, so the encoding
needs to be explicitly set when reading and writing files. So this switches
fdroidserver to expect UTF-8 instead of ASCII when parsing these files. For
now, this commit means that we only support UTF-8 encoded *.java, pom.xml
or *.gradle files. Ideally, the code would detect the encoding and use the
actual one, but that's a lot more work, and its something that will not
happen often. We can cross that bridge when we come to it.
One approach, which is taken in the commit when possible, is to keep the
data as `bytes`, in which case the encoding doesn't matter.
This also fixes this crash when parsing gradle and maven files with
non-ASCII chars:
ERROR: test_adapt_gradle (__main__.BuildTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/tests/build.TestCase", line 59, in test_adapt_gradle
fdroidserver.build.adapt_gradle(testsdir)
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/fdroidserver/build.py", line 445, in adapt_gradle
path)
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/fdroidserver/common.py", line 188, in regsub_file
text = f.read()
File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 9460: ordinal not in range(128)
Though the YAML people recommend .yaml for the file extension, in Android
land it seems clear that .yml has won out:
* .travis.yml
* .gitlab-ci.yml
* .circle.yml
* Ansible main.yml
The start up sequence of processes that are based on the .fdroid.* metadata
is a bit different, so this ensures that the environment variables get
properly initialized in all cases.
This also creates a single function where the environment is set. Before
it was being set in multiple places across multiple files.
serverwebroot has long supported uploading to multiple servers, this bit of
metadata communicates those official mirrors to the client so that it can
automatically do something useful with that information.
closes#14https://gitlab.com/fdroid/fdroidserver/issues/14
Using the same JDK throughout should prevent weird bugs where a setup might
use Java8's jarsigner and Java7's keytool. This also allows the user to
set java_paths and have jarsigner and keytool used from that specified JDK.
This incorporates almost all of the patch that is in the Debian package
that forces fdroidserver to use the default JDK on that Debian release.
closes#93https://gitlab.com/fdroid/fdroidserver/issues/93
Only keep lists in metadata files in the json format, since they don't
support multiline strings that are readable.
This makes the internal code easier, and a bit faster.
This simplifies usage, goes from
build['flag']
to
build.flag
Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.
As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.
While at it, unify "build", "thisbuild", "info", "thisinfo", etc into
just "build".
This simplifies usage, goes from
app['Foo']
to
app.Foo
Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.
As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.
Don't log and exit in an inner metadata function. Handle it at a higher
level and do a proper exception. This also avoids unnecessary passing of
apps all around.
This will report the version embedded in the module if it is installed, and
will report `git describe` if being run from git. If someone installs from
git using pip, this will probably report the version in setup.py, which
will be wrong. But that is not a documented install method, and I haven't
heard of anyone using it. The recommended way is to run straight from git.
For a bit repo like f-droid.org, it makes sense to standardize on a single
format for metadata files. This adds support for enforcing a single data
format, or a reduced set of data formats. So f-droid.org would run like
this if it changed to YAML:
accepted_formats = ['txt', 'yaml']
Then once everything was converted to YAML, it could look like this:
accepted_formats = ['yaml']
In order to prevent confusion caused by multiple metadata files for a given
app, fdroid will exit with an error if it finds any app metadata file with
the same package ID as one that has already been parsed.
YAML is a format that is quite similar to the .txt format, but is a
widespread standard that has editing modes in popular editors. It is also
easily parsable in python.
The .pickle for testing is a lightly edited version of the real metadata
for org.videolan.vlc:
* comments were removed
This puts all of the needed post parsing checks on the metadata into a
single method that is used by all parsing methods (.txt, JSON, XML, YAML).
This provides the single place to normalize the internal representation of
the metadata.
It would be good to also change the internal representation to use more Python
bool/int types so that less post parsing is needed for JSON, XML, and YAML.
The SMSSecure test .pickle was changed to account for the use of lstrip()
and rstrip() on all 'script' types.
This also changes the example JSON to use ints for versionCodes
While the current text metadata format is good for human readability and
editability, it is difficult to produce and parse using code. XML is a
widespread standard format for easy automatic parsing and creating, while
having decent human readability.
The .pickle for testing is a lightly edited version of the real metadata
for net.osmand.plus:
* comments were removed
* "NonFreeNet" was added as an AntiFeature
The AntiFeatures metadata is a comma-separated list of tags, like
Categories, so it should also be stored internally as a list. This makes
parsing XML and JSON easier.
The test cases' .pickle files look like they change a lot, but they really
don't, its only the change of default AntiFeatures value from None to []
The .pickle was created by dumping the output from parsing the current .txt
metadata for org.adaway. The JSON started from that pickle dump, but was
then hand edited to be more proper JSON, e.g. using boolean values.
This is a test to cover future modifications of the .txt metadata parsing.
The pickle file was generated by just dumping the current parsed metadata,
so this test will always succeed if the parsing is not changed.
This addresses the discussion in !64https://gitlab.com/fdroid/fdroidserver/merge_requests/64
Sometimes, buildToolsVersion is a kind of gradle macro call, and other
times it is a variable assignment. This regsub pattern now handles both of
those cases.
