FDroidPopen is used for running many commands - from git to gradle to
custom commands via flags like build=. When any of these invoke calls to
custom build systems or upstream programs/scripts, it's not safe to
assume that the output will be utf8.
Unfortunately, this currently leads to crashes and failed builds:
ERROR: Could not build app org.kiwix.kiwixmobile due to unknown error: Traceback (most recent call last):
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 1155, in main
options.onserver, options.refresh):
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 951, in trybuild
build_local(app, build, vcs, build_dir, output_dir, srclib_dir, extlib_dir, tmp_dir, force, onserver, refresh)
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 648, in build_local
p = FDroidPopen(['bash', '-x', '-c', cmd], cwd=root_dir)
File "/home/vagrant/fdroidserver/fdroidserver/common.py", line 1786, in FDroidPopen
result.output = result.output.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 5397290: invalid start byte
One way to fix this would be to use one of the python libraries that
guess an encoding. But a much safer option is to tell the decode method
to ignore non-utf8 bytes, as opposed to crashing on them.
Since it is now possible to build and include arbitrary files, like OTA
update ZIP files, the update procedure needs to look for non-APK files that
match the packageName_versionCode pattern of fdroid-generated files.
!193
admin#14
privileged-extension#9
`fdroid build` will generate source tarballs, and now with support for
adding any file to a repo, we need to explicitly ignore the fdroid-
generated source tarballs. If people want to include source tarballs in a
repo, they still can, as long as that source tarball doesn't use the
`fdroid build` tarball naming scheme.
This forces <uses-permission maxSdkVersion=""> to be an integer in the
internal dict, and forces it to have no decimal point in XML. Having it as
an integer in the internal dict means data will pass directly through to
the fdroidclient Apk instance, where it is ultimately an integer.
fdroidclient should handle no minSdkVersion fine, since it just parses the
text in <sdkver></sdkver> as an int, and uses a default value if there is
an Exception.
A .asc or .sig file is a detached PGPG signature, `fdroid gpgsign`
generates them. It makes no sense for them to be ever treated as a file
for distribution.
This also adds to forgotten forms of index files.
This lets index-v1 be parsed directly into class instances because the
field/instance var names match exactly. The original index v0 element
must retain the 'lastupdated' name for backwards compatibility.
If a group of items are enclosed in {}, then that will be a Python set,
which does not preserve order. To preserve order, the data must be either
a tuple () or list [].
Since https://gitlab.com/fdroid/ci-test-app is a separate git repo, things
with incompatible changes could get out of sync. Therefore, this test
should specify which git commit is runs against.
For example, the .fdroid.yml file is still a moving target. Just now, the
keys had the spaces removed as part of this MR.
In the future, we should have better internal datatypes for this stuff,
i.e. instead of gradle: ['yes'] for True, actually use a boolean. For now,
make the YAML and JSON metadata produce the same internal data as .txt.
This requires manually running it. I suppose it would be possible to
include a snapshot of the dumped internal representation for each release,
then make the tests run automatically against that. Right now, the dump is
17megs of YAML. Seems large to include in this git repo.
Like with the App class in the commit before, this makes it a lot
easier to work with this data when converting between the internal
formats and external formats like YAML, JSON, MsgPack, protobuf, etc.
The one unfortunate thing here is Build.update. It becomes
dict.update(), which is a method not an attribute.
build.get('update') or build['update'] could be used, but that would
be oddly inconsistent. So instead the field is renamed to
'androidupdate', except for in the .txt v0 metadata files. This better
describes what field does anyway, since it runs `android update`.
Build.update is only referenced in two places right next to each other
for the ant builds, so this change still seems worthwhile.
Python is heavily based on its core data types, and dict is one of the more
important ones. Even classes are basically a wrapper around a dict. This
converts metadata.App to be a subclass of dict so it can behave like a dict
when being dumped and loaded. This makes its drastically easier to use
different data formats for build metadata and for sending data to the
client. This approach will ultimately mean we no longer have to maintain
custom parsing and dumping code.
This also means then that the YAML/JSON field names will not have spaces in
them, and they will match exactly what it used as the dict keys once the
data is parsed, as well as matching exactly the instance attribute names:
* CurrentVersion: 1.2.6
* app['CurrentVersion'] == '1.2.6'
* app.CurrentVersion == '1.2.6'
Inspired by:
https://goodcode.io/articles/python-dict-object/
This makes it a lot easier to work with Build instances with parsing and
dumping libraries, since they expect only core Python types (dict, list,
tuple, str, etc)
The original index.xml format needs to stay around for backwards
compatibility, but we shouldn't touch it anymore once the new format is in
place. This is a test to make sure `fdroid update` can still generate the
correct XML.
install_list and uninstall_list should be tuples or lists in order to
ensure that the order is preserved.
These tests also check that the added and lastupdated dates are
working correct, based on the dates in tests/stats/known_apks.txt. I
could see no useful way to test the timestamp, it is just hardcoded
using a regexp search-and-replace. Running these tests manually might
require deleting tmp/apkcache.
