When making code changes related to the metadata parsing, it is useful to
see how the internal format has changed by seeing the differences in the
dump files. Those files are currently in the binary .pickle format. This
just straight converts them to YAML, which is a text format, so that normal
diff tools work to see changes.
The dump files are named .yaml instead of .yml since .yml is used for hand-
edited YAML files for fdroiddata/metadata, while these dump files here are
a human readable form of a Python pickle.
JSON and YAML are very closely related, so supporting both of them is
basically almost no extra work. Both are also closely related to how
Python works with dicts and pickles. XML is a very different beast, and its
not popular for this kind of thing anyway, so just purge it.
This allows a source repo to include a complete metadata file so that it
can be built directly in place using `fdroid build`. If that app is then
included in fdroiddata, it will first load the source repo type and URL
from fdroiddata, then read .fdroid.yml if it exists, then include the rest
of the metadata as specified in fdroiddata, so that fdroiddata has
precedence over the metadata in the source code.
This lets `fdroid build` apps without having a whole fdroiddata setup, but
instead just directly in place in the source code. This also lets devs
optionallu maintain the fdroid metadata as part of their app, rather than
in fdroiddata without loosing any control. This should make it easier to
spread around the maintenance load.
Something like `gradle: yes` in YAML will be parsed as a boolean, since
'yes' is officially defined as a boolean true in YAML. For metadata fields
that need to be lists, this needs to be converted. Same goes for a single
string like `gradle: customFlavor`.
This makes sure there is a GPG signature on any file that is included in
the repo, including APKs, OBB, source tarballs, media files, OTA update
ZIPs, etc. Having a GPG signature is more important on non-APK files since
they mostly do not have any signature mechanism of their own.
This also adds basic tests of adding non-APK/OBB files to a repo with
`fdroid update`.
closes#232
This makes it so that the final build product can be specified in output=
and it'll work no matter if its an APK or not. This was developed around
the case of building the OTA update.zip for the Privileged Extension. It
should work for any build process in theory but it has not yet been tested.
https://gitlab.com/fdroid/privileged-extension/issues/9
It is now possible for the server operator to specify lists of apps that
must be installed or deleted on the client (aka "push installs). If
the user has opted in, or the device is already setup to respond to
these requests, then fdroidclient will automatically install/delete
the packageNames listed. This is protected by the same signing key
as the app index metadata.
It generates single XML elements with the data set in the attributes. This
keeps the XML compact and easily extensible, e.g. for adding versionCode,
signingKey, etc as attributes:
<install packageName="com.fsck.k9"/>
<install packageName="at.bitfire.davdroid"/>
<delete packageName="com.facebook.orca"/>
Copyright: 2016 Blue Jay Wireless
Signed-off-by: Hans-Christoph Steiner <hans@eds.org>
closes#177
`fdroid update` should be able to handle any valid filename (hopefully
aapt doesn't barf on them). To handle that, the environment where the
shell commands are run in needs to have a UTF-8 locale set. If LANG is
not set, things default to ASCII and UTF-8 filenames fail.
This also renames test APK with lots of Unicode chars as a test case.
closes#167
Also, remove jdk7 as it will become unused. We added jdk8 for
retrolambda, and now that we will use jdk8 as the default, jdk7 is
unnecessary as retrolambda can work fine with just jdk8.
This removes it from the buildserver, and the new CI image also only has
jdk8 from jessie-backports.
Fixes#185.
This replaces the current default behavior of always forcing the
build_tools version and allows the user to set build-tools forcing in
config.py.
closes#147
It always wants to install packages into /usr/lib/python3.4/site-packages
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 295, in run
requirement_set.install(install_options, global_options, root=options.root_path)
File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install
requirement.install(install_options, global_options, *args, **kwargs)
File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install
self.move_wheel_files(self.source_dir, root=root)
File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in move_wheel_files
pycompile=self.pycompile,
File "/usr/lib/python3/dist-packages/pip/wheel.py", line 214, in move_wheel_files
clobber(source, lib_dir, True)
File "/usr/lib/python3/dist-packages/pip/wheel.py", line 176, in clobber
os.makedirs(dest)
File "/usr/lib/python3.4/os.py", line 237, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.4/site-packages'
.java .gradle and XML files all can use any encoding. Most code is ASCII,
but authors' names, etc. can easily be non-ASCII. UTF-8 is by far the most
common file encoding. While UTF-8 is the default encoding inside the code
in Python 3, it still has to deal with the real world, so the encoding
needs to be explicitly set when reading and writing files. So this switches
fdroidserver to expect UTF-8 instead of ASCII when parsing these files. For
now, this commit means that we only support UTF-8 encoded *.java, pom.xml
or *.gradle files. Ideally, the code would detect the encoding and use the
actual one, but that's a lot more work, and its something that will not
happen often. We can cross that bridge when we come to it.
One approach, which is taken in the commit when possible, is to keep the
data as `bytes`, in which case the encoding doesn't matter.
This also fixes this crash when parsing gradle and maven files with
non-ASCII chars:
ERROR: test_adapt_gradle (__main__.BuildTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/tests/build.TestCase", line 59, in test_adapt_gradle
fdroidserver.build.adapt_gradle(testsdir)
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/fdroidserver/build.py", line 445, in adapt_gradle
path)
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/fdroidserver/common.py", line 188, in regsub_file
text = f.read()
File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 9460: ordinal not in range(128)
Though the YAML people recommend .yaml for the file extension, in Android
land it seems clear that .yml has won out:
* .travis.yml
* .gitlab-ci.yml
* .circle.yml
* Ansible main.yml
The start up sequence of processes that are based on the .fdroid.* metadata
is a bit different, so this ensures that the environment variables get
properly initialized in all cases.
This also creates a single function where the environment is set. Before
it was being set in multiple places across multiple files.
serverwebroot has long supported uploading to multiple servers, this bit of
metadata communicates those official mirrors to the client so that it can
automatically do something useful with that information.
closes#14https://gitlab.com/fdroid/fdroidserver/issues/14
Using the same JDK throughout should prevent weird bugs where a setup might
use Java8's jarsigner and Java7's keytool. This also allows the user to
set java_paths and have jarsigner and keytool used from that specified JDK.
This incorporates almost all of the patch that is in the Debian package
that forces fdroidserver to use the default JDK on that Debian release.
closes#93https://gitlab.com/fdroid/fdroidserver/issues/93
Only keep lists in metadata files in the json format, since they don't
support multiline strings that are readable.
This makes the internal code easier, and a bit faster.
This simplifies usage, goes from
build['flag']
to
build.flag
Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.
As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.
While at it, unify "build", "thisbuild", "info", "thisinfo", etc into
just "build".
This simplifies usage, goes from
app['Foo']
to
app.Foo
Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.
As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.
Don't log and exit in an inner metadata function. Handle it at a higher
level and do a proper exception. This also avoids unnecessary passing of
apps all around.
