libcloud has been flaky for years, but it is also widely available and is
more pythonic. So this adds the option for using `s3cmd sync` if it is
available. `s3cmd sync` acts like rsync for uploading to S3. So this code
emulates the rsync uploading code in `fdroid server update`.
Instead of each time deleting the whole local git-mirror repo, and
recreating it, this just deletes the .git/ dir, then lets the rsync in
_local_sync() handle deleting anything that should no longer be in the repo.
This option was not hooked up at all, and does not make sense as a command
line argument. It should just be a config.py item. In that case, the
presence of config.py marks the current dir as a repo, so there is no
longer a need to test for a dir called repo/ as a safety. This makes the
setup easier, since sync_from_localcopy() now creates repo/ for the user.
Since `fdroid server update` is the place where all uploads to servers
happens, it makes sense to also handle the git push for the binary
transparency log here instead of `fdroid btlog`
This makes make_index_v1() support `fdroid update --pretty`, then also uses
pretty output for the binary transparency logs, so that the git history has
nice, readable diffs between commits.
This complements the binary transparency logging that happens on the
server side !226. Anyone can set up an efficient tracker of any
F-Droid repo which stores all index files that it sees. It uses HEAD
requests and ETag checking to be as efficient as possible, so that
this can be automatically run at a frequent pace.
This was failing on environments that did not have any LANG or LC_* locale
variables set. This is a valid setup, and is common in headless setups, so
it needs to be handled.
This also adds a new pass of the test suite without the locale env vars set
so that this situation is also tests on gitlab-ci, not only gpjenkins.
The error this caused was:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-18: ordinal not in range(128)
In 6c2cf2ccdd, the names of some essential
data bits changed. If those names are not in the tmp/apkcache, then
`fdroid update` shouldn't crash but instead just ignore that cache entry.
tmp/apkcache should be deleted since the metadata version was bumped, but
I guess that does not always happen.
With the binary transparency log stored in git, it makes that more readable
and as small as possible if the index files are fully sorted. That will
reduce the differences between two copies of an index file to a bare
minimum.
If in the future we implement some kind of binary diff transfer, this will
also help there.
Google has their own utility for verifying APK signatures on a desktop
machine since Java's jarsigner is bad for the task. For example, it
acts as if an unsigned APK validates. And to check whether an APK is
unsigned using jarsigner is difficult.
apksigner also does the v2 signatures, so it will have to be used
eventually anyway. It is already in Debian/stretch and can be
available in jessie-backports if need be.
https://android.googlesource.com/platform/tools/apksighttps://packages.debian.org/apksigner
There are two SHA1 git commit IDs that fdroidserver includes in the builds
it makes: fdroidserverid and buildserverid. Originally, these were inserted
into AndroidManifest.xml, but that makes the build not reproducible. So
instead they are included as separate files in the APK's META-INF/ folder.
If those files exist in the signed APK, they will be part of the signature
and need to also be included in the unsigned APK for it to validate.
This makes the jarsigner the ultimate and only judge of whether two APKs
match. This is the best tool since APK signatures are jar signatures. This
should be eventually updated to use the official Android APK signing tool
called apksigner.
https://android.googlesource.com/platform/tools/apksig/
The ZIP format allows multiple entries with the exact same filename, and on
top of that, it does not allow deleting or updating entries. To make the
`fdroid verify` procedure failsafe, it needs to create a new temporary APK
that is made up on the contents of the "unsigned APK" and the signature
from the "signed APK". Since it would be possible to give a signed APK as
in the unsigned one's position, `fdroid verify` was not able to update the
signature since it was just adding the new signature to the end of the ZIP
file. When reading a ZIP, the first entry is used.
This is a bit different than index.jar: instead of their being index.xml
and index_unsigned.jar, the presense of index-v1.json means that there is
unsigned data. That file is then stuck into a jar and signed by the
signing process. index-v1.json is never published to the repo. It is
included in the binary transparency log, if that is enabled.
If there are graphics files that are in repo/ but there is no metadata for
the associated app, then do not include the graphics in the index. Issue a
warning about this.
This looks for standard graphics assets in two standardized locations, one
in metadata/ subdirs and another in fastlane's standard dirs in the app's
source repo. When it finds them, it copies them into the repo, where they
will then be included in the index for use by fdroidclient. The images in
the metadata/ folder of fdroiddata take precendence over the files in the
app's source repo.
So like this for including graphics in fdroiddata:
metadata/packageName/locale/filename.(png|jpg|jpeg)
for example:
metadata/org.videolan.vlc/en-US/featureGraphic.png
or
metadata/info.guardianproject.ripple/zh-CN/phoneScreenshots/screenshot1.png
Including graphics in fdroiddata would be optional. The prefered way to get
graphics into the repo would be for the files to be in the git repo in a
standard location. This fastlane layout is currently supported:
https://github.com/fastlane/fastlane/blob/1.109.0/supply/README.md#images-and-screenshots
This just makes it easier for people writing build recipes. Rewriting will
output a list of strings as well.
The test index.xml and categories.txt are updated to include the new number
categories, and the changed CurrentVersionCode to 2147483647 (MAX_VALUE)
This syncs up the field names between the fdroiddata .yml files, the keys
used in the implementation in fdroidserver, the index data format, and the
final data structures in fdroidclient. This makes it easier for devs to
follow, and makes the Jackson parsing library automatically handle
converting the data from the index file to Java instances.
This bumps the metadata version since the apkcache will have to be
discarded.
Here are the name changes:
* apkname --> apkName
* id --> packageName
* sha256 --> hash
* version --> versionName
* versioncode --> versionCode
tests/repo/index.xml was changed only to bump the metadata version
from 17 to 18.
Using datetime instances as the internal format makes it much easier to
convert between the formats needed for index.xml and index-v1. apkcache
still uses time tuples and known_apks.txt still uses the ISO date.
Python encode/decode libs work directly with dicts, so the internal dict
can just be passed directly to any of these libs (pyyaml, pyjson, msgpack,
simplejson, etc). This still generates the exact same index.xml as before.
This converts the internal format for the repo timestamp to a datetime
instance, which can be easily converted to UNIX time in seconds for XML
and UNIX time in milliseconds for the new index formats. UNIX time in
milliseconds is directly serialized into a java.util.Date instance by
Jackson.
A Binary Transparency Log is a append only log of all binaries published by
a repo. This is useful for people to find whether the binary they have
matches what F-Droid has published, and also makes it more difficult for
the published history to be changed without notice, or for a server to give
specific users custom malware binaries.
https://www.eff.org/deeplinks/2014/02/open-letter-to-tech-companies
FDroidPopen is used for running many commands - from git to gradle to
custom commands via flags like build=. When any of these invoke calls to
custom build systems or upstream programs/scripts, it's not safe to
assume that the output will be utf8.
Unfortunately, this currently leads to crashes and failed builds:
ERROR: Could not build app org.kiwix.kiwixmobile due to unknown error: Traceback (most recent call last):
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 1155, in main
options.onserver, options.refresh):
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 951, in trybuild
build_local(app, build, vcs, build_dir, output_dir, srclib_dir, extlib_dir, tmp_dir, force, onserver, refresh)
File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 648, in build_local
p = FDroidPopen(['bash', '-x', '-c', cmd], cwd=root_dir)
File "/home/vagrant/fdroidserver/fdroidserver/common.py", line 1786, in FDroidPopen
result.output = result.output.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 5397290: invalid start byte
One way to fix this would be to use one of the python libraries that
guess an encoding. But a much safer option is to tell the decode method
to ignore non-utf8 bytes, as opposed to crashing on them.
Since it is now possible to build and include arbitrary files, like OTA
update ZIP files, the update procedure needs to look for non-APK files that
match the packageName_versionCode pattern of fdroid-generated files.
!193
admin#14
privileged-extension#9
`fdroid build` will generate source tarballs, and now with support for
adding any file to a repo, we need to explicitly ignore the fdroid-
generated source tarballs. If people want to include source tarballs in a
repo, they still can, as long as that source tarball doesn't use the
`fdroid build` tarball naming scheme.
This forces <uses-permission maxSdkVersion=""> to be an integer in the
internal dict, and forces it to have no decimal point in XML. Having it as
an integer in the internal dict means data will pass directly through to
the fdroidclient Apk instance, where it is ultimately an integer.
fdroidclient should handle no minSdkVersion fine, since it just parses the
text in <sdkver></sdkver> as an int, and uses a default value if there is
an Exception.
A .asc or .sig file is a detached PGPG signature, `fdroid gpgsign`
generates them. It makes no sense for them to be ever treated as a file
for distribution.
This also adds to forgotten forms of index files.
This lets index-v1 be parsed directly into class instances because the
field/instance var names match exactly. The original index v0 element
must retain the 'lastupdated' name for backwards compatibility.
If a group of items are enclosed in {}, then that will be a Python set,
which does not preserve order. To preserve order, the data must be either
a tuple () or list [].
In the future, we should have better internal datatypes for this stuff,
i.e. instead of gradle: ['yes'] for True, actually use a boolean. For now,
make the YAML and JSON metadata produce the same internal data as .txt.
Like with the App class in the commit before, this makes it a lot
easier to work with this data when converting between the internal
formats and external formats like YAML, JSON, MsgPack, protobuf, etc.
The one unfortunate thing here is Build.update. It becomes
dict.update(), which is a method not an attribute.
build.get('update') or build['update'] could be used, but that would
be oddly inconsistent. So instead the field is renamed to
'androidupdate', except for in the .txt v0 metadata files. This better
describes what field does anyway, since it runs `android update`.
Build.update is only referenced in two places right next to each other
for the ant builds, so this change still seems worthwhile.
Python is heavily based on its core data types, and dict is one of the more
important ones. Even classes are basically a wrapper around a dict. This
converts metadata.App to be a subclass of dict so it can behave like a dict
when being dumped and loaded. This makes its drastically easier to use
different data formats for build metadata and for sending data to the
client. This approach will ultimately mean we no longer have to maintain
custom parsing and dumping code.
This also means then that the YAML/JSON field names will not have spaces in
them, and they will match exactly what it used as the dict keys once the
data is parsed, as well as matching exactly the instance attribute names:
* CurrentVersion: 1.2.6
* app['CurrentVersion'] == '1.2.6'
* app.CurrentVersion == '1.2.6'
Inspired by:
https://goodcode.io/articles/python-dict-object/
This makes it a lot easier to work with Build instances with parsing and
dumping libraries, since they expect only core Python types (dict, list,
tuple, str, etc)
