The `force_build_tools` config option was added a long time ago to
brute force the _build-tools_ version by trying to replace the value
in `build.gradle` files. This is never something that should be used
in production, since the app's build metadata should specify this kind
of thing. And now that we're moving towards _androguard_ for
everything except fdroid build and fdroid publish, _build-tools_ will
no longer even be used in the other commands.
This makes apksigner a hard requirement of the signing procedure.
We'll first try to find a globally installed version from PATH and if
that's not available fall back to using a version from build-tools.
Future TODO: always sign with apksigner, blocked on signature transplant
support for apksigv2/v3
Closesfdroid/fdroidserver#634Closesfdroid/fdroidserver#827
publish is currently not reusable from other modules as everything is
happening in main. It's also not testable from python unittests.
There's already a function for getting the key_alias, so we can use
that.
Introduce tests for the split out functions.
Previously this was magically capturing the apps dict when passing it around as a
function. This also moved the code to the metadata module.
Add a test doing read_metadata where the linkresolver is used. This
happens when the apps we read have a [[app.id]] link to another app.
Liberapay was originally included using a numeric ID, since they had
not yet finalized the public URLs. Now it is a username. So this
logic prefers the username in Liberapay: field, and keeps the old
LiberapayID: to ease migration. LiberapayID: will not override
Liberapay:. Clients are expected to prefer Liberapay: over LiberapayID:
GitHub has specified FUNDING.yml, a file to include in a git repo for
pointing people to donation links. Since F-Droid also points people
to donation links, this parses them to fill out Donate:
and OpenCollective:. Specifying those in the metadata file takes
precedence over the FUNDING.yml. This follows the same pattern as how
`fdroid update` includes Fastlane/Triple-T metadata. This lets the
git repo maintain those specific donations links themselves.
https://help.github.com/en/articles/displaying-a-sponsor-button-in-your-repository#about-funding-files
The test file was generated using:
```python
import os, re, yaml
found = dict()
for root, dirs, files in os.walk('.'):
for f in files:
if f == 'FUNDING.yml':
with open(os.path.join(root, f)) as fp:
data = yaml.safe_load(fp)
for k, v in data.items():
if k not in found:
found[k] = set()
if not v:
continue
if isinstance(v, list):
for i in v:
found[k].add(i)
else:
found[k].add(v)
with open('gather-funding-names.yaml', 'w') as fp:
output = dict()
for k, v in found.items():
output[k] = sorted(v)
yaml.dump(output, fp, default_flow_style=False)
```
closes#465
This script generated gradle-maven-blocks.yaml:
```python
import os
import re
import yaml
pat = re.compile(r'\smaven\s*{[^}]+}')
finds = set()
for root, dirs, files in os.walk('.'):
for f in files:
if '.gradle' in f:
with open(os.path.join(root, f), errors='surrogateescape') as fp:
contents = fp.read()
for m in pat.findall(contents):
finds.add(m)
with open('finds.yaml', 'w') as fp:
yaml.dump(sorted(finds), fp, default_flow_style=False)
```
This converts float/int to string for things like commit: or versionName:.
For versionCode, which must be an integer, it throws an exception if the
data is any other type.
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot
This is a second attempt with tests for how `fdroid build` calls the
scanner functions. closes#771. It was previously merged in !748 then
reverted in 68c072c72e
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot
import is a strict keyword in Python, so it is not possible to import a
module called 'import', even with things like:
* import fdroidserver.import
* from fdroidserver import import
These days, the location that overrides all the others is in the android{}
block of the build.gradle file that loads the com.android.application
plugin. So this should be the preferred place to read these values.
test files GPL licensed: https://github.com/Integreight/1Sheeld-Android-App
http://example.org/index-v1.jar now returns the HTTP header
"Content-Encoding: gzip" but then the reply is plain HTML. That
triggers a ContentDecodingError instead of an HTTPError, so this
changes the test to success on any RequestsException.
This makes it so running `../fdroid update --nosign --pretty` in tests/ no
longer creates a diff in the tests/*/index* files. It matches the order
set in tests/run-tests.
These entries are hardcoded as a single line in all the app stores, so
newlines should be stripped to get the data simple to use. This is in
contrast with the on-disk format for Fastlane and Triple-T, which includes
a newline in the title.txt and short_description.txt files. I think all
files in those systems are normalized to end in a newline.
6d0b1bbe6fae0909683f2c6a154515bc4bfcb674 didn't handle the
allow_disabled_algorithm case at all, so we add it back.
This additionally fixes a (previously existing) bug where setting
allow_disabled_algorithms to True didn't move apks back from archive to
repo. Introduce a new test for this.
The disabled_algorithm archiving logic is still all over the place so
ideally that needs a future refactor.
Python PIL is not so tolerant, so bad EXIF causes crashes:
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2088, in main
insert_localized_app_metadata(apps)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 978, in insert_localized_app_metadata
_strip_and_copy_image(os.path.join(root, f), destdir)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 754, in _strip_and_copy_image
in_image = Image.open(fp)
File "/usr/lib/python3/dist-packages/PIL/Image.py", line 2687, in open
% (filename if filename else fp))
OSError: cannot identify image file <_io.BufferedReader name='build/org.sw24softwares.starkeverben/fastlane/metadata/android/en-US/images/featureGraphic.png'>
Using a filename based on the hash of the contents means that the caching
algorithms for fdroidclient and browsers can safely cache the file forever
using the filename, since this guarantees that the contents will never
change for a given filename.
This does not cover screenshots, only icon.png, featureGraphic.png,
tvBanner.png, and promoGraphic.png.
fdroidserver#689
fdroid-website!453
This was done with much help from @uniqx. This is the first level of
supporting APK Signatures v1, v2, and v3. This is enough to include
APKs with any combo of v1/v2/v3 signatures. For this to work at all,
apksigner and androguard 3.3.3+ must be installed.
closes#399
699b3e4c69 got it wrong for targetSdkVersion.
Also, one confusing thing is that aapt outputs "sdkVersion: '3'" for
com.politedroid_3.apk but no "sdkVersion:" for no.min.target.sdk_987.apk.
F-Droid never really supported running on android-1 or android-2, so it
seems pointless to debug support for them.
androguard parses the whole APK before handing the instance back, this uses
the primitives to just find the <application android:debuggable=""> value,
then stop parsing.
#557
Normally, androguard parses the entire APK before it is possible to get any
values from it. This uses androguard primitives to only attempt to parse
the AndroidManifest.xml, then to quit as soon as it gets what it needs.
This greatly speeds up the parsing (1 minute vs 60 minutes).
fdroid/fdroidserver#557
pickle can serialize executable code, while JSON is only ever pure data.
The APK cache is only ever pure data, so no need for the security risks of
pickle. For example, if some malicious thing gets write access on the
`fdroid update` machine, it can write out a custom tmp/apkcache which would
then be executed. That is not possible with JSON.
This does just ignore any existing cache and rebuilds from scratch. That is
so we don't need to maintain pickle anywhere, and to ensure there are no
glitches from a conversion from pickle to JSON.
closes#163
Debian's autopkgtest defaults to considering any output on stderr as a sign
that the tests failed. it is simple to make this achieve that standard, so
it seems worth it for now at least.
Trying to reuse the pip test run is now a lot more work than just
writing it cutstom for each target platform.
Ubuntu/trusty does not have androguard 3.1.x nor aapt >= 26.x, so
using trusty's aapt will always have errors scraping some names from
APKs. This continues to use Google's binary.
`apt-get update` is now disabled by default by Travis, this adds it
back so that the latest files from the PPA are used.
Here's one example of an issue with pip on trusty:
https://github.com/requests/requests/issues/4006
pip3 install --quiet -e /home/travis/build/fdroidtravis/fdroidserver
Exception:
Traceback (most recent call last):
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2482, in _dep_map
return self.__dep_map
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2344, in __getattr__
raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python3.4/site-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python3.4/site-packages/pip/commands/install.py", line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python3.4/site-packages/pip/req.py", line 1266, in prepare_files
req_to_install.extras):
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2291, in requires
dm = self._dep_map
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2484, in _dep_map
self.__dep_map = self._compute_dependencies()
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2517, in _compute_dependencies
common = frozenset(reqs_for_extra(None))
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2514, in reqs_for_extra
if req.marker_fn(override={'extra':extra}):
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/_markerlib/markers.py", line 113, in marker_fn
return eval(compiled_marker, environment)
File "<environment marker>", line 1, in <module>
NameError: name 'platform_system' is not defined
For some reason, the parser stopped working intermittently, even
though the format has been the same since aapt 23 or earlier. Then
also, some of the test cases pointed to symlinks that were no longer
generated, and one test app now has a blank versionName.
Strange that this wasn't caught in the gitlab-ci runs. !484
FAIL: test_get_api_id_aapt (__main__.CommonTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./common.TestCase", line 578, in testA_get_api_id_aapt
self.assertEqual(versionName, vn)
AssertionError: '0.1' != "0.1' platformBuildVersionName='4.3.1-1425645"
- 0.1
+ 0.1' platformBuildVersionName='4.3.1-1425645
Instead of just crashing, first try to use the versionName as written in the
build metadata, otherwise just let it be blank. A blank versionName will
cause fdroidclient < 1.3 to crash. Blank versionNames are not allowed in
the .txt metadata format, only .yml.
closes#477closes#478
closes fdroidclient#1416
closes fdroidclient#1417
closes fdroidclient#1418
fdroiddata!3061
An APK (Netflix) was found to have the following icon filename:
\u2003\u2009\n.xml
This breaks the aapt dump parsing because it iterates line by line and
this filename goes across two lines. Consequently, icon_src will be
None (default value) when it is passed to the icons parser.
This code has never been used and contains some insecure uses of shell=True
Building Kivy apps should be done with the buildozer=yes method. The
buildozer method should probably be moved to a provisioner once that is in
place.
Generating a keystore is quite slow since it means a new RSA key is created.
That only needs to happen in the tests that check that it actually happened,
otherwise the test can just reuse the stored test keystore.
closes#432
These were both spamming the output with lots of confusing messages, even
when --verbose was not used. Jarsigner especially has confusing messages,
since it has warnings that do not pertain to APK signatures at all, like
the ones about timestamps and missing Certificate Authority.
closes#405
A full run of the test suite takes quite a bit of time. This removes one
of the 3 runs from the main 'tests' job, and puts it into the Fedora job.
That test run is mostly to make sure the setup.py and source tarball are
correctly, so that doesn't affect merge requests very often.
This also tests `pip install --user`, which was not really being tested
before.
This came about testing on OSX, where there are often multiple versions of
the JDK installed. This was choosing the oldest version. It should
choose the most recent version.
This works around the gradle 2.x bug where versions newer than 2.9 cannot
run things. This also then specifies the version of Gradle Android Plugin
that is included in Debian/stretch.
This fixes all the bugs I could find that prevented fdroid from
handling files with spaces in them. This is more important now that
fdroid supports random media files, and Repomaker
This filename has some messed up bytes related to bi-directional script
that is included (Left-to-Right and Right-to-Left). GNU/Linux always
interprets filenames as pure byte sequences. Windows and OSX store
filenames as Unicode strings. So on OSX, the invalid filename gets
converted to a valid name. That works fine, but the test fails because it
is compared to a file generated on Ubuntu, where it preserves the byte
sequence.
This includes an APK with a valid Unicode filename that includes
bi-directional script.
apksigner v0.7 (build-tools 26.0.1), Google made it require that the
AndroidManifest.xml was present in the archive before it verifies the
signature. So this needs to stick with the jarsigner hack for JARs.
There is a hardcoded template in update.py, and there is also the
possibility for the user to create a template.yml. This tests both of them
and cleans up the related code a bit.
#352!310
With the new localization support, the name/summary/description in the
metadata file becomes the global override. So most apps are not going to
have those fields present in their metadata file. This fixes the index.xml
generation to fall back to the localized versions of those fields when they
are not set in the metadata field.
https://forum.f-droid.org/t/what-has-happend-to-osmand
It turns out it is error prone to `git push` to a non-bare git repo. For
the offline signing machine, the git remote needs to be a regular git repo
in a directory on a thumbdrive so that once the thumbdrive is plugged into
an online machine, that git repo can be transferred to the online machine.
Since the mirror URLs are per repo section (repo/archive), the mirror URLs
must include the repodir at the end. This was missing for servergitmirrors
found by @cde when working on fdroidclient#35
For cases like the OpenVPN vuln that was recently announced, it is useful
for fdroiddata maintainers to be able to mark builds that have known
vulnerabilities.
The new policy is to move APKs with invalid signatures to the archive,
and only add those APKs to the archive's index if they have valid MD5
signatures.
closes#323closes#292
The original logic was checking keepversions against the len() of ALL the
APKs in the repo/archive. The correct thing is to check against the
number of APKs available for the given packageName/appid.
closes#166
`fdroid update` crashed for apps that only had screenshots but no graphics
or localized texts because destdir was not being set in that case. This
fixes that and adds a test case.
closes#320!286
If working with a random grabbag of APKs, there can be all sorts of
issues like corrupt entries in the ZIP, bad signatures, signatures that
are invalid since they use MD5, etc. Moving these two checks later means
that the APKs can be renamed still.
This does change how common.getsig() works. For years, it returned
None if the signature check failed. Now that I've started working
with giant APK collections gathered from the wild, I can see that
`fdroid update` needs to be able to first index what's there, then
make decisions based on that information. So that means separating
the getsig() fingerprint fetching from the APK signature verification.
This is not hugely security sensitive, since the APKs still have to
get past the Android checks, e.g. update signature checks. Plus the
APK hash is already included in the signed index.
With a generic file, the file name is the only guaranteed name metadata
field. So if the name is not specified in the metadata, then the name
is set to the filename. This changes that so that the file extension is
stripped from that generated name.
This is useful for parsing APK files, which can include packageName,
versionCode, and optionally 7 char signing key ID (i.e. <sig>).
This also can set the packageName and versionCoe for non APK files, so
that it is easy to assign them to metadata files, and to allow for
upgrades by setting the versionCode in the filename.
Really, it is the fdroidclient parser of index.xml that fails, due to the
hardcoded expectation that there will only ever be a single APK for any
given versionCode. We keep index.xml backwards compatible for old
clients, and use index-v1.json to support new things. Having multiple
APKs that have the same packageName and versionCode will break the client
v0.103.* since that version uses index-v1.json, but still has the hard-
coded database parsing stuff.
#153
uses the standard package.name_123.apk. If that exists, it appends the
shasum. If that exists, then its a duplicate, so its deleted. This should
help @SergeWinters with his 12,000 APKs.
There are many APKs out in the wild that claim to be the same app and
version and each other, but they are signed by different keys. fdroid
should be able to index these, and work with them. This supports having
the developer's signature via reproducible builds, random collections of
APKs like repomaker, etc.
This is some very messy logic built up since 2010. This will all go away
once we have a python3 version of androguard available.
The removed imports and `dir(APK)` is to silence pyflakes
closes#303
This is a little omission. keys that are used in metadata/*.yml all start
with an UpperCase letter, but in fdroidserver, index-v1.json, and
fdroidclient, it is all camelCase with lowercase first letter. The keys
from the 'localized' section are currently never in metadata/*.yml, so
these keys never get downcase. This change will break fdroidclient
versions that do not also have this change, but since we're in alpha, that
should be fine.
If support for a 'localized' section is added to metadata/*.yml, then the
keys there should probably be UpperCase CamelCase to match the other keys.
Fastlane Supply, Triple-T Gradle Play Publisher, and many app stores
include the possibility to specify a website for the author, as distinct
from the website for the app.
closes#204
This uses the "What's New" entry for the CurrentVersionCode and includes it
as the current WhatsNew metadata for the App class.
Things like fastlane supply and Google Play support a "What's New" entry
per-APK, but fdroidclient does not current use anything but the current
version of this data. Right now, it seems we probably only want to have
the latest WhatsNew in the index to save space.
In theory, we could make the WhatsNew data structure follow the structure
of fastlane/Play, but that would quite a bit of complexity for something
that might never be used.
fdroidclient#910
