There are so many possible installation paths for Python modules, it has
been very hard to even find and test them all. This adds a fallback option
if the examples dir cannot be found. A repo can work without an icon or
the example config.py.
This removes the fake assumption that the icon can be a full path in the
config.py. While the path was being properly passed through to the index
file, the file was never copied properly into place nor rsynced to the web
server.
The .txt format was the last place where the lowercase "builds" was used,
this converts references everywhere to be "Builds". This makes it possible
to load metadata YAML files with any YAML parser, then have it possible to
use fdroidserver methods on that data, like metadata.write_metadata().
The test files in tests/metadata/dump/*.yaml were manually edited by cutting
the builds: block and putting it the sort order for Builds: so the contents
should be unchanged.
```
sed -i \
-e 's/app\.builds/app.get('Builds', \[\])/g' \
-e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
-e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
-e "s/app\.get(Builds, \[\])/app.get('Builds', \[\])/g" \
-e "s/app\.get('Builds', \[\])\.append/app\['Builds'\].append/g" \
-e "s/app\['builds'\]/app.get('Builds', [])/g" \
*/*.*
```
Ideally, an fdroid repo should be running from a clean git repo, so that
all changes are tracked in git. This change is useful in seeing which
changes and/or files are not in git. If there are modified files, the
dirty flag will be set, so this info can help debugging that.
The key fingerprint should be only hex digits, everything else can be
discarded. That makes it easy to use this function various fingerprint
formats, including the common, human-readable forms spaces between pairs or
quartets.
Virgin-islands-british_centralamerica_2.obf.zip is 1MB, while
Norway_bouvet_europe_2.obf.zip is 12KB. This file gets copied a lot in the
test runs so it adds up fast.
Back when fdroidserver was built around aapt, that was needed to
guarantee that a compatible version of aapt was used. Now, aapt is
only optionally used for getting the APK ID, so this was just
complicating maintenance.
When using fdroidserver methods as an API, the full setup might not
have taken place. `app` instances can always just be a dict, the App
class is mostly just a typing shortcut. This is incremental, it only
affects a couple of functions in fdroidserver/update.py.
None of the config options in config.py require Python code. YAML is a
common config data format, and it is also used for build metadata. It is
also much safer to use since it can be pure data, without anything
executable in it. This also reduces the attack surface of the fdroid
process by eliminating a guaranteed place to write to get code executed.
With config.py, any exploit that can get local write access can turn that
into execute access by writing to the config.py, then cleaning up after
itself once it has what it needs. Switching to YAML removes that vector
entirely.
Also, this removes the config_file argument. It is not used in either
fdroidserver or repomaker. Also, it probably wouldn't work since so
much of the code assumes that the current working dir is the root of the
repo.
Up until now, the buildserver has not included androguard. Since a
good version of androguard (v3.3.3+) is included in stretch-backports
and the buildserver is already setup to use stretch-backports, this
sets up the buildserver with androguard.
closes#627
There must be at least one APK available for this test suite to work, for
example, this test:
grep -F '<application id=' repo/index.xml
This can't be easily implemented using an env vir beccause the while
loop is running in a pipe, so a different process.
copy_apks_into_repo is used with throwaway tmp dirs, so the stamp file
should work well.
The `force_build_tools` config option was added a long time ago to
brute force the _build-tools_ version by trying to replace the value
in `build.gradle` files. This is never something that should be used
in production, since the app's build metadata should specify this kind
of thing. And now that we're moving towards _androguard_ for
everything except fdroid build and fdroid publish, _build-tools_ will
no longer even be used in the other commands.
This makes apksigner a hard requirement of the signing procedure.
We'll first try to find a globally installed version from PATH and if
that's not available fall back to using a version from build-tools.
Future TODO: always sign with apksigner, blocked on signature transplant
support for apksigv2/v3
Closesfdroid/fdroidserver#634Closesfdroid/fdroidserver#827
publish is currently not reusable from other modules as everything is
happening in main. It's also not testable from python unittests.
There's already a function for getting the key_alias, so we can use
that.
Introduce tests for the split out functions.
Previously this was magically capturing the apps dict when passing it around as a
function. This also moved the code to the metadata module.
Add a test doing read_metadata where the linkresolver is used. This
happens when the apps we read have a [[app.id]] link to another app.
Liberapay was originally included using a numeric ID, since they had
not yet finalized the public URLs. Now it is a username. So this
logic prefers the username in Liberapay: field, and keeps the old
LiberapayID: to ease migration. LiberapayID: will not override
Liberapay:. Clients are expected to prefer Liberapay: over LiberapayID:
GitHub has specified FUNDING.yml, a file to include in a git repo for
pointing people to donation links. Since F-Droid also points people
to donation links, this parses them to fill out Donate:
and OpenCollective:. Specifying those in the metadata file takes
precedence over the FUNDING.yml. This follows the same pattern as how
`fdroid update` includes Fastlane/Triple-T metadata. This lets the
git repo maintain those specific donations links themselves.
https://help.github.com/en/articles/displaying-a-sponsor-button-in-your-repository#about-funding-files
The test file was generated using:
```python
import os, re, yaml
found = dict()
for root, dirs, files in os.walk('.'):
for f in files:
if f == 'FUNDING.yml':
with open(os.path.join(root, f)) as fp:
data = yaml.safe_load(fp)
for k, v in data.items():
if k not in found:
found[k] = set()
if not v:
continue
if isinstance(v, list):
for i in v:
found[k].add(i)
else:
found[k].add(v)
with open('gather-funding-names.yaml', 'w') as fp:
output = dict()
for k, v in found.items():
output[k] = sorted(v)
yaml.dump(output, fp, default_flow_style=False)
```
closes#465
This script generated gradle-maven-blocks.yaml:
```python
import os
import re
import yaml
pat = re.compile(r'\smaven\s*{[^}]+}')
finds = set()
for root, dirs, files in os.walk('.'):
for f in files:
if '.gradle' in f:
with open(os.path.join(root, f), errors='surrogateescape') as fp:
contents = fp.read()
for m in pat.findall(contents):
finds.add(m)
with open('finds.yaml', 'w') as fp:
yaml.dump(sorted(finds), fp, default_flow_style=False)
```
This converts float/int to string for things like commit: or versionName:.
For versionCode, which must be an integer, it throws an exception if the
data is any other type.
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot
This is a second attempt with tests for how `fdroid build` calls the
scanner functions. closes#771. It was previously merged in !748 then
reverted in 68c072c72e
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot
import is a strict keyword in Python, so it is not possible to import a
module called 'import', even with things like:
* import fdroidserver.import
* from fdroidserver import import
These days, the location that overrides all the others is in the android{}
block of the build.gradle file that loads the com.android.application
plugin. So this should be the preferred place to read these values.
test files GPL licensed: https://github.com/Integreight/1Sheeld-Android-App
http://example.org/index-v1.jar now returns the HTTP header
"Content-Encoding: gzip" but then the reply is plain HTML. That
triggers a ContentDecodingError instead of an HTTPError, so this
changes the test to success on any RequestsException.
This makes it so running `../fdroid update --nosign --pretty` in tests/ no
longer creates a diff in the tests/*/index* files. It matches the order
set in tests/run-tests.
These entries are hardcoded as a single line in all the app stores, so
newlines should be stripped to get the data simple to use. This is in
contrast with the on-disk format for Fastlane and Triple-T, which includes
a newline in the title.txt and short_description.txt files. I think all
files in those systems are normalized to end in a newline.
6d0b1bbe6fae0909683f2c6a154515bc4bfcb674 didn't handle the
allow_disabled_algorithm case at all, so we add it back.
This additionally fixes a (previously existing) bug where setting
allow_disabled_algorithms to True didn't move apks back from archive to
repo. Introduce a new test for this.
The disabled_algorithm archiving logic is still all over the place so
ideally that needs a future refactor.
Python PIL is not so tolerant, so bad EXIF causes crashes:
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2088, in main
insert_localized_app_metadata(apps)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 978, in insert_localized_app_metadata
_strip_and_copy_image(os.path.join(root, f), destdir)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 754, in _strip_and_copy_image
in_image = Image.open(fp)
File "/usr/lib/python3/dist-packages/PIL/Image.py", line 2687, in open
% (filename if filename else fp))
OSError: cannot identify image file <_io.BufferedReader name='build/org.sw24softwares.starkeverben/fastlane/metadata/android/en-US/images/featureGraphic.png'>
Using a filename based on the hash of the contents means that the caching
algorithms for fdroidclient and browsers can safely cache the file forever
using the filename, since this guarantees that the contents will never
change for a given filename.
This does not cover screenshots, only icon.png, featureGraphic.png,
tvBanner.png, and promoGraphic.png.
fdroidserver#689
fdroid-website!453
This was done with much help from @uniqx. This is the first level of
supporting APK Signatures v1, v2, and v3. This is enough to include
APKs with any combo of v1/v2/v3 signatures. For this to work at all,
apksigner and androguard 3.3.3+ must be installed.
closes#399
699b3e4c69 got it wrong for targetSdkVersion.
Also, one confusing thing is that aapt outputs "sdkVersion: '3'" for
com.politedroid_3.apk but no "sdkVersion:" for no.min.target.sdk_987.apk.
F-Droid never really supported running on android-1 or android-2, so it
seems pointless to debug support for them.
androguard parses the whole APK before handing the instance back, this uses
the primitives to just find the <application android:debuggable=""> value,
then stop parsing.
#557
Normally, androguard parses the entire APK before it is possible to get any
values from it. This uses androguard primitives to only attempt to parse
the AndroidManifest.xml, then to quit as soon as it gets what it needs.
This greatly speeds up the parsing (1 minute vs 60 minutes).
fdroid/fdroidserver#557
pickle can serialize executable code, while JSON is only ever pure data.
The APK cache is only ever pure data, so no need for the security risks of
pickle. For example, if some malicious thing gets write access on the
`fdroid update` machine, it can write out a custom tmp/apkcache which would
then be executed. That is not possible with JSON.
This does just ignore any existing cache and rebuilds from scratch. That is
so we don't need to maintain pickle anywhere, and to ensure there are no
glitches from a conversion from pickle to JSON.
closes#163
Debian's autopkgtest defaults to considering any output on stderr as a sign
that the tests failed. it is simple to make this achieve that standard, so
it seems worth it for now at least.
