This converts float/int to string for things like commit: or versionName:.
For versionCode, which must be an integer, it throws an exception if the
data is any other type.
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot
This is a second attempt with tests for how `fdroid build` calls the
scanner functions. closes#771. It was previously merged in !748 then
reverted in 68c072c72e
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot
import is a strict keyword in Python, so it is not possible to import a
module called 'import', even with things like:
* import fdroidserver.import
* from fdroidserver import import
These days, the location that overrides all the others is in the android{}
block of the build.gradle file that loads the com.android.application
plugin. So this should be the preferred place to read these values.
test files GPL licensed: https://github.com/Integreight/1Sheeld-Android-App
http://example.org/index-v1.jar now returns the HTTP header
"Content-Encoding: gzip" but then the reply is plain HTML. That
triggers a ContentDecodingError instead of an HTTPError, so this
changes the test to success on any RequestsException.
This makes it so running `../fdroid update --nosign --pretty` in tests/ no
longer creates a diff in the tests/*/index* files. It matches the order
set in tests/run-tests.
These entries are hardcoded as a single line in all the app stores, so
newlines should be stripped to get the data simple to use. This is in
contrast with the on-disk format for Fastlane and Triple-T, which includes
a newline in the title.txt and short_description.txt files. I think all
files in those systems are normalized to end in a newline.
6d0b1bbe6fae0909683f2c6a154515bc4bfcb674 didn't handle the
allow_disabled_algorithm case at all, so we add it back.
This additionally fixes a (previously existing) bug where setting
allow_disabled_algorithms to True didn't move apks back from archive to
repo. Introduce a new test for this.
The disabled_algorithm archiving logic is still all over the place so
ideally that needs a future refactor.
Python PIL is not so tolerant, so bad EXIF causes crashes:
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2088, in main
insert_localized_app_metadata(apps)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 978, in insert_localized_app_metadata
_strip_and_copy_image(os.path.join(root, f), destdir)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 754, in _strip_and_copy_image
in_image = Image.open(fp)
File "/usr/lib/python3/dist-packages/PIL/Image.py", line 2687, in open
% (filename if filename else fp))
OSError: cannot identify image file <_io.BufferedReader name='build/org.sw24softwares.starkeverben/fastlane/metadata/android/en-US/images/featureGraphic.png'>
Using a filename based on the hash of the contents means that the caching
algorithms for fdroidclient and browsers can safely cache the file forever
using the filename, since this guarantees that the contents will never
change for a given filename.
This does not cover screenshots, only icon.png, featureGraphic.png,
tvBanner.png, and promoGraphic.png.
fdroidserver#689
fdroid-website!453
This was done with much help from @uniqx. This is the first level of
supporting APK Signatures v1, v2, and v3. This is enough to include
APKs with any combo of v1/v2/v3 signatures. For this to work at all,
apksigner and androguard 3.3.3+ must be installed.
closes#399
699b3e4c69 got it wrong for targetSdkVersion.
Also, one confusing thing is that aapt outputs "sdkVersion: '3'" for
com.politedroid_3.apk but no "sdkVersion:" for no.min.target.sdk_987.apk.
F-Droid never really supported running on android-1 or android-2, so it
seems pointless to debug support for them.
androguard parses the whole APK before handing the instance back, this uses
the primitives to just find the <application android:debuggable=""> value,
then stop parsing.
#557
Normally, androguard parses the entire APK before it is possible to get any
values from it. This uses androguard primitives to only attempt to parse
the AndroidManifest.xml, then to quit as soon as it gets what it needs.
This greatly speeds up the parsing (1 minute vs 60 minutes).
fdroid/fdroidserver#557
pickle can serialize executable code, while JSON is only ever pure data.
The APK cache is only ever pure data, so no need for the security risks of
pickle. For example, if some malicious thing gets write access on the
`fdroid update` machine, it can write out a custom tmp/apkcache which would
then be executed. That is not possible with JSON.
This does just ignore any existing cache and rebuilds from scratch. That is
so we don't need to maintain pickle anywhere, and to ensure there are no
glitches from a conversion from pickle to JSON.
closes#163
Debian's autopkgtest defaults to considering any output on stderr as a sign
that the tests failed. it is simple to make this achieve that standard, so
it seems worth it for now at least.
Trying to reuse the pip test run is now a lot more work than just
writing it cutstom for each target platform.
Ubuntu/trusty does not have androguard 3.1.x nor aapt >= 26.x, so
using trusty's aapt will always have errors scraping some names from
APKs. This continues to use Google's binary.
`apt-get update` is now disabled by default by Travis, this adds it
back so that the latest files from the PPA are used.
Here's one example of an issue with pip on trusty:
https://github.com/requests/requests/issues/4006
pip3 install --quiet -e /home/travis/build/fdroidtravis/fdroidserver
Exception:
Traceback (most recent call last):
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2482, in _dep_map
return self.__dep_map
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2344, in __getattr__
raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python3.4/site-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python3.4/site-packages/pip/commands/install.py", line 278, in run
requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python3.4/site-packages/pip/req.py", line 1266, in prepare_files
req_to_install.extras):
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2291, in requires
dm = self._dep_map
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2484, in _dep_map
self.__dep_map = self._compute_dependencies()
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2517, in _compute_dependencies
common = frozenset(reqs_for_extra(None))
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/pkg_resources.py", line 2514, in reqs_for_extra
if req.marker_fn(override={'extra':extra}):
File "/home/travis/build/fdroidtravis/fdroidserver/env/lib/python-wheels/setuptools-3.3-py2.py3-none-any.whl/_markerlib/markers.py", line 113, in marker_fn
return eval(compiled_marker, environment)
File "<environment marker>", line 1, in <module>
NameError: name 'platform_system' is not defined
For some reason, the parser stopped working intermittently, even
though the format has been the same since aapt 23 or earlier. Then
also, some of the test cases pointed to symlinks that were no longer
generated, and one test app now has a blank versionName.
Strange that this wasn't caught in the gitlab-ci runs. !484
FAIL: test_get_api_id_aapt (__main__.CommonTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "./common.TestCase", line 578, in testA_get_api_id_aapt
self.assertEqual(versionName, vn)
AssertionError: '0.1' != "0.1' platformBuildVersionName='4.3.1-1425645"
- 0.1
+ 0.1' platformBuildVersionName='4.3.1-1425645
Instead of just crashing, first try to use the versionName as written in the
build metadata, otherwise just let it be blank. A blank versionName will
cause fdroidclient < 1.3 to crash. Blank versionNames are not allowed in
the .txt metadata format, only .yml.
closes#477closes#478
closes fdroidclient#1416
closes fdroidclient#1417
closes fdroidclient#1418
fdroiddata!3061
An APK (Netflix) was found to have the following icon filename:
\u2003\u2009\n.xml
This breaks the aapt dump parsing because it iterates line by line and
this filename goes across two lines. Consequently, icon_src will be
None (default value) when it is passed to the icons parser.
This code has never been used and contains some insecure uses of shell=True
Building Kivy apps should be done with the buildozer=yes method. The
buildozer method should probably be moved to a provisioner once that is in
place.
Generating a keystore is quite slow since it means a new RSA key is created.
That only needs to happen in the tests that check that it actually happened,
otherwise the test can just reuse the stored test keystore.
closes#432
