- Update usage documentation.
- Use argparse.
- Simplify config handling (similar to readmeta.py).
- Drop code for fdroidserver before 0.7.0.
- Reformat with black.
13016c5d63 in !602 used a set to prevent
duplicate entries, but that worked poorly because it required lots of
data wrapping. Instead, just normalize to JSON, then equality is easy.
The buildserver VM has not been upgraded yet to bullseye, so it is still on
Debian/stretch. The buildserver VM does not need to run `fdroid update`,
`fdroid signindex`, etc. so this new apksigner requirement should not
affect app builds even though they are stuck on Debian/stretch.
The current signing method uses apksigner to sign the JAR so that it
will automatically select algorithms that are compatible with Android
SDK 23, which added the most recent algorithms:
https://developer.android.com/reference/java/security/Signature
This signing method uses then inherits the default signing algothim
settings, since Java and Android both maintain those. That helps
avoid a repeat of being stuck on an old signing algorithm. That means
specifically that this call to apksigner does not specify any of the
algorithms.
The old indexes must be signed by SHA1withRSA otherwise they will no
longer be compatible with old Androids.
apksigner 30.0.0+ is available in Debian/bullseye, Debian/buster-backports,
Ubuntu 21.10, and Ubuntu 20.04 from the fdroid PPA. Here's a quick way to
test:
for f in `ls -1 /opt/android-sdk/build-tools/*/apksigner | sort ` /usr/bin/apksigner; do printf "$f : "; $f sign --v4-signing-enabled false; done
closes#1005
If a project uses fdroidserver as a library, then just calls
common.get_apk_id(), it will now work. Before, that project would have had
to include something like `common.config = {}` to avoid a stacktrace.
This needs a rerun of `fdroid update --clean`.
In case a build is disabled delete_disabled_builds takes care of
deleting it from the repo. But this only works if the apk follows the
normal name pattern. Otherwise it will stay in the folder and be picked
up by process_apks and added to the index.
Closes: #1002
With ~index-v2, the model is changing to offer the plain JSON file for easy
consumption. Then gpgsign will also provide a detached PGP signature for
systems that would rather verify based on PGP signatures than JAR signatures.
!1080closes#969
In case the version information is inside a submodule we need to
checkout the submodule at the version of the tag we test.
Found with org.courville.nova.
Closes: #622
Traceback (most recent call last):
File "/builds/fdroid/fdroidserver/./tests/gradle-release-checksums.py", line 130, in <module>
mr = project.mergerequests.create({
File "/usr/lib/python3/dist-packages/gitlab/exceptions.py", line 281, in wrapped_f
raise error(e.error_message, e.response_code, e.response_body) from e
gitlab.exceptions.GitlabCreateError: 409: ['Another open merge request already exists for this source branch: !1064']
In case the app repository has a broken submodule, checkupdates failed
and did not search for any version updates. Ignoring the error let's us
at least find new version in the main repo (which is probably the right
place anyhow) and thus an improvement.
allowlist and blocklist are much clearer terms with no cultural baggage.
This changes all "whitelist" references to "allowlist", and all "blacklist"
references to "blocklist".
config.yml requires ASCII or UTF-8 encoding because this code does not
auto-detect the file's encoding. That is left up to the YAML library.
YAML allows ASCII, UTF-8, UTF-16, and UTF-32 encodings. Since it is a
good idea to manage config.yml (WITHOUT PASSWORDS!) in git, it makes
sense to use a globally standard encoding.
Android Studio recommends "you use UTF-8 encoding whenever possible",
so this code assumes the files use UTF-8. UTF-8 is also the default
encoding on GNU/Linux and macOS.
https://sites.google.com/a/android.com/tools/knownissues/encoding
Windows will probably default to UTF16, since that's the native
encoding for files. So forcing things to use UTF-8 should help
compatibility.
Windows seems to require this, otherwise this happens:
Traceback (most recent call last):
File "tests/update.TestCase", line 737, in test_translate_per_build_anti_features
apps = fdroidserver.metadata.read_metadata(xref=True)
File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 813, in read_metadata
app = parse_metadata(metadatapath, appid in check_vcs, refresh)
File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 1023, in parse_metadata
parse_yaml_metadata(mf, app)
File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 1073, in parse_yaml_metadata
yamldata = yaml.safe_load(mf)
File "C:\python37\lib\site-packages\yaml\__init__.py", line 162, in safe_load
return load(stream, SafeLoader)
File "C:\python37\lib\site-packages\yaml\__init__.py", line 112, in load
loader = Loader(stream)
File "C:\python37\lib\site-packages\yaml\loader.py", line 34, in __init__
Reader.__init__(self, stream)
File "C:\python37\lib\site-packages\yaml\reader.py", line 85, in __init__
self.determine_encoding()
File "C:\python37\lib\site-packages\yaml\reader.py", line 124, in determine_encoding
self.update_raw()
File "C:\python37\lib\site-packages\yaml\reader.py", line 178, in update_raw
data = self.stream.read(size)
File "C:\python37\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 37: character maps to <undefined>
gotorevisionx tries to clean up the git repo before checking out a new
revision. In b848b99ba this was changed to reset and clean any submodule
as well. In case upstream has a broken submodule configuration this
could fail and we can't checkout the new revision. As we are doing a
reset and clean after checking out the new revision anyhow, this change
ignores submodule errors before the checkout and only makes sure that
the main repo is reset and clean.
This broke checkupdates for apps where old versions had broken
submodules. It checkout out the old version and got stuck, not able to
checkout any other version.
Use the tag as the version code if no regex was specified. This allows:
UpdateCheckData: '|||'
meaning the tag should be used for version code and name.
com.github.jameshnsears.quoteunquote defines flavours 'fdroid' and
'fdroidS'. The old code used flavour in line, which matches both and the
wrong one was selected.
Closes: #912
Use the tag as version, if no version file was specified:
UpdateCheckData: app/build.gradle|versionCode\s(\d+)||
Extract version from tag, if a regex was specified:
UpdateCheckData: app/build.gradle|versionCode\s(\d+)||Android-([\d.]+)
Use the tag for both if no file was specified:
UpdateCheckData: |\+(\d+)||Android-([\d.]+)
Since 24dd6740 UpdateCheckMode: Tags uses the found tag instead of
regenerating it from the AutoUpdateMode pattern making the pattern
superfluous. This adds support for dropping the pattern and a test case.
'ndk_paths' will be automatically filled out from well known sources
like $ANDROID_HOME/ndk-bundle and $ANDROID_HOME/ndk/*. If a required
version is missing in the buildserver VM, it will be automatically
downloaded and installed into the standard $ANDROID_HOME/ndk/
directory. Manually setting it here will override the auto-detected
values. The keys can either be the "release" (e.g. r21e) or the
"revision" (e.g. 21.4.7075529).
https://developer.android.com/studio/projects/configure-agp-ndk#agp_version_41
* sdkmanager installs "ndk;12.3.4567890" into $ANDROID_SDK_ROOT/ndk/
* sdkmanager installs "ndk-bundle" into $ANDROID_SDK_ROOT/ndk-bundle/
There are two version numbers used for NDKs: the "release" and the
"revision". The "release" is used in the download URL and zipball and the
"revision" is used in the source.properties and the gradle ndkVersion field.
Also, there are some builds which need multiple NDKs installed, so this
makes it possible to have a list of release/revision entries in build.ndk.
This does not yet add full support since _fdroidserver/build.py_ will also
need changes.
Following the pattern of the gradle bot, this will check the transparency
log for any new NDK release. If there are any, it will make a merge
request from @fdroid-bot.
Before, whenever an unsigned APK failed to reproduce, it was just deleted.
That makes debugging hard. This makes it keep the unsigned APK, which is
written in tmp/ when using --test.
@jspricke this is related to !864
We switched to apksigner in 50f0534d but old apks where still verified
with jarsigner (or an old apksigner version). Bumping the
METADATA_VERSION to force a rebuild of apkcache.
Hopefully this resolves de.chagemann.regexcrossword getting the
KnownVuln, DisabledAlgorithm tags.
This is a vestige of implementing builds from a .fdroid.yml file directly
in the app's source repo. It was never fully complete and seems to not be
used in any apps in fdroiddata. This makes `fdroid build --all` runs much
faster since it does not need to do any git handling for apps that do not
have any new builds to run.
4e8e29794948689281a4e431080e37be9b06e775d330c
If a non-APK is added with the appid/packageName that matches some APKs, it
should through an error.
Traceback (most recent call last):
File "/home/hans/code/fdroid/server/fdroid", line 22, in <module>
fdroidserver.__main__.main()
File "/home/hans/code/fdroid/server/fdroidserver/__main__.py", line 211, in main
mod.main()
File "/home/hans/code/fdroid/server/fdroidserver/update.py", line 2343, in main
index.make(apps, sortedids, apks, repodirs[0], False)
File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 142, in make
fdroid_signing_key_fingerprints)
File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 166, in make_v1
v1_sort_packages(packages, fdroid_signing_key_fingerprints)
File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 292, in v1_sort_packages
packages.sort(key=v1_sort_keys)
File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 288, in v1_sort_keys
.format(apkfilename=package['apkName']))
fdroidserver.exception.FDroidException: at.roteskreuz.stopcorona_8.jobf does not have a valid signature!
```
Traceback (most recent call last):
File "../fdroid", line 22, in <module>
fdroidserver.__main__.main()
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 227, in main
raise e
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 208, in main
mod.main()
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2340, in main
repoapps = prepare_apps(apps, apks, repodirs[0])
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2176, in prepare_apps
copy_triple_t_store_metadata(apps_with_packages)
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 1076, in copy_triple_t_store_metadata
_set_author_entry(app, 'authorWebSite', os.path.join(root, f))
File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 784, in _set_author_entry
with open(f, errors='replace') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'build/player.efis.cfd/pfd/src/main/play/contact-website.txt'
```
