This commit removes the mypy CI job, since there are no or few type
hints in the codebase and it failed on unrelated changes in the past.
It might be reintroduced when type hints get added.
- Update usage documentation.
- Use argparse.
- Simplify config handling (similar to readmeta.py).
- Drop code for fdroidserver before 0.7.0.
- Reformat with black.
gitlab.com has added pretty small quotas to free tier accounts, and this
job pushes a docker image that is nearly 1GB. And those rarely get used,
so this makes it opt in via a variable that can be set when manually
running a job, or globally in the "CI/CD Variables" Settings.
The current signing method uses apksigner to sign the JAR so that it
will automatically select algorithms that are compatible with Android
SDK 23, which added the most recent algorithms:
https://developer.android.com/reference/java/security/Signature
This signing method uses then inherits the default signing algothim
settings, since Java and Android both maintain those. That helps
avoid a repeat of being stuck on an old signing algorithm. That means
specifically that this call to apksigner does not specify any of the
algorithms.
The old indexes must be signed by SHA1withRSA otherwise they will no
longer be compatible with old Androids.
apksigner 30.0.0+ is available in Debian/bullseye, Debian/buster-backports,
Ubuntu 21.10, and Ubuntu 20.04 from the fdroid PPA. Here's a quick way to
test:
for f in `ls -1 /opt/android-sdk/build-tools/*/apksigner | sort ` /usr/bin/apksigner; do printf "$f : "; $f sign --v4-signing-enabled false; done
closes#1005
Run the whole test suite in an environment that is like the
buildserver guest VM. This installs python3-babel because that is
only used by the test suite, and not needed in the buildserver.
Some extra packages are required for this test run that are not
provided by the buildserver since they are not needed there:
* gnupg-agent for the full signing setup
* python3-babel for compiling localization files
* python3-clint for fancy progress bars for users
fdroid/ci-images-base is deprecated and archived.
* https://gitlab.com/eighthave/fdroidserver/-/jobs/1961701458
Collecting pynacl>=1.0.1 (from paramiko->fdroidserver==2.1a0)
Downloading 27582568be/PyNaCl-1.5.0.tar.gz (3.4MB)
Installing build dependencies: started
Installing build dependencies: finished with status 'error'
Complete output from command /builds/eighthave/fdroidserver/env/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-xokvr6uk --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=40.8.0 wheel "cffi>=1.4.1; python_implementation != 'PyPy'":
Collecting setuptools>=40.8.0
Using cached 0dd4c79605/setuptools-60.5.0-py3-none-any.whl
Collecting wheel
Using cached 003e593296/wheel-0.37.1-py2.py3-none-any.whl
Collecting cffi>=1.4.1
Downloading 92de7e1217/cffi-1.15.0.tar.gz (484kB)
Collecting pycparser (from cffi>=1.4.1)
Downloading 5f610ebe42/pycparser-2.21-py2.py3-none-any.whl (118kB)
Building wheels for collected packages: cffi
Running setup.py bdist_wheel for cffi: started
Running setup.py bdist_wheel for cffi: finished with status 'error'
Complete output from command /builds/eighthave/fdroidserver/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-djg9jc8p/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-d1knhl7l --python-tag cp37:
unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
No working compiler found, or bogus compiler options passed to
the compiler from Python's standard "distutils" module. See
the error messages above. Likely, the problem is not related
to CFFI but generic to the setup.py of any Python package that
tries to compile C code. (Hints: on OS/X 10.8, for errors about
-mno-fused-madd see http://stackoverflow.com/questions/22313407/
Otherwise, see https://wiki.python.org/moin/CompLangPython or
the IRC channel #python on irc.libera.chat.)
Trying to continue anyway. If you are trying to install CFFI from
a build done in a different context, you can ignore this warning.
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/lock.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/recompiler.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/__init__.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/cparser.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/verifier.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/api.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/commontypes.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/error.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/model.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.7/cffi
copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.7/cffi
copying cffi/_embedding.h -> build/lib.linux-x86_64-3.7/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.7/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/c
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/builds/eighthave/fdroidserver/env/include -I/usr/include/python3.7m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.7/c/_cffi_backend.o
unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Failed building wheel for cffi
Running setup.py clean for cffi
Failed to build cffi
Installing collected packages: setuptools, wheel, pycparser, cffi
Running setup.py install for cffi: started
Running setup.py install for cffi: finished with status 'error'
Complete output from command /builds/eighthave/fdroidserver/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-djg9jc8p/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-ek80c81s/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-xokvr6uk --compile --install-headers /builds/eighthave/fdroidserver/env/include/site/python3.7/cffi:
unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
No working compiler found, or bogus compiler options passed to
the compiler from Python's standard "distutils" module. See
the error messages above. Likely, the problem is not related
to CFFI but generic to the setup.py of any Python package that
tries to compile C code. (Hints: on OS/X 10.8, for errors about
-mno-fused-madd see http://stackoverflow.com/questions/22313407/
Otherwise, see https://wiki.python.org/moin/CompLangPython or
the IRC channel #python on irc.libera.chat.)
Trying to continue anyway. If you are trying to install CFFI from
a build done in a different context, you can ignore this warning.
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/lock.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/recompiler.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/__init__.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/cparser.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/verifier.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/api.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/commontypes.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/error.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/model.py -> build/lib.linux-x86_64-3.7/cffi
copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.7/cffi
copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.7/cffi
copying cffi/_embedding.h -> build/lib.linux-x86_64-3.7/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.7/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/c
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/builds/eighthave/fdroidserver/env/include -I/usr/include/python3.7m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.7/c/_cffi_backend.o
unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Command "/builds/eighthave/fdroidserver/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-djg9jc8p/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-ek80c81s/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-xokvr6uk --compile --install-headers /builds/eighthave/fdroidserver/env/include/site/python3.7/cffi" failed with error code 1 in /tmp/pip-install-djg9jc8p/cffi/
----------------------------------------
Command "/builds/eighthave/fdroidserver/env/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-xokvr6uk --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=40.8.0 wheel "cffi>=1.4.1; python_implementation != 'PyPy'"" failed with error code 1 in None
mypy 0.901 was released and everything broke:
https://gitlab.com/linsui/fdroidserver/-/jobs/1330206567
My point is to reduce the number of false job fails like this one. We
have a lot of checkers running, I really think they need to prove they are
adding value before we invest any time maintaining them. mypy is still
"wait and see" in terms of the adding any value.
!951
Following the pattern of the gradle bot, this will check the transparency
log for any new NDK release. If there are any, it will make a merge
request from @fdroid-bot.
`Builds:` is directly in the .yml metadata, and the internal representation
now uses the same "Builds" name. Reading in .txt metadata files used the
name "builds" for that. So this test now uses "Builds" as canonical.
ci-images-server was created separately from ci-images-base back in the day
when we had to support Debian/stretch without stretch-backports. Those
days are gone now, so ci-images-server has been archived and should no
longer be used.
GIT_DEPTH sets how many commits of history to clone in CI Jobs.
gitlab.com defaults to 50 with a max of 1000. The metadata_v0 job is
the only job that needs history, and it needs more than 50. So this
sets the default to 1, then metadata_v0 to 1000.
https://docs.gitlab.com/ee/ci/pipelines/settings.html#git-shallow-clone
Liberapay was originally included using a numeric ID, since they had
not yet finalized the public URLs. Now it is a username. So this
logic prefers the username in Liberapay: field, and keeps the old
LiberapayID: to ease migration. LiberapayID: will not override
Liberapay:. Clients are expected to prefer Liberapay: over LiberapayID:
