This SSH connection is only ever to the Vagrant VM on the same machine.
>> Issue: [B507:ssh_no_host_key_verification] Paramiko call with policy set to automatically trust the unknown host key.
Severity: High Confidence: Medium
CWE: CWE-295 (https://cwe.mitre.org/data/definitions/295.html)
More Info: https://bandit.readthedocs.io/en/1.7.6/plugins/b507_ssh_no_host_key_verification.html
Location: ./fdroidserver/build.py:104:8
103 sshs = paramiko.SSHClient()
104 sshs.set_missing_host_key_policy(paramiko.AutoAddPolicy())
105 sshs.connect(sshinfo['hostname'], username=sshinfo['user'],
Python 3.12 completely removed the builtin `distutils` module. This
commit replaces its use with the `packaging` package, which is an
external dependency, but maintained by the Python developers.
Signed-off-by: Andrew Gunnerson <accounts+gitlab@chiller3.com>
The mirrors: entry in config.yml is great for quick access and shorter
mirror lists. Now that we are adding a lot more metadata to the mirrors,
including potentially the full text of the privacy policy, having this
in its own file should make these cases easier to manage.
Found while compiling io.privatestorage.privatestoragemobile_9:
Traceback (most recent call last):
File "/home/fdroid/fdroidserver/fdroidserver/build.py", line 1053, in main
if trybuild(app, build, build_dir, output_dir, log_dir,
File "/home/fdroid/fdroidserver/fdroidserver/build.py", line 813, in trybuild
build_server(app, build, vcs, build_dir, output_dir, log_dir, force)
File "/home/fdroid/fdroidserver/fdroidserver/build.py", line 253, in build_server
logging.debug("buildserver > " + str(line, 'utf-8').rstrip())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfa in position 8: invalid start byte
Before, lots of pieces of the Android SDK were required for fdroidserver to
operate, like aapt, zipalign, etc. Now, apksigner is the only requirement.
%"support APK Signature v2+"
!889
https://gitlab.com/fdroid/fdroidclient/-/issues/2619#note_1421280589
The test needed to change because the test index files contained category
definitions that were not ever used in the "copy tests/repo, generate java/gpg
keys, update, and gpgsign" test in tests/run-tests.
!1366 makes it so categories are now defined by the repo. Categories can be
defined in the config so that lint has a list of categories to enforce. This
also provides a place for localization and icons for the categories. The old
way of defining categories was just listing them in app metadata files. This
restores that way of functioning when using index-v2.
closes#1137
The case of Anti-Features keys and filenames is quite confusing. I was
confused to find out that config/antiFeatures.yml is correct, while
config/antifeatures.yml is nothing. This throws an error message to make
that clear.
_builds_to_yaml does not use any features of the metadata.Build class, so
it can operate on plain dicts as well. It also does not need to output
Build instances because those are converted to plain dicts when writing out
to YAML.
The type conversion should all happen in post_parse_yaml_metadata whenever
possible. Also, when `if` blocks end in `return`, it is clearer if no
`elif` or `else` is used.
This should reduce surprises when dealing with filenames in things like
`rm:`. So any float/int/bool value can be used directly, without quoting.
* A plain str/int/float value is interpreted as a list of one string.
* Dictionaries as values throws error.
* A set is treated like a list.
Even for people who know what the special floats not-a-number, infinity,
and negative infinity, they don't necessarily know the YAML 1.2 syntax for
these. I didn't. And I've spent some quality time fighting things with
those values. They are also easy to reliably convert to string values.
If the metadata file contains NoSourceSince:, it is added to the collection
of Anti-Features. When rewriting the .yml file, NoSourceSince should only
be written into the AntiFeatures: collection if there are manual changes,
e.g. the user had provided translations.
Throughout the code, variables named "app" are instances of the App class.
In this case, this is related, but it is a dict not an App instance, since
it is being prepared for including in the index-v1.json.
If there was a global default on a machine that was something other than
'master', these things would crash with:
Traceback (most recent call last):
File "/home/hans/code/fdroid/server/fdroid", line 22, in <module>
fdroidserver.__main__.main()
File "/home/hans/code/fdroid/server/fdroidserver/__main__.py", line 230, in main
raise e
File "/home/hans/code/fdroid/server/fdroidserver/__main__.py", line 211, in main
mod.main()
File "/home/hans/code/fdroid/server/fdroidserver/deploy.py", line 833, in main
push_binary_transparency(BINARY_TRANSPARENCY_DIR,
File "/home/hans/code/fdroid/server/fdroidserver/deploy.py", line 705, in push_binary_transparency
local.pull('master')
File "/usr/lib/python3/dist-packages/git/remote.py", line 1045, in pull
res = self._get_fetch_info_from_stderr(proc, progress, kill_after_timeout=kill_after_timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/git/remote.py", line 848, in _get_fetch_info_from_stderr
proc.wait(stderr=stderr_text)
File "/usr/lib/python3/dist-packages/git/cmd.py", line 604, in wait
raise GitCommandError(remove_password_if_present(self.args), status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(1)
cmdline: git pull -v -- local master
stderr: 'fatal: couldn't find remote ref master'
The builder should check the `AllowedAPKSigningKeys` at build time, so
that the CI can check if somebody gives a wrong value that doesn't match
a compared RB binary. In the event it fails, it gives useful
information, and in the event it succeeds, it makes it clear that this
build has verification back to the developer's original key.
Also, add tests for this to the test suite.
Per fdroiddata!12911 the linter should error out if somebody uses `binary` or
`Binaries` without supplying an `AllowedAPKSigningKeys`.
There are two reasons for this:
- Security: this allows full verification that the binaries built match
the developers, not just what happened to get uploaded onto github at
some later point in time.
- Reliable updates: if the signing key changes, users won't be able to
update, so this is something we should learn about when upstreams send
in commits changing their signing key, rather than just leaving it to
chance.
I profiled this with timeit and a dict with 1000000000 items, and this is
the time difference:
with_equals: 0.8466835720173549
with_is: 0.8536969239939936
with_old: 1.4458542719949037
I also compared using `==` and `is`, and `==` was slightly faster.
I tried to get this to indent the .yaml files properly so yamllint defaults
work with tests/metadata/dump/*.yaml, but it didn't take for some reason:
yaml.indent(mapping=4, sequence=4, offset=2)
This function is only used in checkupdates, and removing it from the App
class moves the App class one step closer to being a plain dict, which is a
more Pythonic style.
Before this, there were separate post-parse paths for app-fields versus
build-flags. This makes all TYPE_STRING values always go through the same
post-parse code path.
My guess is that this is some kind of vestige of the old code structure,
back when there was .txt and .yml formats. This makes it a normal Python
function: input as arg, return value is the result.
This lets mirrors: in config.yml be the same list-of-dicts format as it is
in index-v2. This also includes a data format conversion to maintain the
right format for the old, unchanging index v0 and v1 formats.
#928#1107
It turns out that the maven: field was originally declared as a TYPE_STRING,
given that it was not given a different type in metadata.py's flagtypes.
The code was confused because it was given a default value of `False` rather
than `None` as the rest of the TYPE_STRING fields have.
This construct in build.py means maven: should always be a string:
if '@' in build.maven:
maven_dir = os.path.join(root_dir, build.maven.split('@', 1)[1])
else:
maven_dir = root_dir
The paths in the config must be strings because they are used in things
like env vars where they must be strings. Plus lots of other places in the
code assumes they are strings. This is the first step to defining the
border of where paths can be pathlib.Path() and where they must be strings.
make_v0() modified the apps data structure to copy an anti feature for a
specific version to all versions resulting in index-v1 and -v2 to
contain wrong anti feature annotations. This patch fixes this and adds a
test that the data structure is no longer modified.
The bug shadowed bugs in the AF implementation of -v1 and -v2 resulting
in not coping the version specific data. This is corrected as well.
This is also tested now.
For -v2 the AF dict is now sorted to make the result reproducible.
Finally The NoSourceSince AF was added as a per version and overall AF
in -v1 and is now only applied as an overall AF and the test is updated
accordingly.
Things like apksigner and @obfusk's tools handle this now.
jarsigner is used in the test, since that's the most common use of
`common.find_sdk_tools_cmd()`.
closes#1100
The pattern in Debian is:
java-$VERSION-openjdk-$DEB_HOST_ARCH
The regex for $DEB_HOST_ARCH did not match arm64, armel and armhf. As it
can be any string just match anything.
* for f in locale/*/LC_MESSAGES/fdroidserver.po; do msgattrib --set-obsolete --no-wrap --ignore-file=locale/fdroidserver.pot -o $f $f; done
* sed -i 's, \.\./fdroidserver/stats\.py,,' locale/*/LC_MESSAGES/fdroidserver.po
Ran this to generate/refresh the test index files:
`cd tests; ../fdroid update --nosign --pretty`
* converts config.py description to a single line, since the values are no
longer stripped, so this give the same output.
closes#835
The build can get stuck when running into the timeout and paramiko not
noticing it. This can be reproduced by setting a small build timeout for
an app. I tried setting set_keepalive() but it did not help. This
forcefully closes the ssh channel to the VM so the fdroid build will
continue. As we destroy the VM in the next step anyhow I think this is
fine.
