The bug fixed in !1370 was due to a reuse of the per app anti features
dictionary in the per version one and adding the per version anti
features into it. This was not caught by the test suite as the app
versions are processed from newest to oldest and the test data had only
anti features in the oldest version. This patch adds an anti feature to
a newest version so only those in between should not have it.
The type conversion should all happen in post_parse_yaml_metadata whenever
possible. Also, when `if` blocks end in `return`, it is clearer if no
`elif` or `else` is used.
These test cases were writing assuming they had to transform the data
format. That is no longer the case. Going forward, the parsing process
converts everything to a standardized format. That will hopefully be
enforceable by the JSON Schema in the future.
* The metadata.Builds() class initializes all possible flags, then the
flags with init values are filtered out when writing out YAML.
* TYPE_SCRIPT flags with one entry will be converted to a string.
This should reduce surprises when dealing with filenames in things like
`rm:`. So any float/int/bool value can be used directly, without quoting.
* A plain str/int/float value is interpreted as a list of one string.
* Dictionaries as values throws error.
* A set is treated like a list.
Even for people who know what the special floats not-a-number, infinity,
and negative infinity, they don't necessarily know the YAML 1.2 syntax for
these. I didn't. And I've spent some quality time fighting things with
those values. They are also easy to reliably convert to string values.
If the metadata file contains NoSourceSince:, it is added to the collection
of Anti-Features. When rewriting the .yml file, NoSourceSince should only
be written into the AntiFeatures: collection if there are manual changes,
e.g. the user had provided translations.
If there was a global default on a machine that was something other than
'master', these things would crash with:
Traceback (most recent call last):
File "/home/hans/code/fdroid/server/fdroid", line 22, in <module>
fdroidserver.__main__.main()
File "/home/hans/code/fdroid/server/fdroidserver/__main__.py", line 230, in main
raise e
File "/home/hans/code/fdroid/server/fdroidserver/__main__.py", line 211, in main
mod.main()
File "/home/hans/code/fdroid/server/fdroidserver/deploy.py", line 833, in main
push_binary_transparency(BINARY_TRANSPARENCY_DIR,
File "/home/hans/code/fdroid/server/fdroidserver/deploy.py", line 705, in push_binary_transparency
local.pull('master')
File "/usr/lib/python3/dist-packages/git/remote.py", line 1045, in pull
res = self._get_fetch_info_from_stderr(proc, progress, kill_after_timeout=kill_after_timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/git/remote.py", line 848, in _get_fetch_info_from_stderr
proc.wait(stderr=stderr_text)
File "/usr/lib/python3/dist-packages/git/cmd.py", line 604, in wait
raise GitCommandError(remove_password_if_present(self.args), status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(1)
cmdline: git pull -v -- local master
stderr: 'fatal: couldn't find remote ref master'
The builder should check the `AllowedAPKSigningKeys` at build time, so
that the CI can check if somebody gives a wrong value that doesn't match
a compared RB binary. In the event it fails, it gives useful
information, and in the event it succeeds, it makes it clear that this
build has verification back to the developer's original key.
Also, add tests for this to the test suite.
I tried to get this to indent the .yaml files properly so yamllint defaults
work with tests/metadata/dump/*.yaml, but it didn't take for some reason:
yaml.indent(mapping=4, sequence=4, offset=2)
Before this, there were separate post-parse paths for app-fields versus
build-flags. This makes all TYPE_STRING values always go through the same
post-parse code path.
My guess is that this is some kind of vestige of the old code structure,
back when there was .txt and .yml formats. This makes it a normal Python
function: input as arg, return value is the result.
This lets mirrors: in config.yml be the same list-of-dicts format as it is
in index-v2. This also includes a data format conversion to maintain the
right format for the old, unchanging index v0 and v1 formats.
#928#1107
It turns out that the maven: field was originally declared as a TYPE_STRING,
given that it was not given a different type in metadata.py's flagtypes.
The code was confused because it was given a default value of `False` rather
than `None` as the rest of the TYPE_STRING fields have.
This construct in build.py means maven: should always be a string:
if '@' in build.maven:
maven_dir = os.path.join(root_dir, build.maven.split('@', 1)[1])
else:
maven_dir = root_dir
The paths in the config must be strings because they are used in things
like env vars where they must be strings. Plus lots of other places in the
code assumes they are strings. This is the first step to defining the
border of where paths can be pathlib.Path() and where they must be strings.
This job just started failing, but wasn't before:
https://gitlab.com/eighthave/fdroidserver/-/jobs/4060582594
But I look at it, and it looks right that it fails. So how on earth
was it succeeding before? Basically the `os.getenv('ANDROID_HOME')`
returns `None` when `ANDROID_HOME` is not set. It is not set in both the
jobs, so how did it not stacktrace before?
make_v0() modified the apps data structure to copy an anti feature for a
specific version to all versions resulting in index-v1 and -v2 to
contain wrong anti feature annotations. This patch fixes this and adds a
test that the data structure is no longer modified.
The bug shadowed bugs in the AF implementation of -v1 and -v2 resulting
in not coping the version specific data. This is corrected as well.
This is also tested now.
For -v2 the AF dict is now sorted to make the result reproducible.
Finally The NoSourceSince AF was added as a per version and overall AF
in -v1 and is now only applied as an overall AF and the test is updated
accordingly.
This script is some of the only documentation of certain processes that
run in the buildserver setup. It is not really maintained, but is still
here as a kind of reference.
Signing files is a slow operation, especially with jarsigner. This speeds
up the full test run from 10-12 seconds to 2-3 seconds, which makes it
possible to run the tests interactively again. And it stops signing the
file entirely for tests that do not even touch that file.
In the long run, it would probably make sense to have each test case sign
the file as it needs it, but that's a much bigger change.
Things like apksigner and @obfusk's tools handle this now.
jarsigner is used in the test, since that's the most common use of
`common.find_sdk_tools_cmd()`.
closes#1100
* for f in locale/*/LC_MESSAGES/fdroidserver.po; do msgattrib --set-obsolete --no-wrap --ignore-file=locale/fdroidserver.pot -o $f $f; done
* sed -i 's, \.\./fdroidserver/stats\.py,,' locale/*/LC_MESSAGES/fdroidserver.po
Ran this to generate/refresh the test index files:
`cd tests; ../fdroid update --nosign --pretty`
* converts config.py description to a single line, since the values are no
longer stripped, so this give the same output.
closes#835
This adds the option to configure which set of signatures `fdroid
scanner` should use, by configuring it in `config.yml`. It allows
fetching signatures in our custom json format. It also adds 3 additional
sources: 'suss', 'exodus', 'etip'
makebuildserver.config.py is no more, builder/Vagrantfile is now where the
CPU and memory is configured for the buildserver VM. In fact, that was
always the actual place, the makebuildserver.config.py thing was just
confused.
This should have been part of !1222
openjdk-11 11.0.17 in Debian unstable fails to verify weak signatures:
jarsigner -verbose -strict -verify tests/signindex/guardianproject.jar
131 Fri Dec 02 20:10:00 CET 2016 META-INF/MANIFEST.MF
252 Fri Dec 02 20:10:04 CET 2016 META-INF/1.SF
2299 Fri Dec 02 20:10:04 CET 2016 META-INF/1.RSA
0 Fri Dec 02 20:09:58 CET 2016 META-INF/
m ? 48743 Fri Dec 02 20:09:58 CET 2016 index.xml
s = signature was verified
m = entry is listed in manifest
k = at least one certificate was found in keystore
? = unsigned entry
- Signed by "EMAILADDRESS=root@guardianproject.info, CN=guardianproject.info, O=Guardian Project, OU=FDroid Repo, L=New York, ST=New York, C=US"
Digest algorithm: SHA1 (disabled)
Signature algorithm: SHA1withRSA (disabled), 4096-bit key
WARNING: The jar will be treated as unsigned, because it is signed with a weak algorithm that is now disabled by the security property:
jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024, DSA keySize < 1024, SHA1 denyAfter 2019-01-01, include jdk.disabled.namedCurves
getsig.java was replaced by a Python implementation in 6e2d0a9e (2014)
and the test was only there to compare the results for the transition.
Dropping this as it no longer works starting with 11.0.17+8.
Make sudo, init prebuild, build and Prepare fields lists and only
concatenate them with '; ' before execution. This allows arbitrary
commands inside the fileds (even && and ';') as we don't need to split
the commands again for rewritemeta.
