https://gitlab.com/fdroid/fdroidclient/-/issues/2619#note_1421280589
The test needed to change because the test index files contained category
definitions that were not ever used in the "copy tests/repo, generate java/gpg
keys, update, and gpgsign" test in tests/run-tests.
The type conversion should all happen in post_parse_yaml_metadata whenever
possible. Also, when `if` blocks end in `return`, it is clearer if no
`elif` or `else` is used.
These test cases were writing assuming they had to transform the data
format. That is no longer the case. Going forward, the parsing process
converts everything to a standardized format. That will hopefully be
enforceable by the JSON Schema in the future.
* The metadata.Builds() class initializes all possible flags, then the
flags with init values are filtered out when writing out YAML.
* TYPE_SCRIPT flags with one entry will be converted to a string.
This should reduce surprises when dealing with filenames in things like
`rm:`. So any float/int/bool value can be used directly, without quoting.
* A plain str/int/float value is interpreted as a list of one string.
* Dictionaries as values throws error.
* A set is treated like a list.
Even for people who know what the special floats not-a-number, infinity,
and negative infinity, they don't necessarily know the YAML 1.2 syntax for
these. I didn't. And I've spent some quality time fighting things with
those values. They are also easy to reliably convert to string values.
If the metadata file contains NoSourceSince:, it is added to the collection
of Anti-Features. When rewriting the .yml file, NoSourceSince should only
be written into the AntiFeatures: collection if there are manual changes,
e.g. the user had provided translations.
I tried to get this to indent the .yaml files properly so yamllint defaults
work with tests/metadata/dump/*.yaml, but it didn't take for some reason:
yaml.indent(mapping=4, sequence=4, offset=2)
Before this, there were separate post-parse paths for app-fields versus
build-flags. This makes all TYPE_STRING values always go through the same
post-parse code path.
My guess is that this is some kind of vestige of the old code structure,
back when there was .txt and .yml formats. This makes it a normal Python
function: input as arg, return value is the result.
The paths in the config must be strings because they are used in things
like env vars where they must be strings. Plus lots of other places in the
code assumes they are strings. This is the first step to defining the
border of where paths can be pathlib.Path() and where they must be strings.
Make sudo, init prebuild, build and Prepare fields lists and only
concatenate them with '; ' before execution. This allows arbitrary
commands inside the fileds (even && and ';') as we don't need to split
the commands again for rewritemeta.
Windows seems to require this, otherwise this happens:
Traceback (most recent call last):
File "tests/update.TestCase", line 737, in test_translate_per_build_anti_features
apps = fdroidserver.metadata.read_metadata(xref=True)
File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 813, in read_metadata
app = parse_metadata(metadatapath, appid in check_vcs, refresh)
File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 1023, in parse_metadata
parse_yaml_metadata(mf, app)
File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 1073, in parse_yaml_metadata
yamldata = yaml.safe_load(mf)
File "C:\python37\lib\site-packages\yaml\__init__.py", line 162, in safe_load
return load(stream, SafeLoader)
File "C:\python37\lib\site-packages\yaml\__init__.py", line 112, in load
loader = Loader(stream)
File "C:\python37\lib\site-packages\yaml\loader.py", line 34, in __init__
Reader.__init__(self, stream)
File "C:\python37\lib\site-packages\yaml\reader.py", line 85, in __init__
self.determine_encoding()
File "C:\python37\lib\site-packages\yaml\reader.py", line 124, in determine_encoding
self.update_raw()
File "C:\python37\lib\site-packages\yaml\reader.py", line 178, in update_raw
data = self.stream.read(size)
File "C:\python37\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 37: character maps to <undefined>
The .txt format was the last place where the lowercase "builds" was used,
this converts references everywhere to be "Builds". This makes it possible
to load metadata YAML files with any YAML parser, then have it possible to
use fdroidserver methods on that data, like metadata.write_metadata().
The test files in tests/metadata/dump/*.yaml were manually edited by cutting
the builds: block and putting it the sort order for Builds: so the contents
should be unchanged.
```
sed -i \
-e 's/app\.builds/app.get('Builds', \[\])/g' \
-e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
-e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
-e "s/app\.get(Builds, \[\])/app.get('Builds', \[\])/g" \
-e "s/app\.get('Builds', \[\])\.append/app\['Builds'\].append/g" \
-e "s/app\['builds'\]/app.get('Builds', [])/g" \
*/*.*
```
