1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-07-14 13:00:08 +02:00
Commit Graph

66 Commits

Author SHA1 Message Date
Hans-Christoph Steiner
a6d35a7ee1
metadata: always open metadata files as UTF-8
Windows seems to require this, otherwise this happens:

Traceback (most recent call last):

  File "tests/update.TestCase", line 737, in test_translate_per_build_anti_features

    apps = fdroidserver.metadata.read_metadata(xref=True)

  File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 813, in read_metadata

    app = parse_metadata(metadatapath, appid in check_vcs, refresh)

  File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 1023, in parse_metadata

    parse_yaml_metadata(mf, app)

  File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 1073, in parse_yaml_metadata

    yamldata = yaml.safe_load(mf)

  File "C:\python37\lib\site-packages\yaml\__init__.py", line 162, in safe_load

    return load(stream, SafeLoader)

  File "C:\python37\lib\site-packages\yaml\__init__.py", line 112, in load

    loader = Loader(stream)

  File "C:\python37\lib\site-packages\yaml\loader.py", line 34, in __init__

    Reader.__init__(self, stream)

  File "C:\python37\lib\site-packages\yaml\reader.py", line 85, in __init__

    self.determine_encoding()

  File "C:\python37\lib\site-packages\yaml\reader.py", line 124, in determine_encoding

    self.update_raw()

  File "C:\python37\lib\site-packages\yaml\reader.py", line 178, in update_raw

    data = self.stream.read(size)

  File "C:\python37\lib\encodings\cp1252.py", line 23, in decode

    return codecs.charmap_decode(input,self.errors,decoding_table)[0]

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 37: character maps to <undefined>
2021-06-18 10:30:55 +02:00
Hans-Christoph Steiner
1abbc9732e
metadata: add test_build_ndk_path 2021-06-18 10:30:44 +02:00
linsui
8f21f1e510 metadata.py/rewritemeta.py: use pathlib and support Windows 2021-06-08 21:31:55 +08:00
Hans-Christoph Steiner
d05ff9db1d
easy changes to black code format in test cases
This does not change areas of code that should be manually reformatted.
2021-06-07 11:53:58 +02:00
Hans-Christoph Steiner
bf25b4ca03 eliminate app.builds everywhere, it should be app['Builds']
The .txt format was the last place where the lowercase "builds" was used,
this converts references everywhere to be "Builds".  This makes it possible
to load metadata YAML files with any YAML parser, then have it possible to
use fdroidserver methods on that data, like metadata.write_metadata().

The test files in tests/metadata/dump/*.yaml were manually edited by cutting
the builds: block and putting it the sort order for Builds: so the contents
should be unchanged.

```
sed -i \
 -e 's/app\.builds/app.get('Builds', \[\])/g' \
 -e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
 -e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
 -e "s/app\.get(Builds, \[\])/app.get('Builds', \[\])/g" \
 -e "s/app\.get('Builds', \[\])\.append/app\['Builds'\].append/g" \
 -e "s/app\['builds'\]/app.get('Builds', [])/g" \
 */*.*
```
2020-12-15 08:55:05 +01:00
Hans-Christoph Steiner
828d6015ef purge code that modifies the app description, including linkifying
closes #845
2020-12-08 09:37:49 +01:00
Marcus Hoffmann
709f4c9b18 pickle -> yaml rename 2020-09-10 18:38:43 +02:00
Marcus Hoffmann
4cd96d4a9f use yaml safeloader in tests
Try to use CSafeLoader when possible because its significantly faster.
Use the normal Safeloader otherwise. (This mirrors the non-test code
behaviour)
2020-09-10 14:18:36 +02:00
Michael Pöhn
cb368a674c remove txt srclib support 2020-08-20 20:40:15 +02:00
Marcus Hoffmann
3ebc44c54f fix tests after they switched to yaml 2020-06-25 00:07:42 +02:00
Marcus Hoffmann
03881154c6 metadata: make linkresolver an actual object
Previously this was magically capturing the apps dict when passing it around as a
function. This also moved the code to the metadata module.

Add a test doing read_metadata where the linkresolver is used. This
happens when the apps we read have a [[app.id]] link to another app.
2020-06-24 23:53:35 +02:00
Hans-Christoph Steiner
0183592526
update: insert donation links based on FUNDING.yml
GitHub has specified FUNDING.yml, a file to include in a git repo for
pointing people to donation links.  Since F-Droid also points people
to donation links, this parses them to fill out Donate:
and OpenCollective:.  Specifying those in the metadata file takes
precedence over the FUNDING.yml.  This follows the same pattern as how
`fdroid update` includes Fastlane/Triple-T metadata.  This lets the
git repo maintain those specific donations links themselves.

https://help.github.com/en/articles/displaying-a-sponsor-button-in-your-repository#about-funding-files

The test file was generated using:

```python
import os, re, yaml

found = dict()
for root, dirs, files in os.walk('.'):
    for f in files:
        if f == 'FUNDING.yml':
            with open(os.path.join(root, f)) as fp:
                data = yaml.safe_load(fp)
            for k, v in data.items():
                if k not in found:
                    found[k] = set()
                if not v:
                    continue
                if isinstance(v, list):
                    for i in v:
                        found[k].add(i)
                else:
                    found[k].add(v)

            with open('gather-funding-names.yaml', 'w') as fp:
                output = dict()
                for k, v in found.items():
                    output[k] = sorted(v)
                yaml.dump(output, fp, default_flow_style=False)
```
2020-06-16 15:35:26 +02:00
Hans-Christoph Steiner
37f37ebd88
use default accepted_formats since all the files are .yml anyway 2020-06-10 10:43:21 +02:00
Hans-Christoph Steiner
b63e9e68c5
move all test metadata files to .yml format
for f in metadata/*.txt; do ../fdroid rewritemeta --to yml $(basename $f | sed 's,\.txt,,'); done
2020-06-10 09:56:31 +02:00
Hans-Christoph Steiner
13d9a122bf metadata: validate STRING and INT build field types
This converts float/int to string for things like commit: or versionName:.
For versionCode, which must be an integer, it throws an exception if the
data is any other type.
2020-06-02 22:06:01 +02:00
Hans-Christoph Steiner
a0e3b01e94 metadata: parsed srclibs must always return a dict as the container 2020-05-14 16:28:13 +02:00
Michael Pöhn
4e69ff582f run yamllint when parsing failed and also on fdroid lint runs 2020-04-24 15:47:31 +02:00
Michael Pöhn
5741e6930b yml srclibs: support Subdir as list 2020-04-16 11:31:59 +02:00
Michael Pöhn
1ac7d612b1 yml srclibs: support multiline Prepare 2020-04-16 11:31:59 +02:00
Michael Pöhn
58776da694 get tests working 2020-04-16 11:31:59 +02:00
Michael Pöhn
ee3d8d2f18 srclib: yml parsing + tests for yml and txt 2020-04-16 11:31:59 +02:00
Michael Pöhn
2c86d38028 srclib: no spaces in dict key 2020-04-16 11:31:59 +02:00
Michael Pöhn
c6a97939f1 rename parse_srclib to parese_txt_srclib + test case 2020-04-16 11:31:59 +02:00
Michael Pöhn
dcf3837bcb parse yaml: ignore (and warn) deprecated field: Provides 2019-07-15 15:45:02 +02:00
Michael Pöhn
6e48663230 test that write yaml does not write provides 2019-07-11 03:35:23 +02:00
Michael Pöhn
723bd110a6 test that yaml parsing does not accept provides 2019-07-11 03:35:23 +02:00
Michael Pöhn
66105de833 improve litecoin validation + tests 2019-05-28 11:04:43 +02:00
Michael Pöhn
06cec2041d improve bitcoin validation regex + testcases 2019-05-07 22:43:05 +02:00
Michael Pöhn
67731470cc Revert "Merge branch 'write-yaml-overhaul' into 'master'"
This reverts merge request !630
2019-04-01 10:24:00 +00:00
Michael Pöhn
2683b37044 yml metadata write: do not use local functions 2019-03-19 01:01:18 +01:00
Michael Pöhn
881a79fa84 test writing all yaml fields 2019-03-18 22:45:35 +01:00
Michael Pöhn
63afc0acb5 use pyyaml for writing metadata instead of ruamel 2019-03-18 22:45:35 +01:00
Michael Pöhn
31ca2092a1 yaml parsing: script build flags can now be lists 2018-11-29 21:53:37 +01:00
Michael Pöhn
c15a7508e7 write yaml script metadata as lists 2018-11-29 21:53:37 +01:00
Michael Pöhn
a21635ae2e fix metadata test case: write yaml prebuild 2018-11-29 21:53:37 +01:00
Michael Pöhn
d0a129c216 add test for parsing build field prebuild as string 2018-11-29 21:53:37 +01:00
Michael Pöhn
942de28fa5 yaml metadata: split prebuild build field to list 2018-11-29 21:53:37 +01:00
Hans-Christoph Steiner
57556aceee remove redundant open() arg: encoding='utf8'
By default, open() returns a str:
https://docs.python.org/3/library/functions.html#open

By default, str is UTF-8:
https://docs.python.org/3/library/stdtypes.html#str

This used to matter on Python 2.x, but this code is 3.x only now.
2018-10-19 15:01:34 +02:00
Hans-Christoph Steiner
bfdf581201 import: use valid placeholder values for versionCode/versionName
fdroid/fdroidserver!559
closes fdroid/fdroidserver#548
2018-10-10 16:02:34 +02:00
Michael Pöhn
67e46694d3 add test for allowing to write placeholder values to yaml metadata files 2018-10-10 16:02:34 +02:00
Hans-Christoph Steiner
05be4bc814 Merge branch '471-lint-check-unknown-keys' into 'master'
check for unknown app fields and build flags when parsing yml

Closes #471

See merge request fdroid/fdroidserver!554
2018-08-14 10:50:59 +00:00
Michael Pöhn
6a06944945 better exception and waring message when yaml metadata checks fail; build flags check optional 2018-08-14 12:31:12 +02:00
Michael Pöhn
e83221eb03 improve yaml build flag check cost; added unit test for unknown app fields, builds flags checks when parsing yaml 2018-08-14 12:31:12 +02:00
Michael Pöhn
7d5f4d3ab9 added cwd setup boiler-plate to *.TestCase where missing 2018-08-14 10:34:13 +02:00
Hans-Christoph Steiner
5b22ff7dc6 tests: use standard dir setup so all tests start in same dir 2017-11-30 17:32:53 +01:00
Hans-Christoph Steiner
30b2f5a48a build: sort fdroid build --all by most recently changed first 2017-11-30 17:32:53 +01:00
Hans-Christoph Steiner
17efa13183 tests: pre-set failfast as reminder of a handy time saver 2017-10-25 23:01:25 +02:00
Hans-Christoph Steiner
df99c85ca6 support manually adding per-build antiFeatures in metadata
For cases like the OpenVPN vuln that was recently announced, it is useful
for fdroiddata maintainers to be able to mark builds that have known
vulnerabilities.
2017-07-06 12:25:48 +02:00
Michael Pöhn
071ad46b55 moved some yaml test files into a separate forlder to avoid conflicts with other test cases 2017-07-04 13:35:05 +02:00
Michael Pöhn
fc070ee657 refactored yaml tests into metadata test case; fixed typo 2017-07-04 11:51:08 +02:00