1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-04 22:40:12 +01:00
Commit Graph

5063 Commits

Author SHA1 Message Date
Hans-Christoph Steiner
8215bbbe9a Merge branch 'checkupdates' into 'master'
tighten up checkupdates

See merge request fdroid/fdroidserver!474
2018-03-06 19:20:38 +00:00
Hans-Christoph Steiner
70d9633555 build/checkupdates/update: log current fdroiddata commit to wiki 2018-03-05 21:49:09 +01:00
Hans-Christoph Steiner
86907d2ebf checkupdates: exit with error if fdroiddata git repo is dirty
One key security property of the F-Droid ecosystem is that the sensitive
code is all stored forever in git repos and source tarballs.  That means
we can easily go back and see if there where exploits and where they came
from.  Therefore, checkupdates should require everything in fdroiddata be
committed to git before running.

This provides --allow-dirty to override that behavior.
2018-03-05 21:49:09 +01:00
Hans-Christoph Steiner
67d386d925 Merge branch 'more-security-fixes' into 'master'
More security fixes

See merge request fdroid/fdroidserver!471
2018-03-05 09:10:57 +00:00
Hans-Christoph Steiner
8f30c892c5 VercodeOperation: only allow simple math expresssions and %c 2018-03-05 09:45:58 +01:00
Hans-Christoph Steiner
6876e28bb4 hg: use /bin/false to clarify that it is an executable 2018-03-05 09:45:58 +01:00
Hans-Christoph Steiner
6cd8f2ffea SVN: only allow redirects to HTTPS
"SVN follows HTTP 301 redirects to svn+ssh:// URLs. As a result, an
innocent looking HTTP URL can be used to trigger a Command Execution with a
301 redirect."
https://blog.recurity-labs.com/2017-08-10/scm-vulns.html#third-round-svn-and-mercurial

I scanned fdroiddata and found no suspicious redirects.  Here's how:

grep -A1 '^Repo *Type: *git-svn' *.txt *.yml| sed -n 's,.*Repo:\(.*\),\1,p' > /tmp/urls.txt

import requests
with open('/tmp/urls.txt') as fp:
    for line in fp:
        try:
            r = requests.head(line.strip())
            print(r.status_code, line)
        except requests.exceptions.SSLError:
            print('SSLError', line)
2018-03-05 09:45:58 +01:00
Marcus
654b3cb9dc Merge branch 'lint_fix' into 'master'
lint: fix update check data https check

See merge request fdroid/fdroidserver!473
2018-03-02 22:59:26 +00:00
Marcus Hoffmann
c81d5da953
lint: fix update check data https check
The urlver field can be '.', this was not considered in
26bfd7fb28.
2018-03-02 23:56:49 +01:00
Marcus
c35b120ff2 Merge branch 'tighten-up-UpdateCheckData' into 'master'
Tighten up UpdateCheckData

See merge request fdroid/fdroidserver!470
2018-03-02 11:45:53 +00:00
Hans-Christoph Steiner
7da0747849 checkupdates: require UpdateCheckData has valid HTTPS URL 2018-03-01 23:51:36 +01:00
Hans-Christoph Steiner
26bfd7fb28 lint: require UpdateCheckData to contain only valid HTTPS URLs 2018-03-01 23:51:27 +01:00
Hans-Christoph Steiner
1c9bc32bf6 lint: tighten up HTTPS checks on URLs 2018-03-01 23:38:56 +01:00
Hans-Christoph Steiner
4197a4a64a Merge branch 'master' into 'master'
update: make icon extraction less dependent on aapt

Closes fdroid-website#192

See merge request fdroid/fdroidserver!469
2018-02-28 21:42:06 +00:00
Hans-Christoph Steiner
498ea5d609 lint: ban all dangerous HTML tags
* https://en.wikipedia.org/wiki/HTML_sanitization
* https://asostack.com/enhance-your-google-play-store-description-with-rich-formatting-and-emojis-5f50ff354e5f
2018-02-27 12:09:54 +01:00
Hans-Christoph Steiner
b2ca49b26c update: make icon extraction less dependent on aapt
For androguard, @thezero already developed a way to get all the icons after
only extracting the icon name.  So this uses that for the aapt-based scans
also, to make them less brittle.

This should fix the problem where `fdroid update` was choosing the XML icon
for apps that include one, like NewPipe.

closes fdroid/fdroid-website#192
2018-02-26 23:43:42 +01:00
Hans-Christoph Steiner
01a73071c7 gitlab-ci: set metadata_v0 test to use 1.0.2 as the baseline 2018-02-23 22:48:44 +01:00
Hans-Christoph Steiner
0fa50ebcb2 rewritemeta: fix proper_format() so lint works with .yml files 2018-02-23 22:43:03 +01:00
Hans-Christoph Steiner
5a6a51a29e Merge branch 'allow-dashes-and-underscores-in-signature-file-names-when-checking-for-reproducability' into 'master'
allow dashes and underscores in signature file names when checking for reproducability

See merge request fdroid/fdroidserver!468
2018-02-22 23:23:50 +00:00
Michael Pöhn
8cca83aec4 allow dashes and underscores in signature file names when checking for reproducability 2018-02-22 23:30:42 +01:00
Hans-Christoph Steiner
2f563f533b bump to 1.0.2 to placate pypi
I mistakenly uploaded the dist tarball to pypi without the PGP signature.
So I deleted the release, thinking I could reupload it.  It is not possible:
https://github.com/pypa/packaging-problems/issues/74

So this is really just a bump so I can reupload to pypi.
2018-02-22 21:23:19 +01:00
Hans-Christoph Steiner
06fb855a27 common: tighten up regexs when searching for version name/code and appid
This should have less of a change of matching bad things.
thanks to @stf for the report.  I ran tests comparing the original vs these
new patterns, and it was a 100% match. So at least it didn't make things
worse.

Here's the test script:
#!/usr/bin/env python3

import os
import re


old_vcsearch_g = re.compile(r'''.*[Vv]ersionCode[ =]+["']*([0-9]+)["']*''').search
old_vnsearch_g = re.compile(r'.*[Vv]ersionName *=* *(["\'])((?:(?=(\\?))\3.)*?)\1.*').search
old_psearch_g = re.compile(r'.*(packageName|applicationId) *=* *["\']([^"]+)["\'].*').search
new_vcsearch_g = re.compile(r'''.*[Vv]ersionCode\s*=?\s*["']*([0-9]+)["']*''').search
new_vnsearch_g = re.compile(r'''.*[Vv]ersionName\s*=?\s*(["'])((?:(?=(\\?))\3.)*?)\1.*''').search
new_psearch_g = re.compile(r'''.*(packageName|applicationId)\s*=*\s*["']([^"']+)["'].*''').search

old = re.compile(r'.*(packageName|applicationId) *=* *["\']([^"]+)["\'].*').search
new = re.compile(r'''.*(packageName|applicationId)\s*=*\s*["']([^"']+)["'].*''').search


for root, dirs, files in os.walk('build'):
    for f in files:
        if f.endswith('.gradle'):
            with open(os.path.join(root, f)) as fp:
                for line in fp:
                    for old, new in ((old_vcsearch_g, new_vcsearch_g),
                                     (old_vnsearch_g, new_vnsearch_g),
                                     (old_psearch_g, new_psearch_g)):
                        found_old = old(line)
                        found_new = new(line)
                        oldresult = None
                        newresult = None
                        if found_old or found_new:
                            if found_old:
                                oldresult = found_old.groups()
                                #print('OLD', oldresult)
                            if found_new:
                                newresult = found_new.groups()
                                #print('NEW', newresult)
                            if oldresult != newresult:
                                print('--------------------------------')
                                print(f, oldresult, newresult)
2018-02-22 21:15:41 +01:00
Hans-Christoph Steiner
e9320017b4 Bump to 1.0.1! 2018-02-22 20:01:50 +01:00
Hans-Christoph Steiner
991b78660c Merge branch 'weblate' into 'master'
Weblate

See merge request fdroid/fdroidserver!467
2018-02-22 14:59:32 +00:00
Hans-Christoph Steiner
4b63f4ded6 Weblate 2018-02-22 14:59:32 +00:00
Hans-Christoph Steiner
f55bb62a56 Merge branch 'androguard-fixes' into 'master'
Androguard fixes

See merge request fdroid/fdroidserver!466
2018-02-22 14:39:34 +00:00
Hans-Christoph Steiner
e75bf70be6 signatures: future-proof fetching app ID info from APK
We're not using platformBuildVersionName and it might go away just like it
appeared: with no good reason or announcement.
2018-02-22 15:08:55 +01:00
Hans-Christoph Steiner
88e24dc4e3 update: switch to improved androguard detection 2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner
f420a037d5 gitlab-ci: remove apt-get progress dumps from build log 2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner
0e9252db37 gitlab-ci: try to download PPA keys until they succeed
This download occasionally fails, so this keeps retrying till it succeeds.
The CI job has a time limit, so no need to figure out an exit condition.
2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner
089712c012 tests: do not automatically run install.TestCase, its troublesome
`fdroid install` is rarely used, if at all, and the test frequently fails
for no reason in gitlab-ci, because it can't start the adb server.
2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner
89498208fc gitlab-ci: test against latest build-tools 27.0.3 2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner
11bed21071 gitlab-ci: androguard from Debian/unstable til it stabilizes 2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner
52b3436ff6 make is_apk_and_debuggable() default to using androguard before aapt 2018-02-22 15:08:53 +01:00
Hans-Christoph Steiner
847bbb6e43 init: do not try to find aapt if androguard is available 2018-02-22 13:38:02 +01:00
Hans-Christoph Steiner
aa4f54bf18 update: include implied permissions when using androguard
`aapt dump badging` includes these when listing uses-permissions:
https://github.com/androguard/androguard/pull/428
2018-02-21 12:34:54 +01:00
Hans-Christoph Steiner
63d4d46291 update: 'features' list only includes required features
The F-Droid index 'features' list is not the same as what is in the
AndroidManifest.xml.  It only includes "required" features, for example.
2018-02-20 17:08:55 +01:00
Hans-Christoph Steiner
9f553186e8 gitlab-ci: switch debian_testing to use androguard 2018-02-20 16:30:06 +01:00
Hans-Christoph Steiner
008110889a update: fix crash from missing file extension in extracted icon
I missed this in 40fac10ebc, yay tests!
2018-02-20 16:29:52 +01:00
Hans-Christoph Steiner
c679b5b144 Merge branch 'use-androguard-first' into 'master'
use androguard first

Closes #236

See merge request fdroid/fdroidserver!465
2018-02-19 18:53:42 +00:00
Hans-Christoph Steiner
44ebf701e1 Merge branch 'support-xml-icons' into 'master'
Support XML icons

Closes #392

See merge request fdroid/fdroidserver!464
2018-02-19 16:58:28 +00:00
Hans-Christoph Steiner
d1ded7f64d update: use androguard by default if it is available
closes #236
2018-02-15 14:28:48 +01:00
Hans-Christoph Steiner
40fac10ebc update: extract and store XML icons
These can then be used by the client.

#344
#392
2018-02-15 14:28:45 +01:00
Hans-Christoph Steiner
5281228ea5 update: switch to new androguard v3.1 API based on lxml
apkobject.get_android_manifest_xml() used to return a xml.dom.minidom
object, now it returns an lxml.etree.Element object.
2018-02-15 14:28:45 +01:00
Hans-Christoph Steiner
5713b54e0b update: find alternate PNGs for apps that have an XML app icon
Apps can now use an XML icon, but if the app supports older Android
versions, it'll also contain PNG versions of the same icon.  This finds
those PNGs and uses them instead.

#344
closes #392
fdroiddata#913
2018-02-15 14:28:45 +01:00
Hans-Christoph Steiner
699b3e4c69 update: fix min/target/max edge case parsing with androguard
In order to test that aapt defaults minSdkVersion to 3, I ran this script
then compared the output with meld:

cd $ANDROID_HOME/build-tools
for d in *.*; do echo $d; $ANDROID_HOME/build-tools/$d/aapt dump badging /home/hans/code/fdroid/server/tests/repo/com.politedroid_3.apk > /tmp/${d}.txt; done
meld /tmp/17.0.0.txt /tmp/26.0.2.txt /tmp/27.0.3.txt
2018-02-15 14:28:45 +01:00
Marcus
e6208da7ca Merge branch 'fix-excessive-ndk' into 'master'
makebuildserver: do not provision excessive NDKs

See merge request fdroid/fdroidserver!463
2018-02-14 23:54:00 +00:00
Marcus
997289d9ca Merge branch 'fdroid-deploy' into 'master'
rename `fdroid server` to `fdroid deploy` and deprecate 'init'

Closes #264

See merge request fdroid/fdroidserver!462
2018-02-14 21:51:11 +00:00
relan
2c0e430a2d makebuildserver: do not provision excessive NDKs
When a new minor version of an NDK is released, it replaces an older one,
e.g. r16 with r16b (see commit 6f295cb). But old NDK package remains in
the cache and provisioning script unpacks it too as it matches the mask.

Fix NDK provisioning to unzip only while-listed versions.
2018-02-14 09:11:40 +03:00
Hans-Christoph Steiner
1725e09f7e rename fdroid server to fdroid deploy and deprecate 'init'
`fdroid server init` is has not been needed for a long time.  And 'server'
is the only subcommand that has its own subsubcommands.  This turns it into
only `fdroid deploy`, which does what `fdroid server update` does. This
also changes the bash completion to use `fdroid deploy`.  But the old
`fdroid server update` and `fdroid server init` commands remain working.

closes #264
2018-02-13 12:54:48 +01:00