This was introduced in cd23bb933c
apparently to work around problems in vagrant 1.3.5.
This seems not to be necessary any more, I have never had a problem with
those sleeps removed.
We never allow git via SSH or password/key access, and right now, this
causes things to hang forever. This sets things up to fail quickly
with invalid ssh connections.
BatchMode=yes - passphrase/password querying will be disabled.
StrictHostKeyChecking=yes - never automatically prompt, or add host keys to
the ~/.ssh/known_hosts file, and refuse to connect to hosts whose host key
has changed.
If a git fetch/clone/submodule URL points to gitlab, github, bitbucket, etc
and that repo does not exist any more, those services will prompt the user
for a username/password so that the service can check if its a private
repo. Private repos show up the same as non-existent repos. This employs
two techniques for making sure that git never waits at those prompts. It
instead should just fail immediately. The buildserver has been hanging on
these prompts forever, until manually killed. This change will apply to
updates both on the buildserver host, and the buildserver guest vm.
This uses the "insteadOf" git config option to rewrite URLs to always use
HTTPS and then include a fake username/password so that git will use those
in the prompts and fail immediately. This trick has been in use on the
verification server for a long while and has been working well. It has
also been used on jenkins.debian.net in the host.
https://f-droid.org/en/docs/Verification_Server/
It also includes GIT_TERMINAL_PROMPT, which also prevents the bad behavior,
which was added in git 2.3.
https://github.com/blog/1957-git-2-3-has-been-released
Debug logging is not saved in buildserver logs. Also move this line into
build.py's main() to avoid duplication: common.setup_vcs(app) may be
called from parse_metadata() too.
Since this code was not setting the "create system" and "compress type",
Python uses it's defaults. Those will be different than what the Android
tools produces if this is run on UNIX. The Android tools uses the bare
bones "Windows" ZIP format, e.g. no permissions, etc.
For example:
https://verification.f-droid.org/eu.siacs.conversations_234.apk.diffoscope.html
This came about testing on OSX, where there are often multiple versions of
the JDK installed. This was choosing the oldest version. It should
choose the most recent version.
This reverts commit 5f5d3ea896.
These new scanner rules for JARs causes most builds on fdroiddata to
break. For example, at least the past 20 releases of
org.fdroid.fdroid. fdroiddata needs to be cleaned up and prepared for
this before we can include 5f5d3ea896
I'm running fdroid scanner on the whole archive now, I'll post the log
to an issue once its done. Ignoring test files would eliminate some
of these build failures, at least for most org.fdroid.fdroid builds.
refs #!325
This fixes a problem when there is a broken symlink with the name
AndroidManifest.xml which will then lead to a crash when trying to open
it for trying to remove the debuggable flag.
Wow, this is hacky and weird, but its been there for a long time. I
don't know if its even relevant anymore, but is isolated so it
shouldn't hurt anything. Plus there is a test for it in
tests/complete-ci-tests, so it seems worth fixing.
Running `fdroid verify` I was seeing FDroidExceptions from verify.py:98
that had a tuple rather than a string.
Traceback (most recent call last):
File "/home/hans/code/fdroid/server/fdroid", line 152, in <module>
main()
File "/home/hans/code/fdroid/server/fdroid", line 128, in main
mod.main()
File "/home/hans/code/fdroid/server/fdroidserver/verify.py", line 98, in main
logging.info("...NOT verified - {0}".format(e))
File "/home/hans/code/fdroid/server/fdroidserver/exception.py", line 22, in __str__
ret += "\n==== detail begin ====\n%s\n==== detail end ====" % ''.join(self.detail).strip()
TypeError: sequence item 1: expected str instance, HTTPError found
An app may not have a top-level build.gradle file, while Gradle still
creates top-level .gradle/ directory. When build.gradle is absent,
fdroidserver will not remove .gradle/, scanner will find binaries in it
and fail the whole build.
Fix this by also checking for settings.gradle file (should always exist
in the top-level directory) in addition to build.gradle.
This fixes all the bugs I could find that prevented fdroid from
handling files with spaces in them. This is more important now that
fdroid supports random media files, and Repomaker
apksigner v0.7 (build-tools 26.0.1), Google made it require that the
AndroidManifest.xml was present in the archive before it verifies the
signature. So this needs to stick with the jarsigner hack for JARs.
os.mknod() fails on OSX with:
Traceback (most recent call last):
File "/Users/travis/build/fdroidtravis/fdroidserver/tests/../fdroid", line 154, in <module>
main()
File "/Users/travis/build/fdroidtravis/fdroidserver/tests/../fdroid", line 130, in main
mod.main()
File "/Users/travis/build/fdroidtravis/fdroidserver/fdroidserver/update.py", line 1768, in main
common.write_to_config(config, 'repo_keyalias', config['repo_keyalias'])
File "/Users/travis/build/fdroidtravis/fdroidserver/fdroidserver/common.py", line 2328, in write_to_config
os.mknod(cfg)
PermissionError: [Errno 1] Operation not permitted
Since !335 changed the long standing behavior, now allowing `fdroid` to
work without a config.py being present, this message should be quite
visible in case people are expecting `fdroid` to fail when there is no
config.py present.
This allows all the text to be localized via Weblate. This is a quick
overview of all the strings, but there are certainly some that were left
out.
closes#342
There were multiple conventions used in the code, but mostly it was already
using the convention from the docs, so this converts things to using that
convention:
https://docs.python.org/3/library/os.html#os.walk
This commit adds support for new gradle plugin 3.0 output directories.
The new structure looks like this:
build/outputs/apk/<flavour1><Flavour2>/release/*.apk
Note the capitalization on the different flavour components.
So if we build a specific flavour combination we add this combination to
the path where we look for an output .apk.
Closes#363
Based on !320 by: Michel Le Bihan <michel@lebihan.pl>
Signed-off-by: Marcus Hoffmann <bubuiic@aol.com>
There is a hardcoded template in update.py, and there is also the
possibility for the user to create a template.yml. This tests both of them
and cleans up the related code a bit.
#352!310
With the new localization support, the name/summary/description in the
metadata file becomes the global override. So most apps are not going to
have those fields present in their metadata file. This fixes the index.xml
generation to fall back to the localized versions of those fields when they
are not set in the metadata field.
https://forum.f-droid.org/t/what-has-happend-to-osmand
The Builds entries in metadata/ files do not easily say whether a
given APK is in the repo/ or the archive/. So it should also try to
download the official APK from the archive/ when verifying.
This makes `fdroid server update` fail if pushing to one of the git mirrors
fails. This is what happens if the other methods fail, e.g. rsync or S3.
closes#347
gitlab serves raw files from a CDN, so its appropriate to use the raw URL.
@pserwylo @grote and I discussed it and found a reference, but I can't find
that reference now.
Since the client will try the next mirror if one fails, it makes sense to
include both the gitlab raw and gitlab pages URLs to the mirror. The
gitlab pages deploy process is still a bit flaky anyway.
git hosts like github, gitlab, bitbucket usually allow 1 gig repos. This
changes the git mirroring behavior to keep the history until the repo hits
1 gig. Keeping history makes updates a lot faster, since the whole repo
does not need to be pushed on each update.
It turns out it is error prone to `git push` to a non-bare git repo. For
the offline signing machine, the git remote needs to be a regular git repo
in a directory on a thumbdrive so that once the thumbdrive is plugged into
an online machine, that git repo can be transferred to the online machine.
Since the mirror URLs are per repo section (repo/archive), the mirror URLs
must include the repodir at the end. This was missing for servergitmirrors
found by @cde when working on fdroidclient#35
If ruamel.yaml is not available, this will fallback to using PyYAML. This
also adds some blank fields to the newly created template to make it easy
for human editors to fill in.
closes#343
APKs can now use XML files for vector graphics like the app icon. `aapt`
returns the XML file by default, and perhaps also androguard. This
checks if the icon is an XML file, and if so, it tries to find a PNG in
the APK with the same name and density to use instead
closes#322
This should also ultimately make the XML file available as an icon source
as well fdroidclient#1091
For cases like the OpenVPN vuln that was recently announced, it is useful
for fdroiddata maintainers to be able to mark builds that have known
vulnerabilities.
Normally, just 'repo/' is created by default, e.g. `fdroid init`. If APKs
are dumped into 'repo/', then have invalid signatures, then they'll be
automatically moved to 'archive/', which therefore needs to exist.
The new policy is to move APKs with invalid signatures to the archive,
and only add those APKs to the archive's index if they have valid MD5
signatures.
closes#323closes#292
In April 2017, Oracle's jarsigner and Google's apksigner both switched to
considering any APK signature that uses MD5 as unsigned. Any old build
is likely to have a MD5 signature. This sets up the archive as the only
place where these "disabled algorithms" are allowed in the repo, and
marks any APK signed by a "disabled algorithm" as having a "known
vulnerability"
This also now automatically moves APKs with invalid signatures to the
archive section.
#323
The original logic was checking keepversions against the len() of ALL the
APKs in the repo/archive. The correct thing is to check against the
number of APKs available for the given packageName/appid.
closes#166
This exploit is old, and was fixed in 4.4. But it was easy to exploit,
so it is still worth scanning for it. It is also easy to scan for, since
valid APKs should not have files with duplicate names. In theory, this
could look for duplicate file names for any file, but this limits the
false positives by only checking names of files related to executing code.
fdroidclient#40
`fdroid update` crashed for apps that only had screenshots but no graphics
or localized texts because destdir was not being set in that case. This
fixes that and adds a test case.
closes#320!286
Since the Summary: and Description: in the metadata file has the highest
priority of all the localized texts, adding blank versions means that
apps would always have blank Summary and Description even if the app has
those fields in the localized sections of fdroiddata and/or in the app's
source repo itself.
fdroiddata!2262
Since the Summary/Description can now be set in the app's source code, or
in fdroiddata/metadata/<packageName>/<locale>/*.txt, this lint check is
no longer valid. It is important to check whether these texts are empty,
but it'll require some thinking about how and where to best to that.
`fdroid update` will have access to all that data, but perhaps at that
point it is too late.
Also, the current text prioritization puts Summary/Description in the
.txt/.yml file at the highest priority, overriding every other copy,
including in fdroiddata/metadata/<packageName>/<locale> and in the app's
source code.
If working with a random grabbag of APKs, there can be all sorts of
issues like corrupt entries in the ZIP, bad signatures, signatures that
are invalid since they use MD5, etc. Moving these two checks later means
that the APKs can be renamed still.
This does change how common.getsig() works. For years, it returned
None if the signature check failed. Now that I've started working
with giant APK collections gathered from the wild, I can see that
`fdroid update` needs to be able to first index what's there, then
make decisions based on that information. So that means separating
the getsig() fingerprint fetching from the APK signature verification.
This is not hugely security sensitive, since the APKs still have to
get past the Android checks, e.g. update signature checks. Plus the
APK hash is already included in the signed index.
With a generic file, the file name is the only guaranteed name metadata
field. So if the name is not specified in the metadata, then the name
is set to the filename. This changes that so that the file extension is
stripped from that generated name.
This is useful for parsing APK files, which can include packageName,
versionCode, and optionally 7 char signing key ID (i.e. <sig>).
This also can set the packageName and versionCoe for non APK files, so
that it is easy to assign them to metadata files, and to allow for
upgrades by setting the versionCode in the filename.
Really, it is the fdroidclient parser of index.xml that fails, due to the
hardcoded expectation that there will only ever be a single APK for any
given versionCode. We keep index.xml backwards compatible for old
clients, and use index-v1.json to support new things. Having multiple
APKs that have the same packageName and versionCode will break the client
v0.103.* since that version uses index-v1.json, but still has the hard-
coded database parsing stuff.
#153
uses the standard package.name_123.apk. If that exists, it appends the
shasum. If that exists, then its a duplicate, so its deleted. This should
help @SergeWinters with his 12,000 APKs.
There are many APKs out in the wild that claim to be the same app and
version and each other, but they are signed by different keys. fdroid
should be able to index these, and work with them. This supports having
the developer's signature via reproducible builds, random collections of
APKs like repomaker, etc.
Graphics and localized text can now be stored in the package folders,
always in a folder that is named for the locale. The upstream developer
signature is also now stored, so that the upstream APK can be reproduced
even if they remove their APKs.
#291
fdroiddata!2229
fdroiddata!2224
fdroidclient#15
fdroidserver#174
This is some very messy logic built up since 2010. This will all go away
once we have a python3 version of androguard available.
The removed imports and `dir(APK)` is to silence pyflakes
closes#303
_ is a valid character for Java package names, so it should also work in
the repo file naming scheme. This makes it so it only splits the file
name based on the last _.
cgi.escape is deprecated in Python 3.x and has security issues:
https://bugs.python.org/issue26398
html.escape() differs from cgi.escape() by its defaults to quote=True:
s = html.escape( """& < " ' >""" ) # s = '& < " ' >'
This replaces the broken, custom code with the standard YAML lib.
In rewritemeta, do not call app.metadatapath since it will be deleted when
the dict is cleaned up for outputing. metadatapath is only used internally
and should not be written out.
closes#169
refs #290
This is a little omission. keys that are used in metadata/*.yml all start
with an UpperCase letter, but in fdroidserver, index-v1.json, and
fdroidclient, it is all camelCase with lowercase first letter. The keys
from the 'localized' section are currently never in metadata/*.yml, so
these keys never get downcase. This change will break fdroidclient
versions that do not also have this change, but since we're in alpha, that
should be fine.
If support for a 'localized' section is added to metadata/*.yml, then the
keys there should probably be UpperCase CamelCase to match the other keys.
glob != regexp, [A-Z-.@] means one of those chars is required in that
position, so the glob pattern was requiring a least 3 chars. Locales are
usually just two lower case letters, e.g. vi, de, ar.
Fastlane Supply, Triple-T Gradle Play Publisher, and many app stores
include the possibility to specify a website for the author, as distinct
from the website for the app.
closes#204
This uses the "What's New" entry for the CurrentVersionCode and includes it
as the current WhatsNew metadata for the App class.
Things like fastlane supply and Google Play support a "What's New" entry
per-APK, but fdroidclient does not current use anything but the current
version of this data. Right now, it seems we probably only want to have
the latest WhatsNew in the index to save space.
In theory, we could make the WhatsNew data structure follow the structure
of fastlane/Play, but that would quite a bit of complexity for something
that might never be used.
fdroidclient#910
This helps with uploading faster when hitting the public API rating
limiting since it will not hit the API if there is already a matching
response on the local filesystem. This also provides a source of data for
things like repomaker to diplay to users.
This restructures the virustotal uploading so that first checks whether
virustotal already has the file, and only if not does it upload it. This
also handles the public API rate limiting, which returns an HTTP 204. This
will now try again until it succeeds, even when rate limited. Instead of
just getting the list of files from the filesystem, this reads the index-v1
which also already has the SHA256 in there. virustotal also uses SHA256 as
a unique ID for files.
libcloud has been flaky for years, but it is also widely available and is
more pythonic. So this adds the option for using `s3cmd sync` if it is
available. `s3cmd sync` acts like rsync for uploading to S3. So this code
emulates the rsync uploading code in `fdroid server update`.
Instead of each time deleting the whole local git-mirror repo, and
recreating it, this just deletes the .git/ dir, then lets the rsync in
_local_sync() handle deleting anything that should no longer be in the repo.
