1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-18 20:50:10 +01:00
Commit Graph

8565 Commits

Author SHA1 Message Date
est31
d6da0a283d Makebs: add build tools 25.0.2 2016-12-17 23:36:48 +01:00
Boris Kraut
fd51f7fab0 docs: switch to spdx license list 2016-12-17 11:00:40 +01:00
Hans-Christoph Steiner
7c8823e94e Merge branch 'feature/dscanner' into master
dscanner - drozer scanner work.

closes !187
2016-12-06 14:03:34 +01:00
Kevin C. Krinke
df27bae6a0 dscanner - Drozer based post-build dynamic vulnerability scanner command
* New command `dscanner`, enables one to scan signed APKs with Drozer
 * Drozer is a dynamic vulnerability scanner for Android
 * Drozer runs in a emulator or on-device, this new `dscanner` command...
  * starts a docker image with Drozer and the Android Emulator pre-installed,
  * loads the signed APK into the emulator
  * activates Drozer automated tests for the APK
  * gathers the report output and places it next to the original APK
 * The Drozer docker image can be:
  * cached locally for re-use (just don't run --clean*)
  * retrieved from dockerhub.com for more efficient runtime
  * or be built from scratch (in the new "./docker" directory)
 * New "Vulnerability Scanning" documentation section (run gendocs.sh)
2016-12-06 14:00:44 +01:00
Hans-Christoph Steiner
f439266303 Merge branch 'collection-fixes-and-KnownVuln' into 'master'
add KnownVuln anti-feature, and other fixes

This is a bit of a random collection of things that I have added in the process of working on the drozer/scanner support, YAML support, etc.  The only new things are:

* adding new AntiFeature for security issues: `KnownVuln`
* removing broken, incomplete XML metadata support

Everything else included are just code improvements.  This also includes the first check for `KnownVuln`, which is a scanner to check whether custom OpenSSL binaries in apps are not old with known vulnerabilities.  `KnownVuln` will then ultimately be used for things like the drozer scanner !187

See merge request !189
2016-12-06 12:37:11 +00:00
Hans-Christoph Steiner
e37be9b063 fix fdroid build with non-git repos
HEAD is really only in git.  This was introduced in
a4e4310803
2016-12-05 20:19:56 +01:00
Hans-Christoph Steiner
b1a943d562 convert metadata test dumps to YAML for easy comparison
When making code changes related to the metadata parsing, it is useful to
see how the internal format has changed by seeing the differences in the
dump files.  Those files are currently in the binary .pickle format.  This
just straight converts them to YAML, which is a text format, so that normal
diff tools work to see changes.

The dump files are named .yaml instead of .yml since .yml is used for hand-
edited YAML files for fdroiddata/metadata, while these dump files here are
a human readable form of a Python pickle.
2016-12-05 20:16:54 +01:00
Hans-Christoph Steiner
ce3efe4168 remove support for XML app metadata, its broken
JSON and YAML are very closely related, so supporting both of them is
basically almost no extra work.  Both are also closely related to how
Python works with dicts and pickles. XML is a very different beast, and its
not popular for this kind of thing anyway, so just purge it.
2016-12-05 20:16:54 +01:00
Hans-Christoph Steiner
b91cdffe17 App.get_last_build() method to replace duplicated code
Look @mvdan, I added a method to the App class!
2016-12-05 20:16:54 +01:00
Hans-Christoph Steiner
3db2c9869c do proper checking of versionCode value
versionCode is defined as a Java Integer, so any value between -2147483648
(Integer.MIN_VALUE) and 2147483647 (Integer.MIN_VALUE) is valid, including
0.

https://developer.android.com/guide/topics/manifest/manifest-element.html#vcode
2016-12-05 20:16:54 +01:00
Hans-Christoph Steiner
98297278bd check all APKs for old versions of OpenSSL
This scans all APKs for old versions of OpenSSL libraries that are known to
be vulnerable to issues, or fully unsupported.

This really should be implemented as a per-APK AntiFeature, so that it can
apply to any version that is vulnerable.  Since AntiFeatures are currently
only per-App, this instead sets the AntiFeature only if the latest APK is
vulnerable.

Google also enforces this:
https://support.google.com/faqs/answer/6376725?hl=en

apk['antiFeatures'] has the first letter small, since all build fields
start with a lowercase letter.  app.AntiFeatures has the first
uppercase since all App fields are that way.
2016-12-05 20:16:54 +01:00
Hans-Christoph Steiner
95656e6c61 check aapt version to make sure its new enough #236
Since `fdroid update` parses the output of aapt, and since aapt's output
format changes in non-compatible ways, test to make sure that the version
of aapt is new enough to prevent mystery stacktraces.  This only prints a
warning since in many cases, the old version will work just fine.
2016-12-05 20:16:12 +01:00
Ciaran Gultnieks
32182867ad Merge branch 'master' into 'master'
Add gradle 3.2.1

See merge request !192
2016-12-02 16:09:22 +00:00
est31
d39f54f04d Add gradle 3.2.1 2016-12-02 08:16:04 +01:00
Daniel Martí
6b2aa444d3 Merge branch 'buildtools-25.0.1' into 'master'
makebs: add buildtools 25.0.1

Please verify hash before merging.

See merge request !191
2016-12-01 12:40:25 +00:00
Boris Kraut
51083fdeb4 makebs: add buildtools 25.0.1 2016-12-01 08:44:47 +01:00
Hans-Christoph Steiner
a5cc4d74fc Merge branch 'master' into 'master'
Add ndk 13b

See merge request !190
2016-11-26 08:02:42 +00:00
est31
ddefec33cd Add ndk 13b 2016-11-25 21:18:37 +01:00
Hans-Christoph Steiner
d7ec321198 Merge branch 'fdroid-yml-builds' into 'master'
builds straight from source repo using .fdroid.yml

The overarching theme of the merge request is allowing _.fdroid.yml_ to be included in an app's source repo, then letting `fdroid build` build the app straight out of the git repo without requiring a setup like _fdroiddata_ (e.g. _config.py_, _metadata/packagename.txt_, etc.).  _fdroiddata_ repos can then include source repos with a _.fdroid.yml_ by having _metadata/packagename.txt_ that includes just:

```
Repo Type:git
Repo:https://gitlab.com/upstream/app.git
```

Any other metadata fields that are included in _metadata/packagename.txt_ will override what is in _.fdroid.yml_, giving the repo manager the final say about what is included in their repo.  This setup provides a number of benefits:

* CI systems like jenkins, travis, gitlab-ci can build from _.fdroid.yml_
* very easy to start building apps using `fdroid build`, no separate repo needed
* some maintenance can be offloaded to the upstream dev

See merge request !184
2016-11-24 14:04:09 +00:00
Hans-Christoph Steiner
6cadefbe4e Merge branch 'patch-1' into 'master'
docs: remove one-line requirement for links

See merge request !188
2016-11-24 08:38:04 +00:00
Boris Kraut
d07b88d9f1 docs: remove one-line requirement for links 2016-11-24 07:30:03 +00:00
Hans-Christoph Steiner
28ea33b8d1 if building directly in git repo, use file path for remote
When a git repo has a .fdroid.yml file in it, and `fdroid build` is run
directly in that git repo, then this uses the file path as the remote for
the git repo in build/appid that is actually built.  That makes it possible
to run builds of commits that are only local, and makes things a whole lot
faster.
2016-11-16 23:28:03 +01:00
Hans-Christoph Steiner
c41daf1e7e ignore files created by tests 2016-11-16 23:28:03 +01:00
Hans-Christoph Steiner
a4e4310803 allow metadata to be embedded in source repos via .fdroid.yml
This allows a source repo to include a complete metadata file so that it
can be built directly in place using `fdroid build`.  If that app is then
included in fdroiddata, it will first load the source repo type and URL
from fdroiddata, then read .fdroid.yml if it exists, then include the rest
of the metadata as specified in fdroiddata, so that fdroiddata has
precedence over the metadata in the source code.

This lets `fdroid build` apps without having a whole fdroiddata setup, but
instead just directly in place in the source code.  This also lets devs
optionallu maintain the fdroid metadata as part of their app, rather than
in fdroiddata without loosing any control.  This should make it easier to
spread around the maintenance load.
2016-11-16 23:28:03 +01:00
Hans-Christoph Steiner
b4a39ee272 switch import test to custom, small test app
The test project should be moved to https://gitlab.com/fdroid/ci-test-app
2016-11-16 23:28:03 +01:00
Hans-Christoph Steiner
f8dca60a20 convert comments above functions to python docstrings
This is how to write per-function comments.
https://www.python.org/dev/peps/pep-0257/
2016-11-16 23:28:03 +01:00
Hans-Christoph Steiner
1f55a40caa properly parse build metadata list types like gradle=
Something like `gradle: yes` in YAML will be parsed as a boolean, since
'yes' is officially defined as a boolean true in YAML.  For metadata fields
that need to be lists, this needs to be converted.  Same goes for a single
string like `gradle: customFlavor`.
2016-11-16 23:28:03 +01:00
Hans-Christoph Steiner
3fb4cba178 include version, commit, and android tools versions in local log
This includes more info to help track down problems with reproducible
builds, like the specific version being built, and which exact versions of
the Android SDK and NDK were used.
2016-11-16 23:28:03 +01:00
Hans-Christoph Steiner
ad2b9b99c2 log versions of all installed Android SDK/NDK components
Any variation in the Android tools used to build an APK can cause the build
to be unreproducible.  To help troubleshoot these times, this posts the
installed versions of the Android SDK and NDK components to the lastbuild
log, for the long term record.

refs #148
2016-11-16 23:28:03 +01:00
Hans-Christoph Steiner
2237b6f7d3 Merge branch 'makebs' into 'master'
makebs updates

See merge request !185
2016-11-16 21:09:17 +00:00
Daniel Martí
8cdf1b0223 makebs: add gradle 3.2 2016-11-15 21:07:35 +00:00
Daniel Martí
483338843f makebs: update to support repo r40 2016-11-15 20:56:04 +00:00
Daniel Martí
82b1d7ad14 all: make newer pycodestyle happy
Apparently the "two empty lines" rule is now stricter.
2016-11-15 20:55:06 +00:00
Daniel Martí
8ee13a47e4 Merge branch 'node' into 'master'
buildserver: install nodejs

Install nodejs to allow webapps to be build (in the future...). 


Ref: https://gitlab.com/fdroid/fdroidserver/issues/60

See merge request !183
2016-11-08 21:33:38 +00:00
Boris Kraut
c788774d85 buildserver: install nodejs 2016-11-07 22:54:47 +01:00
Daniel Martí
ffb6c97deb Merge branch 'build-FPE-update-zip' into 'master'
Build Privileged Extension OTA update.zip using `fdroid build`

This allows `fdroid build` to build the OTA update ZIP file for F-Droid Privileged Extension, so that the official releases can be built and distributed via the normal F-Droid channels.   This is related to #233 

Ultimately the client will also have to be updated to allow it to handle the non-APK files.

See merge request !181
2016-11-07 15:11:14 +00:00
Hans-Christoph Steiner
fb0c824d3c wp-fdroid: show GPG Signature link for source tarballs
Now that source tarballs can be GPG-signed, the website should also display
a link to fetch them.
2016-11-07 14:53:01 +01:00
Hans-Christoph Steiner
56d51fcd6b gpg-sign all valid files in the repo, including source tarballs
This makes sure there is a GPG signature on any file that is included in
the repo, including APKs, OBB, source tarballs, media files, OTA update
ZIPs, etc.  Having a GPG signature is more important on non-APK files since
they mostly do not have any signature mechanism of their own.

This also adds basic tests of adding non-APK/OBB files to a repo with
`fdroid update`.

closes #232
2016-11-07 14:53:01 +01:00
Hans-Christoph Steiner
84e09cd2a2 allow arbitrary build products, not only APKs
This makes it so that the final build product can be specified in output=
and it'll work no matter if its an APK or not.  This was developed around
the case of building the OTA update.zip for the Privileged Extension. It
should work for any build process in theory but it has not yet been tested.

https://gitlab.com/fdroid/privileged-extension/issues/9
2016-11-07 14:53:01 +01:00
Hans-Christoph Steiner
8ecff5bd61 get_release_filename() to handle any file type, not just APKs
In order to support non-APK files that are built by `fdroid build`, this
function that names the file releases needs to be generic.
2016-11-03 08:07:11 +01:00
Hans-Christoph Steiner
ab8d51012d use versionName unmodified as specified
The versionName is defined as a string or string resource that can be any
arbitrary data.  fdroid should not second guess the developer here, and
should just use the versionName unmodified.  For anything that needs to
compare different versions of apps, versionCode should always be used since
that's what Android uses.

https://developer.android.com/guide/topics/manifest/manifest-element.html#vname
2016-11-03 08:07:11 +01:00
Hans-Christoph Steiner
a8a1333b80 update buildserver scripts in sdist tarball manifest
This was overlooked in previous work on ./makebuildserver
2016-11-03 08:07:11 +01:00
Daniel Martí
7dde16dd11 Merge branch 'support-media-files' into 'master'
Support media files

This is the first step in supporting adding artibitrary files to fdroid repos, targeted add supporting media files and flashable _update.zip_ files like used to install the Privileged Extension.  This reuses the existing metadata fields to keep compatibilty with older versions, but that means that lots of names are confusingly named since they refer to aspects of an APK rather than a generic file.  We can address that later when we refactor the whole index metadata.

See merge request !177
2016-11-02 21:11:05 +00:00
Hans-Christoph Steiner
7d0bd345d1 Merge branch 'master' into 'master'
Scanner: handle utf8 errors more gracefully

Don't throw an error without any info which file caused it.

Fixes #226

See merge request !180
2016-11-02 17:18:02 +00:00
Hans-Christoph Steiner
8e45d30020 reuse os.stat() result when checking for non-APK files
This should make things a bit more efficient when running on lots of files,
unless python was already caching the result...
2016-11-02 18:10:13 +01:00
Hans-Christoph Steiner
07ce948809 support adding arbitrary files to a repo
This adds the most basic level of support for including arbitrary files in
an F-Droid repository.  This is useful for things like including videos,
ebooks, update.zip files for ROM updates, and more.  The aim is to have
this as generic as possible to keep it flexible for unforeseen uses.

Code-wise, this is really just a first effort.  This area of code has not
been touched in a very long time, and the repo parsing is done in a giant
function that is not easy to break apart.  It should be broken up to more
cleanly support arbitrary files.

Also remove the TODO line, we've decided to keep the old permission
format for now, at least until there is a major overhaul of the index
data format.  And the issue tracker the proper place for TODOs.
2016-11-02 18:10:13 +01:00
Hans-Christoph Steiner
36a585c2fc create addElementIfInApk() function for clean up common operation
There are currently a couple different ways this is done in the code, this
commit changes all of them to be like addElementNonEmpty().
2016-11-02 18:10:13 +01:00
est31
af22f18296 Scanner: handle utf8 errors more gracefully
Fixes #226
2016-11-02 16:58:59 +01:00
Hans-Christoph Steiner
47d9fd330d remove unused 'apps' argument from update.scan_apks() 2016-11-02 16:11:52 +01:00
Hans-Christoph Steiner
5faef55d67 support all valid versionCode values, i.e. Java Integer values
versionCode can be any Java Integer value, from Integer.MAX_VALUE
(2147483648) to Integer.MIN_VALUE (-2147483647)
2016-11-02 16:11:46 +01:00