1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-09-17 18:50:11 +02:00
Commit Graph

4280 Commits

Author SHA1 Message Date
Hans-Christoph Steiner
daade7656a buildserver: add copy_caches_from_host config option
For people using slow, expensive, and/or flaky internet, liberal use of
caching can make a huge difference.  The restricted environment of the
gpjenkins box has been a good test environment for this (Tor-only,
whitelist of allowed IPs to visit, home internet connection).
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
4cde71552f buildserver: run_via_vagrant_ssh() to run cmds via python-vagrant
This moves the last vagrant call in a subprocess.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
6464ec55b7 buildserver: move code into main() method to always stop thread
By running the whole program in a main() function, it can be wrapped in
try/finally in order to stop the background display thread.  This is also
done in ./fdroid, its standard practice for Python CLI utilities.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
675500ad88 buildserver: display verbose logging in a background tail 2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
4b03c3d42d buildserver: replace custom code with python-vagrant
I ran into some annoying issues with UTF-8 output in the vagrant logs, and
it was hard to solve.  So I switched to using python-vagrant, which handles
it all for us.  Its been around since 2012, has a number of contributors,
and is still actively maintained, so it seems like a good bet. I also
packaged it for Debian, including a backport in jessie-backports.

On Debian/jessie, do `apt-get install python3-vagrant/jessie-backports`
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
7ef0d5dfd8 include class like UNIX tail -f for displaying logs
This allows fdroidserver to easily log activity while displaying it at the
same time.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
c62e3fd0cc move bulk of reproducible_fdroid_build_apps.sh to jenkins-build
This lets us quickly and frequently test things.
2017-05-23 20:04:08 +02:00
Hans-Christoph Steiner
1710428de1 Merge branch 'exceptions' into 'master'
Replace sys.exit() in non-main functions by exceptions

See merge request !273
2017-05-22 20:50:54 +00:00
Torsten Grote
1fcd8e63a3
Replace sys.exit() in non-main functions by exceptions
Also move all exceptions into one module
2017-05-22 16:51:11 -03:00
Michael Pöhn
a8420817cb fix calling vagrant global-status 2017-05-22 17:40:19 +02:00
Michael Pöhn
592a292847 use stable version of priv ext instead of latest when testing in makebuildserver 2017-05-22 16:04:30 +02:00
Hans-Christoph Steiner
84dc415837 jenkins-build-makebuildserver: retry git fetch until it works
It seems that gitlab gives 500 errors a bit too frequently, so keep
retrying the `git pull` until it works so that there isn't a error email
sent out over failed pulls.
2017-05-22 16:04:30 +02:00
Hans-Christoph Steiner
780b0e9502 Merge branch 'indexing-fixes' into 'master'
indexing fixes

Closes #303

See merge request !270
2017-05-22 09:32:39 +00:00
Hans-Christoph Steiner
0345d2c79f update: find aapt when it is not in the PATH
This is some very messy logic built up since 2010.  This will all go away
once we have a python3 version of androguard available.

The removed imports and `dir(APK)` is to silence pyflakes

closes #303
2017-05-18 17:21:05 +02:00
Hans-Christoph Steiner
c348186ad6 update: fix fastlane scraping, it uses a subdir called 'images'
https://commons.wikimedia.org/wiki/File:GetChromium_FeatureGraphic_1024x500.png
2017-05-18 16:13:17 +02:00
Hans-Christoph Steiner
77a65be9db update: skip packages without metadata in index V1
If a package does not have a matching "app" metadata file, then it should
not be added to the index. index V0 already does this properly.
2017-05-18 16:13:17 +02:00
Hans-Christoph Steiner
a17b95a860 update: improved logging
the aapt warnings were very verbose, and the other now includes the
file name type for screenshots, rather than the filename twice.
2017-05-18 16:13:17 +02:00
Hans-Christoph Steiner
5d705452f5 update: allow repo files to use _ in the file names
_ is a valid character for Java package names, so it should also work in
the repo file naming scheme. This makes it so it only splits the file
name based on the last _.
2017-05-18 16:13:17 +02:00
Hans-Christoph Steiner
84bb41a91f metadata: switch from deprecated cgi.escape to html.escape
cgi.escape is deprecated in Python 3.x and has security issues:
https://bugs.python.org/issue26398

html.escape() differs from cgi.escape() by its defaults to quote=True:

 s = html.escape( """& < " ' >""" )   # s = '&amp; &lt; &quot; &#x27; &gt;'
2017-05-18 16:13:17 +02:00
Torsten Grote
ee57285817 Merge branch 'cdn-git-mirror' into 'master'
Fix double pushing on remote git-mirror

See merge request !272
2017-05-17 22:47:56 +00:00
thez3ro
ec30d73f13
fix double pushing on remote 2017-05-17 23:08:25 +02:00
Hans-Christoph Steiner
87066bc1d5 Merge branch 'cdn-git-mirror' into 'master'
Use Gitlab Pages as mirror instead of Raw

Closes #293

See merge request !271
2017-05-17 20:29:33 +00:00
thez3ro
846f5422fd
Use gitlab pages as mirror 2017-05-17 21:08:34 +02:00
Hans-Christoph Steiner
b144029517 Merge branch 'hg39' into 'master'
buildserver: use hg 3.9 from backports

See merge request !266
2017-05-10 17:22:15 +00:00
Boris Kraut
fec2f677bf buildserver: use hg 3.9 from backports 2017-05-10 18:55:38 +02:00
Hans-Christoph Steiner
eb49c9e8f3 Merge branch 'master' into 'master'
replace aapt with androguard

See merge request !234
2017-05-05 09:10:32 +00:00
Michael Pöhn
ff3baefbed Merge branch 'yaml' into 'master'
basic support for YAMLin `fdroid rewritemeta`

Closes #285 and #169

See merge request !260
2017-05-05 08:35:44 +00:00
thez3ro
06598ae406
use androguard if aapt isn't found 2017-05-04 23:35:17 +02:00
Hans-Christoph Steiner
9607cdb621 Merge branch 'download-index-etag' into 'master'
Support ETag when downloading repository index

See merge request !264
2017-05-02 19:12:41 +00:00
Torsten Grote
8d424f19ec
Support ETag when downloading repository index 2017-05-02 15:37:02 -03:00
Hans-Christoph Steiner
4f758f5c78 fix white space trimming when writing .txt format
Some of the internal data format changes somehow broke the white space
trimming when writing out .txt.

closes #285
2017-05-02 14:55:23 +02:00
Hans-Christoph Steiner
0d00705aed use pyyaml for rewriting .yml metadata files
This replaces the broken, custom code with the standard YAML lib.

In rewritemeta, do not call app.metadatapath since it will be deleted when
the dict is cleaned up for outputing. metadatapath is only used internally
and should not be written out.

closes #169
refs #290
2017-05-02 14:55:23 +02:00
Michael Pöhn
e7e97654b1 Merge branch 'libvirt-support' into 'master'
Check for libvirt specific shutoff message

See merge request !262
2017-05-02 07:39:51 +00:00
Torsten Grote
f758b5c205 Merge branch 'pre-0.103-alpha4-rename' into 'master'
downcase all 'localized' key names to match the rest of index-v1

See merge request !261
2017-05-01 17:59:59 +00:00
Andrew Patrikalakis
9ee96117d8 Check for libvirt specific shutoff message 2017-04-29 18:38:44 -07:00
Hans-Christoph Steiner
cdef5bcd92 downcase all 'localized' key names to match the rest of index-v1
This is a little omission.  keys that are used in metadata/*.yml all start
with an UpperCase letter, but in fdroidserver, index-v1.json, and
fdroidclient, it is all camelCase with lowercase first letter. The keys
from the 'localized' section are currently never in metadata/*.yml, so
these keys never get downcase.  This change will break fdroidclient
versions that do not also have this change, but since we're in alpha, that
should be fine.

If support for a 'localized' section is added to metadata/*.yml, then the
keys there should probably be UpperCase CamelCase to match the other keys.
2017-04-27 21:12:49 +02:00
Hans-Christoph Steiner
9f9f0d1a16 Merge branch 'git-identity-file' into 'master'
Make git server mirror upload honor config['identity_file'] option

See merge request !259
2017-04-24 21:24:07 +00:00
Torsten Grote
c137231d8b
Make git server mirror upload honor config['identity_file'] option 2017-04-24 16:22:33 -03:00
Hans-Christoph Steiner
591bfc4474 Merge branch '148-no-verify-logs-on-split-build-publish-infrastructure' into 'master'
compare apk with Binaries from metadata right after building

Closes #148

See merge request !247
2017-04-24 18:50:58 +00:00
Michael Pöhn
fc4e40255d remove build output when not reproducible 2017-04-23 11:33:51 +02:00
Michael Pöhn
5b7209d882 unified method for naming build output 2017-04-22 12:04:32 +02:00
Michael Pöhn
62af9b8e3d check meld config for manual apk diffs 2017-04-22 10:48:50 +02:00
Michael Pöhn
4044e06e7b use verify_apks for checking build output 2017-04-22 10:48:50 +02:00
Michael Pöhn
f4a7c19043 fix java architecture in buildserver config 2017-04-22 10:48:50 +02:00
Michael Pöhn
a3e7eacc9b compare apk with Binaries from metadata right after building 2017-04-22 10:48:50 +02:00
Hans-Christoph Steiner
1c25c516aa Merge branch '273-invalid-apk-gets-through-to-publish-stage' into 'master'
throw proper error when download of apks fail

Closes #273

See merge request !258
2017-04-21 07:56:55 +00:00
Michael Pöhn
abe04272b1 throw proper error when download of apks fail 2017-04-20 17:48:38 +02:00
Daniel Martí
c7ae6bc826 Merge branch 'gradleplugin' into 'master'
bs: use only prefixes for gradle kv pairs

See merge request !257
2017-04-19 09:27:42 +00:00
Hans-Christoph Steiner
650b3c95b4 Merge branch 'full-app-store-metadata-scraping' into 'master'
Full app store metadata scraping

Closes #204 and #143

See merge request !253
2017-04-19 08:25:56 +00:00
Hans-Christoph Steiner
82095c7a9a add basic test for Triple-T Gradle Play Publisher scraping 2017-04-19 10:05:24 +02:00