1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-16 03:40:33 +01:00
Commit Graph

5477 Commits

Author SHA1 Message Date
Hans-Christoph Steiner
84e09cd2a2 allow arbitrary build products, not only APKs
This makes it so that the final build product can be specified in output=
and it'll work no matter if its an APK or not.  This was developed around
the case of building the OTA update.zip for the Privileged Extension. It
should work for any build process in theory but it has not yet been tested.

https://gitlab.com/fdroid/privileged-extension/issues/9
2016-11-07 14:53:01 +01:00
Hans-Christoph Steiner
8ecff5bd61 get_release_filename() to handle any file type, not just APKs
In order to support non-APK files that are built by `fdroid build`, this
function that names the file releases needs to be generic.
2016-11-03 08:07:11 +01:00
Hans-Christoph Steiner
ab8d51012d use versionName unmodified as specified
The versionName is defined as a string or string resource that can be any
arbitrary data.  fdroid should not second guess the developer here, and
should just use the versionName unmodified.  For anything that needs to
compare different versions of apps, versionCode should always be used since
that's what Android uses.

https://developer.android.com/guide/topics/manifest/manifest-element.html#vname
2016-11-03 08:07:11 +01:00
Hans-Christoph Steiner
a8a1333b80 update buildserver scripts in sdist tarball manifest
This was overlooked in previous work on ./makebuildserver
2016-11-03 08:07:11 +01:00
Daniel Martí
7dde16dd11 Merge branch 'support-media-files' into 'master'
Support media files

This is the first step in supporting adding artibitrary files to fdroid repos, targeted add supporting media files and flashable _update.zip_ files like used to install the Privileged Extension.  This reuses the existing metadata fields to keep compatibilty with older versions, but that means that lots of names are confusingly named since they refer to aspects of an APK rather than a generic file.  We can address that later when we refactor the whole index metadata.

See merge request !177
2016-11-02 21:11:05 +00:00
Hans-Christoph Steiner
7d0bd345d1 Merge branch 'master' into 'master'
Scanner: handle utf8 errors more gracefully

Don't throw an error without any info which file caused it.

Fixes #226

See merge request !180
2016-11-02 17:18:02 +00:00
Hans-Christoph Steiner
8e45d30020 reuse os.stat() result when checking for non-APK files
This should make things a bit more efficient when running on lots of files,
unless python was already caching the result...
2016-11-02 18:10:13 +01:00
Hans-Christoph Steiner
07ce948809 support adding arbitrary files to a repo
This adds the most basic level of support for including arbitrary files in
an F-Droid repository.  This is useful for things like including videos,
ebooks, update.zip files for ROM updates, and more.  The aim is to have
this as generic as possible to keep it flexible for unforeseen uses.

Code-wise, this is really just a first effort.  This area of code has not
been touched in a very long time, and the repo parsing is done in a giant
function that is not easy to break apart.  It should be broken up to more
cleanly support arbitrary files.

Also remove the TODO line, we've decided to keep the old permission
format for now, at least until there is a major overhaul of the index
data format.  And the issue tracker the proper place for TODOs.
2016-11-02 18:10:13 +01:00
Hans-Christoph Steiner
36a585c2fc create addElementIfInApk() function for clean up common operation
There are currently a couple different ways this is done in the code, this
commit changes all of them to be like addElementNonEmpty().
2016-11-02 18:10:13 +01:00
est31
af22f18296 Scanner: handle utf8 errors more gracefully
Fixes #226
2016-11-02 16:58:59 +01:00
Hans-Christoph Steiner
47d9fd330d remove unused 'apps' argument from update.scan_apks() 2016-11-02 16:11:52 +01:00
Hans-Christoph Steiner
5faef55d67 support all valid versionCode values, i.e. Java Integer values
versionCode can be any Java Integer value, from Integer.MAX_VALUE
(2147483648) to Integer.MIN_VALUE (-2147483647)
2016-11-02 16:11:46 +01:00
Hans-Christoph Steiner
f90adcc912 Merge branch 'texinfo' into 'master'
CI: use new docker image with texinfo

docs/gendocs.sh uses makeinfo.

See merge request !179
2016-10-31 17:51:16 +00:00
Daniel Martí
2e96fc1cd9 CI: use new docker image with texinfo
docs/gendocs.sh uses makeinfo.
2016-10-31 14:02:39 +00:00
Daniel Martí
559f38cd21 Merge branch 'updates' into 'master'
makebs and CI updates

See merge request !178
2016-10-25 11:06:47 +00:00
Daniel Martí
148f077ad8 Bump ci images and default build-tools 2016-10-23 17:04:58 +01:00
Daniel Martí
98b7538789 makebs: bump m2repo, add api25 and build-tools 25 2016-10-23 16:47:19 +01:00
Ciaran Gultnieks
aa4aa45703 Merge branch 'build-tools' into 'master'
makebs: add build-tools 24.0.3



See merge request !175
2016-10-05 18:09:46 +00:00
Ciaran Gultnieks
749c04c66a Merge branch 'fixes-on-the-way-to-kvm' into 'master'
buildserver fixes on the way to KVM

Here are a couple of relatively basic fixes I found while working on the KVM support (merge request coming soon).

See merge request !174
2016-10-05 18:09:18 +00:00
Daniel Martí
b34cef8776 makebs: add build-tools 24.0.3 2016-10-03 09:10:41 +01:00
Hans-Christoph Steiner
764b99acd6 buildserver: only auto-detect KVM in ./makebuildserver
Having a second, different KVM auto-detect routine in Vagrantfile will only
confuse things.  This also removes the direct call to the systemd utility.
2016-09-28 10:16:30 +02:00
Hans-Christoph Steiner
1b09ab5a79 buildserver: run dir check as very first thing
No point in running any other code if the script is just going to bail
out with an error.  This assumes that ./makebuildserver is only ever
run from a git clone of fdroidserver.git.
2016-09-28 09:52:49 +02:00
Hans-Christoph Steiner
a3d32c65a4 buildserver: use py3's pathlib to generate cache file URL
This should handle edge cases better, like odd characters in the path, etc.
2016-09-28 09:52:49 +02:00
Hans-Christoph Steiner
97be5a1d00 jenkins-build-makebuildserver: remove errant blank line 2016-09-28 09:52:49 +02:00
Hans-Christoph Steiner
84500b0049 Merge branch 'master' into 'master'
Add qt sdk support

Qt SDK extraction on the VM

See merge request !172
2016-09-27 12:54:02 +00:00
est31
91c5fb567a Add qt sdk support 2016-09-27 14:36:12 +02:00
Ciaran Gultnieks
d3a215c12b Merge branch 'checkupdates-crash' into 'master'
checkupdates: avoid crash with --auto and None CVC

As reported by @CiaranG.

See merge request !173
2016-09-26 21:46:16 +00:00
Daniel Martí
bcd31401ac checkupdates: avoid crash with --auto and None CVC
Reproducible via `fdroid checkupdates --auto subreddit.android.appstore`
at fdroiddata HEAD (e76449ab).

WARNING: ...subreddit.android.appstore : Couldn't find package ID
CRITICAL: Unknown exception found!
Traceback (most recent call last):
  File "/home/mvdan/.bin/fdroid", line 147, in <module>
    main()
  File "/home/mvdan/.bin/fdroid", line 124, in main
    mod.main()
  File "/home/mvdan/git/fsr/fdroidserver/checkupdates.py", line 571, in main
    checkupdates_app(app)
  File "/home/mvdan/git/fsr/fdroidserver/checkupdates.py", line 469, in checkupdates_app
    if int(build.vercode) >= int(app.CurrentVersionCode):
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
2016-09-26 21:07:55 +01:00
Ciaran Gultnieks
7cbdaedf2e Merge branch 'fix-post-kvm' into 'master'
Fix post KVM

A couple of fixes to the KVM support that was just merged, as reported by @CiaranG

See merge request !171
2016-09-23 16:33:52 +00:00
Hans-Christoph Steiner
1001e5e9b9 buildserver: bochs can also be used with KVM
jenkins.debian.net is being detected as 'bochs' rather than 'qemu'.
2016-09-23 17:41:41 +02:00
Hans-Christoph Steiner
1e8fd01c1e buildserver: only check cache permissions when using libvirt
VirtualBox runs as the same user as `fdroid`, so the cache does not need to
be accessible by the world.  On libvirt, libvirtd runs the VMs as its own
user, so in that case, the cache dirs must have permissions to let that
user access them.
2016-09-23 15:20:05 +02:00
Hans-Christoph Steiner
f6a487eee4 buildserver: wipe snapshot from libvirt store on --clean 2016-09-23 15:06:41 +02:00
Ciaran Gultnieks
5667d16498 Merge branch 'buildserver-qemu-kvm' into 'master'
buildserver running in qemu/kvm to support KVM on KVM

jenkins.debian.net runs in QEMU/KVM instances, so in order to run the F-Droid buildserver there, it needs to work inside of a KVM guest.  The best way I found to do that is to create QEMU/KVM instances via KVM's "nested" virtualization support.  This collection of commits enables using QEMU/KVM as the buildserver when `./makebuildserver` detects that it is running inside of KVM.  Otherwise, the old behavior is default: running in VirtualBox.

I have run these tests inside of ubuntu/16.04 on bare metal, which uses VirtualBox, and ubuntu/16.04 KVM guest, which uses QEMU/KVM.  It'll also run on the Guardian Project jenkins box, which is Debian/jessie.

@mvdan @CiaranG @krt

See merge request !168
2016-09-23 12:25:12 +00:00
Hans-Christoph Steiner
1216125f6e Merge branch 'master' into 'master'
makebs: add gradle 3.1



See merge request !170
2016-09-23 06:36:58 +00:00
est31
56a0077183 makebs: add gradle 3.1 2016-09-23 07:50:52 +02:00
Daniel Martí
ee54fbc1b8 Merge branch 'vagrant' into 'master'
docs: update vagrant link

As pointed out in https://f-droid.org/forums/topic/documentation-bug-fix-report/#post-21839 , the link to the vagrant doc is 404. I'd assume that this is the correct link, but please verify this before merge.

See merge request !169
2016-09-20 16:19:08 +00:00
Boris Kraut
42031031e6 docs: update vagrant link 2016-09-20 11:38:12 +02:00
Hans-Christoph Steiner
a2a630c83c buildserver: only include latest m2 when provisioning
I think the `android update sdk` tool is installing all of the m2 files
that are present in the temp cache, and it seems to do it in order of
newest to oldest.  Well done, and I thought that tool couldn't get any
worse.  So only include the latest version of android_m2repository*.zip in
the temp cache.
2016-09-19 16:33:12 +02:00
Hans-Christoph Steiner
0a0ba6de84 buildserver: ensure dirs to mount cache exist in guest VM instance
It seems that the 9p synced folder setup is not as flexible and easy as the
VirtualBox one, so we have to do more little things like this.
2016-09-19 16:33:12 +02:00
Hans-Christoph Steiner
988647a387 buildserver: fix jenkins build to support QEMU/KVM 2016-09-19 16:33:12 +02:00
Hans-Christoph Steiner
3651e7eb9a buildserver: retry apt-get downloads 20 times
Try harder before failing the whole buildserver setup.
2016-09-19 16:33:12 +02:00
Hans-Christoph Steiner
68ec7c2e88 buildserver: auto-detect and use libvirt's QEMU+KVM
For running in QEMU/KVM guests like on jenkins.debian.net, this sets up the
whole process automatically.  This only really covers the case where this
is running in a KVM guest, and the original case of running VirtualBox on
bare metal.  It could be extended to cover more cases if someone wanted to.
2016-09-19 16:33:12 +02:00
Hans-Christoph Steiner
f306e32636 buildserver: make sure cachedir is accessible to VM instance
It seems that KVM/QEMU's 9p sharing is more sensitive to file perms.  If
~/.cache is 0700, then ~/.cache/fdroidserver cannot be mounted in the guest
2016-09-19 16:33:12 +02:00
Hans-Christoph Steiner
0ae2df6646 buildserver: support optionally using qemu+kvm
This makes it possible to run the full build process in a KVM virtual
machine, like jenkins.debian.net.

closes #190
2016-09-19 16:33:12 +02:00
Hans-Christoph Steiner
5af53238d3 buildserver: ignore android sdk repos of proprietary bits
The Android SDK by default includes some Google repositories of their
proprietary SDKs.  We of course do not want that stuff ever.  We also do
not need the emulator images since this process does not currently install
or run an emulator.
2016-09-19 16:33:12 +02:00
Hans-Christoph Steiner
b5e57a7aa6 example file for public read only S3 bucket policy
This is the locked down S3 policy as created by Benetech for their Secure
App Generator project.
2016-09-15 23:20:18 +02:00
Hans-Christoph Steiner
008b4a31eb skip fdroid import test if gitlab is not available
This prevents CI build failures when gitlab is deploying or has other
issues that might cause a 500.
2016-09-15 23:20:18 +02:00
Daniel Martí
89010a1ba5 Merge branch 'gradle-wrapper' into 'master'
Make the gradle wrapper smarter

@eighthave PTAL

This will help with issues like https://gitlab.com/fdroid/fdroiddata/issues/515

See merge request !166
2016-09-15 20:45:44 +00:00
Hans-Christoph Steiner
8fc27680cb Merge branch 'makebs-updates' into 'master'
Makebs updates



See merge request !167
2016-09-15 20:38:43 +00:00
Daniel Martí
8797cc01a8 makebs: bump support repo to 38 2016-09-15 20:07:25 +01:00