1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-05 06:50:10 +01:00
Commit Graph

4025 Commits

Author SHA1 Message Date
Torsten Grote
2a421d653c Merge branch 'make-Build-App-act-like-dicts' into 'master'
make Build and App classes act like dicts

See merge request !210
2017-02-24 11:11:11 +00:00
Hans-Christoph Steiner
4e39621601 provide warning if config items will not preserve order
If a group of items are enclosed in {}, then that will be a Python set,
which does not preserve order.  To preserve order, the data must be either
a tuple () or list [].
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
388c336e76 build: ensure test is running on git commit that this code works with
Since https://gitlab.com/fdroid/ci-test-app is a separate git repo, things
with incompatible changes could get out of sync.  Therefore, this test
should specify which git commit is runs against.

For example, the .fdroid.yml file is still a moving target.  Just now, the
keys had the spaces removed as part of this MR.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
593613043e normalize Build TYPE_STRING data based on .txt
Unfortunately, things like versionCode must be strings.  That should be
converted to be ints throughout.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
c178cfb843 normalize Build TYPE_LIST data based on .txt
In the future, we should have better internal datatypes for this stuff,
i.e. instead of gradle: ['yes'] for True, actually use a boolean.  For now,
make the YAML and JSON metadata produce the same internal data as .txt.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
f7ae398aae add script for mass testing metadata parsing after changes
This requires manually running it.  I suppose it would be possible to
include a snapshot of the dumped internal representation for each release,
then make the tests run automatically against that.  Right now, the dump is
17megs of YAML.  Seems large to include in this git repo.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
e0f39a7e7b rename Build fields: version -> versionName, vercode -> versionCode
Since the YAML/JSON/etc. field names are now exactly the same as the field
names used in the internal dict in the Build class, this is a global rename

This keeps with the standard names used in Android:
https://developer.android.com/guide/topics/manifest/manifest-element.html
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
c0bc3afda9 convert metadata.Build to a subclass of dict
Like with the App class in the commit before, this makes it a lot
easier to work with this data when converting between the internal
formats and external formats like YAML, JSON, MsgPack, protobuf, etc.

The one unfortunate thing here is Build.update. It becomes
dict.update(), which is a method not an attribute.
build.get('update') or build['update'] could be used, but that would
be oddly inconsistent. So instead the field is renamed to
'androidupdate', except for in the .txt v0 metadata files. This better
describes what field does anyway, since it runs `android update`.

Build.update is only referenced in two places right next to each other
for the ant builds, so this change still seems worthwhile.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
b7fc7f2228 convert App to subclass of dict to support parsing/dumping libs
Python is heavily based on its core data types, and dict is one of the more
important ones.  Even classes are basically a wrapper around a dict. This
converts metadata.App to be a subclass of dict so it can behave like a dict
when being dumped and loaded.  This makes its drastically easier to use
different data formats for build metadata and for sending data to the
client.  This approach will ultimately mean we no longer have to maintain
custom parsing and dumping code.

This also means then that the YAML/JSON field names will not have spaces in
them, and they will match exactly what it used as the dict keys once the
data is parsed, as well as matching exactly the instance attribute names:

* CurrentVersion: 1.2.6
* app['CurrentVersion'] == '1.2.6'
* app.CurrentVersion == '1.2.6'

Inspired by:
https://goodcode.io/articles/python-dict-object/
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
4625651192 make Build class act more like a dict
This makes it a lot easier to work with Build instances with parsing and
dumping libraries, since they expect only core Python types (dict, list,
tuple, str, etc)
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
14f204cfe1 sort repo index data to make index.xml generation reproducible
This makes it easy to test that the code is still generating the exact same
index.xml.  It also might help the ZIP compression work better in index.jar
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
56ef716a4e test for original "v0" index XML compatibility
The original index.xml format needs to stay around for backwards
compatibility, but we shouldn't touch it anymore once the new format is in
place.  This is a test to make sure `fdroid update` can still generate the
correct XML.

install_list and uninstall_list should be tuples or lists in order to
ensure that the order is preserved.

These tests also check that the added and lastupdated dates are
working correct, based on the dates in tests/stats/known_apks.txt. I
could see no useful way to test the timestamp, it is just hardcoded
using a regexp search-and-replace.  Running these tests manually might
require deleting tmp/apkcache.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
327ba335a7 Merge branch 'support-v44' into 'master'
makebs: update support repo to r44, closes fdroid/rfp#49

Closes rfp#49

See merge request !213
2017-02-23 11:48:01 +00:00
Hans-Christoph Steiner
ed3af92cbd Merge branch 'firebase' into 'master'
scanner: add firebase to usual suspect list, closes #259

Closes #259

See merge request !212
2017-02-23 11:46:56 +00:00
Boris Kraut
687061f47e makebs: update support repo to r43, closes fdroid/rfp#49 2017-02-23 01:27:10 +01:00
Boris Kraut
daafc422ae scanner: add firebase to usual suspect list, closes #259 2017-02-23 00:41:00 +01:00
Hans-Christoph Steiner
284e272f48 Merge branch 'name' into 'master'
update: add name to skeleton, closes #258

See merge request !211
2017-02-21 14:08:47 +00:00
Boris Kraut
823e40eed0 update: add name to skeleton 2017-02-19 16:25:49 +01:00
Hans-Christoph Steiner
7fdf35014f Merge branch 'feature/git-mirroring' into 'master'
Add git repo mirroring

Closes #235

See merge request !206
2017-02-17 15:26:15 +00:00
thez3ro
5a9bcfbbf4
add git repo mirroring 2017-02-17 16:05:57 +01:00
Hans-Christoph Steiner
fc433c803b Merge branch 'ossjfrog' into 'master'
scanner: allow oss.jfrog.org/artifactory/oss-snapshot-local

See merge request !208
2017-02-13 21:06:18 +00:00
Hans-Christoph Steiner
bb5c2ec3de Merge branch 'build-publish-staging-test' into 'master'
build/publish staging test

See merge request !207
2017-02-13 19:42:21 +00:00
Boris Kraut
c87cdb91db scanner: allow oss.jfrog.org/artifactory/oss-snapshot-local 2017-02-13 20:24:35 +01:00
Hans-Christoph Steiner
2a39566906 jenkins-build-makebuildserver: remove VLC as test, its too flaky
Revert "jenkins-build-makebuildserver: include VLC as a test build"

This reverts commit 6debb3ebbf.
2017-02-13 20:10:49 +01:00
Hans-Christoph Steiner
1d91a400a2 buildserver: update to latest tools and m2repository 2017-02-13 20:10:48 +01:00
Hans-Christoph Steiner
3046156a89 jenkins-build-makebuildserver: handle git update of fdroiddata better
This prevents attempts to merge when there was rebasing, which can fail.
2017-02-13 20:08:45 +01:00
Hans-Christoph Steiner
837fc99d74 build: include buildserverid in build log for wiki 2017-02-13 20:08:45 +01:00
Hans-Christoph Steiner
57c3ba413c log Android SDK/NDK component versions on buildserver
ad2b9b99c2 put this in the wrong place, it
was running it on the buildserver host rather than in the buildserver VM
itself, where the builds actually run.

refs #148
2017-02-13 20:08:45 +01:00
Hans-Christoph Steiner
0cf08de347 build: fix fdroidserverid in build log post on wiki
The carriage return in the fdroidserverid file messes up the wiki format.
This was forgotten in 69d39bb301
2017-02-13 20:08:45 +01:00
Hans-Christoph Steiner
03223ab9df jenkins-build-makebuildserver: test the whole build/publish workflow
For full deployments like f-droid.org, the building happens on a separate
machine from the signing.  This adds a basic test of that kind of setup.
2017-02-13 20:08:45 +01:00
Hans-Christoph Steiner
9c36894955 Merge branch 'supportlib' into 'master'
makebs: bump support repo to r43

See merge request !205
2017-02-13 19:05:44 +00:00
Boris Kraut
e382f09fd6 makebs: bump support repo to r43 2017-02-11 14:40:40 +01:00
Ciaran Gultnieks
70c347264c Merge branch 'fix-build-and-vlc' into 'master'
fix CI, makebuildserver, and VLC build

See merge request !204
2017-02-07 13:27:56 +00:00
Hans-Christoph Steiner
6debb3ebbf jenkins-build-makebuildserver: include VLC as a test build 2017-02-07 11:05:18 +01:00
Hans-Christoph Steiner
1eb65c7087 buildserver: use automake and cmake from jessie-backports
These should be reasonably backwards compatible, and there is already
automake1.11 for those that need a version that old.  As for cmake, there
are five apps that seem to it:

com.amaze.filemanager
org.dolphinemu.dolphinemu
org.navitproject.navit
org.yabause.android
org.videolan.vlc

It looks like VLC is the only app that is currently building and using
cmake in the most recent releases.  Some of them used to use cmake, but no
longer.
2017-02-07 10:47:20 +01:00
Hans-Christoph Steiner
36272656fc gitlab-ci: workaround "ImportError: No module named 'packaging'"
https://github.com/pypa/setuptools/issues/937

fdroid/ci-images#1
2017-02-06 17:28:07 +01:00
Hans-Christoph Steiner
4929349555 buildserver: add openjdk-8-jdk-headless depends from backports 2017-02-06 14:59:15 +01:00
Ciaran Gultnieks
c344c31271 Merge branch 'supportlib' into 'master'
makebs: upgrade support repo to r42

See merge request !203
2017-02-04 17:34:49 +00:00
Boris Kraut
f32f4a68d1 makebs: upgrade support repo to r42 2017-02-04 15:01:15 +01:00
Hans-Christoph Steiner
1993cc874d Merge branch 'collection-of-fixes' into 'master'
include fdroidserverid in build log post on wiki

See merge request !201
2017-01-21 21:34:33 +00:00
Hans-Christoph Steiner
69d39bb301 build: include fdroidserverid in build log post on wiki 2017-01-20 12:12:20 +01:00
Hans-Christoph Steiner
d08bd32a35 update URL for git repo of test app 2017-01-20 12:12:20 +01:00
Hans-Christoph Steiner
78a3e35ef7 break out categories.txt generation to standalone method
This is to clean up the code for the introduction of a new index format.
This also makes it so that only repo/categories.txt is generated, and not
archive/categories.txt.

Currently, the only thing that I can find that reads categories.txt is the
wordpress plugin, e.g. wp-fdroid.  And it only reads repo/categories.txt
not archive/categories.txt.
2017-01-20 10:47:35 +01:00
Hans-Christoph Steiner
d54ad41938 fix bad caching of non-APK files in the repo
Silly mistake in 07ce948809
2017-01-20 10:47:35 +01:00
Hans-Christoph Steiner
8147f92355 Merge branch 'fdroid-publish-ota-zip' into 'master'
`fdroid publish` now includes OTA ZIPs and related source

See merge request !193
2017-01-12 13:03:51 +00:00
Hans-Christoph Steiner
7a1b99f11a Merge branch 'libtool-from-testing' into 'master'
buildserver: support installing packages from Debian/testing

Closes #224

See merge request !200
2017-01-12 12:59:57 +00:00
Hans-Christoph Steiner
4741f75f03 buildserver: update android_m2repository to latest (r41) 2017-01-12 10:19:38 +01:00
Hans-Christoph Steiner
948689281a ensure fdroiddata metadata file overrides .fdroid.yml in source
If the already parsed App instance from metadata/*.* has a field, then the
value coming from .fdroid.yml should not override it.
2017-01-11 22:42:41 +01:00
Hans-Christoph Steiner
7af7297ca8 buildserver: support installing packages from Debian/testing
Sometimes, a build process requires newer versions of build tools than are
available in Debian/stable.  Oftentimes, using the package straight from
Debian/testing works fine when a package is not available as a backport.
libtool 2.4.6 is needed for building VLC, so it is one example of this.

The preferences file sets up the apt "pinning" so that all updates are not
installed from testing, only the packages that are requested by adding
"/testing" after then package name.

closes #224
2017-01-10 10:56:03 +01:00
Hans-Christoph Steiner
bc27dee950 fdroid publish now includes OTA ZIPs and related source
This adds support for publishing ZIP files which were built with
`fdroid build`.  This is for "Over-The-Air" (OTA) update ZIP files for
flashing to ROMs.  The first example of this is the Privileged Extension,
which must be installed by flashing an OTA ZIP on Android > 5.0.

!181
https://gitlab.com/fdroid/privileged-extension/issues/9
https://gitlab.com/fdroid/privileged-extension/issues/10
https://gitlab.com/fdroid/fdroiddata/merge_requests/1804

Also, "if app.Binaries:" is the same as "if app.Binaries is not None:", but
is the standard Python style.
2017-01-09 11:32:52 +01:00