1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-15 03:20:10 +01:00
Commit Graph

4052 Commits

Author SHA1 Message Date
Daniel Martí
9520839984 gradle wrapper script: add plugin 1.5 2016-02-15 12:02:45 +00:00
Daniel Martí
c2270f8434 Merge branch 'master' into 'master'
a couple fixes

these are three fixes, one related to my previous merge request, as discussed with @mvdan in IRC

See merge request !99
2016-02-12 09:13:53 +00:00
Hans-Christoph Steiner
5780c14df2 fix PEP9 E713 test for membership should be 'not in' 2016-02-12 08:33:22 +01:00
Hans-Christoph Steiner
dfca237329 fix errors in pre-commit hook when run in test suite 2016-02-12 08:33:22 +01:00
Hans-Christoph Steiner
8489047bcf build: check that metadata is present before creating tmp dirs
Before, `fdroid build` would create some subdirs, then fail when it could
not read the metadata via metadata.read_metadata().  This checks before
whether there is any metadata available, and warns the user accordingly.
2016-02-12 08:33:22 +01:00
Hans-Christoph Steiner
751fd3fb0a common: do not crash if the java_paths are not what is expected
Many commands work without the JDK installed, and it is also possible that
someone is using only JDK 8 or 9.
2016-02-12 08:33:21 +01:00
Daniel Martí
7feeaad095 common: add regexp for arch oracle paths 2016-02-11 21:48:51 +00:00
Daniel Martí
28db5b82c2 common: fix JDK detection on latest arch 2016-02-11 21:34:44 +00:00
Hans-Christoph Steiner
6fc62fe609 Merge branch 'auto-detect-java-homes' into 'master'
Auto detect java homes

There are a wide variety of possible JDKs (OpenJDK, Oracle, OSX, etc.) as well as standard paths for the JDK.  This includes code to look in standard paths for various platforms and enumerate the installed JDKs (e.g. 6, 7, 8, 9).  That is then used for `JAVA[6-9]_HOME` env vars and the path for `jarsigner` and `keytool`.

See merge request !98
2016-02-11 20:47:29 +00:00
Hans-Christoph Steiner
69c81c3817 use jarsigner and keytool from same JDK as is being set in JAVA7_HOME
Using the same JDK throughout should prevent weird bugs where a setup might
use Java8's jarsigner and Java7's keytool.  This also allows the user to
set java_paths and have jarsigner and keytool used from that specified JDK.

This incorporates almost all of the patch that is in the Debian package
that forces fdroidserver to use the default JDK on that Debian release.

closes #93 https://gitlab.com/fdroid/fdroidserver/issues/93
2016-02-11 21:17:23 +01:00
Hans-Christoph Steiner
fdf17e809c automatically detect various installed JDKs and set JAVA[6-9]_HOME
This checks for which JDKs are installed in common locations, then sets the
JAVA[6-9]_HOME env vars needed by some build environments.
2016-02-11 21:17:23 +01:00
Hans-Christoph Steiner
7f451a815b makebuildserver: add workaround to Ubuntu/trusty's old paramiko
Ubuntu trusty 14.04's paramiko does not work with jessie's openssh's
default settings, so they need to be tweaked in order to provide working
ssh to the instance.

https://stackoverflow.com/questions/7286929/paramiko-incompatible-ssh-peer-no-acceptable-kex-algorithm/32691055#32691055
2016-02-11 20:27:30 +01:00
Daniel Martí
1f59d6f0cf Merge branch 'jmastr/use_tag_if_version_name_is_unknown' into 'master'
use tag if version name is unknown

If we don't find the version name neither in the AndroidManifest.xml nor
in the build.grade file, we fallback to the tag the user specified to
search for.

That way we have a better version name than 'Unknown'.

See merge request !97
2016-02-11 19:06:22 +00:00
Julian Strobl
dd0b38a7e5 use tag if version name is unknown
If we don't find the version name neither in the AndroidManifest.xml nor
in the build.grade file, we fallback to the tag the user specified to
search for.

That way we have a better version name than 'Unknown'.
2016-02-11 19:50:16 +01:00
Daniel Martí
45d99d1cc9 Merge branch 'prevo' into 'master'
Add libexpat1-dev and libglib2.0-dev to the default recipe

These are needed to build the PrevoDB srcdep of
uk.co.busydoingnothing.prevo.

See merge request !96
2016-02-10 15:57:19 +00:00
Neil Roberts
c9404c8e39 Add libexpat1-dev and libglib2.0-dev to the default recipe
These are needed to build the PrevoDB srcdep of
uk.co.busydoingnothing.prevo.
2016-02-10 14:59:13 +00:00
Daniel Martí
a26a057b90 Merge branch 'jenkins-makebuildserver-works' into 'master'
Jenkins ./makebuildserver works!

These commits get `./makebuildserver` running in a Jenkins job. It builds the whole setup from scratch every time, including the vagrant and virtualbox setups.

Also, this modifies the git pre-commit hook to only run on the files being committed when called by git.

See merge request !95
2016-02-08 16:07:27 +00:00
Hans-Christoph Steiner
2ea38fa7e3 test ./makebuildserver by running a build in jenkins 2016-02-04 20:18:03 +01:00
Hans-Christoph Steiner
2d2ac66a26 script to run ./makebuildserver on Jenkins instances
This is being used to run it on both Guardian Project's and Debian's
Jenkins instances.
2016-02-04 20:16:35 +01:00
Hans-Christoph Steiner
09daa5eee0 makebuildserver: default memory to 1024MB, so it runs on normal machines
4 gigs is still a common amount of RAM these days for laptops, if the VM
takes almost all of that, it makes the machine drag to almost a halt. Most
apps build fine in 1gig of RAM, indeed that's the default for most CI
instances, like travis-ci and gitlab-ci.
2016-02-03 14:54:14 +01:00
Hans-Christoph Steiner
dd93505fce allow setting up an apt package cache for the build server setup
This creates a cache folder which will store the apt cache from the VM.
2016-02-03 14:54:14 +01:00
Hans-Christoph Steiner
b6a5978b11 makebuildserver: make chef find the custom cache locations 2016-02-03 14:54:14 +01:00
Hans-Christoph Steiner
7c30569aed make git pre-commit hook only test files to be committed
Right now, the git pre-commit hook is pretty annoying to work with when
development since it tests every single file.  That means notes, incomplete
stuff, etc. will be run through the tests.  So all of the files need to be
clean in order to commit even a single trivial fix.  This commit changes it
so that the pre-commit hook is only run on the files staged to be committed.
2016-02-03 14:54:14 +01:00
Daniel Martí
f586161361 lint: also check for trailing spaces in names
I was wrong - we don't check for trailing whitespaces on lines. That
would have to happen at parse time, not in the linter, so it would slow
things down and would become an error, not a warning. Not really worth
it.
2016-01-31 21:29:31 +00:00
Daniel Martí
8f36adfd7b Merge branch 'authorregex' into 'master'
lint: check whitespaces in "Author Name"



See merge request !94
2016-01-31 21:26:31 +00:00
Boris Kraut
5858ba5680 lint: check whitespaces in "Author Name" 2016-01-31 21:37:25 +01:00
Daniel Martí
d180f4f8c4 Merge branch 'licenses' into 'master'
wp-fdroid: Linkify more licenses



See merge request !90
2016-01-31 13:26:26 +00:00
Daniel Martí
0c68d13560 Merge branch 'patch-1' into 'master'
Support 'gpgkey' option as announced in sample config.



See merge request !93
2016-01-31 13:25:03 +00:00
Marvin W
6130e6eaf0 Support 'gpgkey' option as announced in sample config. 2016-01-27 15:51:03 +00:00
Ciaran Gultnieks
3629437b09 Tidy up/fix some vagrant issues
1. It is packaged in modern versions of distros, update docs accordingly
2. 1.1 is hopelessly outdated - support 1.4 onwards
3. Check the version properly, and write a better config (previously it
   wrote the 1.1 config for 1.1 and 1.3+
4. Ensure that the default share from later versions is not present when
   building, it's only required for provisioning.
2016-01-26 22:05:31 +00:00
Ciaran Gultnieks
f6b9e46246 Handle with invalid manifests better
In other words, don't blow up and kill a whole checkupdates run just
because one version of one package has some invalid xml.
2016-01-26 22:05:31 +00:00
Daniel Martí
5952c46bd0 Merge branch 'makebuildserver-jenkins-fixes' into 'master'
Makebuildserver jenkins fixes

more stuff related to the Debian jenkins build

See merge request !92
2016-01-26 19:52:12 +00:00
Hans-Christoph Steiner
90d81eba2d makebuildserver: settable vm boot timeout, plus longer default
On slow machines or VMs like the Debian jenkins box, the VM boot timeout
needs to be a lot longer, otherwise vagrant times out before setting up
the VM.
2016-01-26 17:47:17 +01:00
Hans-Christoph Steiner
3511c44357 makebuildserver: explicitly set debian_mirror by default
There needs to be a default, otherwise it crashes.  I should have included
this in 8d66742dd7
2016-01-26 17:46:51 +01:00
Daniel Martí
88d369e293 Merge branch 'makebuildserver-caching-fixes' into 'master'
./makebuildserver CI fixes

These are fixes for caching and flexible config related to the work of getting `./makebuildserver` running on Debian's and Guardian Project's Jenkins boxes.  @mvdan and I have been working with h01lger to get that going, here's the latest build log: https://jenkins.debian.net/view/reproducible/job/reproducible_setup_fdroid_build_environment_profitbricks3/lastBuild/console

See merge request !91
2016-01-26 11:00:58 +00:00
Hans-Christoph Steiner
980339d2f5 make all apt repos in build server use the configured debian mirror 2016-01-26 11:31:40 +01:00
Hans-Christoph Steiner
8d66742dd7 makebuildserver: set debian mirror in config file only
Setting it in the config file was not working, and right now, all of the
options are in the config file and not as command line flags, so remove
--debian-mirror to keep that consistent.
2016-01-26 11:31:40 +01:00
Hans-Christoph Steiner
82e26cf59e makebuildserver: use jessie32.box from cache, if present 2016-01-26 11:31:40 +01:00
Daniel Martí
1ff4f63de3 scanner: Allow clojars.org maven repository
See https://github.com/inorichi/tachiyomi/issues/46.
2016-01-24 18:13:45 +00:00
Boris Kraut
c773fdf633 wp-fdroid: Linkify more licenses 2016-01-24 13:27:15 +01:00
Hans-Christoph Steiner
16c220517d Merge branch 'master' into 'master'
Reproducibly build documentation

The two commits fix reproducibility problems detected by the Debian CI:

https://reproducible.debian.net/rb-pkg/unstable/amd64/fdroidserver.html

See merge request !84
2016-01-22 10:27:54 +00:00
Daniel Martí
0003e4c742 Merge branch 'makebuildserver-for-debian-jenkins' into 'master'
Makebuildserver for debian jenkins

@mvdan and I worked out a plan with Holger, the Debian Developer running the reproducible builds jenkins build farm.  The idea is for that to just run `./makebuildserver` in the jenkins script.  Then we'll make sure that it works.  This also makes it easy for anyone to set up and run an FDroid build instance, which is important for things like the verification server and for people to be able to run reproducible builds on their own machine.

This does change some of the defaults, but they can all be overridden in the `makebs.config.py` for f-droid.org.

More comments in the commits.


See merge request !89
2016-01-21 15:49:37 +00:00
Hans-Christoph Steiner
24a9f3b816 makebuildserver: delete cached files with zero size
If download fails to connect, it'll make a zero size file.  Then the next
run on ./makebuildserver will fail because the checksum will not be right.
2016-01-21 15:41:26 +01:00
Hans-Christoph Steiner
cda8fa00f5 makebuildserver: cache defaults to ~/.cache/fdroidserver
This also provides a config option to override that default.  ~/.cache is
a standard location on GNU/Linux machines for cached content.  It is also
good to have the cache outside of the git repo in case `git clean -fdx` is
run, which would delete all files in the directory that are not part of the
git repo, including buildserver/cache/
2016-01-21 15:34:55 +01:00
Hans-Christoph Steiner
3bd8ef2c8c makebuildserver: set defaults in script so it can run without config
This makes it so that ./makebuildserver will run without any config file,
using the defaults that are embedded in the script itself.  This is like
how `fdroid` works.
2016-01-21 15:34:55 +01:00
Hans-Christoph Steiner
50713c7ce9 makebuildserver: set default Debian mirror to http.debian.net
This host automatically detects which is the closest mirror, then uses that
one.  It does so dynamically, so it'll work on machines that move too. Now
that we are pushing more people to run F-Droid build servers, the defaults
should take those use cases into account.
2016-01-21 15:34:55 +01:00
Hans-Christoph Steiner
b33add9ce7 name makebuildserver config the same as the script
This keeps the numbers of names down to a minimum, and since the config
is placed right next to the script, this keeps tab completion working
nicely when the config file is in place.

The old file name is still supported.
2016-01-21 15:34:55 +01:00
Hans-Christoph Steiner
25caa09420 run-tests: verbose output for fdroid update to aid debugging
I'm stumped by this stacktrace, hopefully debug logging will help:

Unknown exception found!
Traceback (most recent call last):
  File "./fdroidserver-0.4.0/fdroid", line 141, in <module>
    main()
  File "./fdroidserver-0.4.0/fdroid", line 119, in main
    mod.main()
  File "<http://localhost:18080/job/fdroidserver/ws/.testfiles/tmp.9hr9NiSmZs/fdroidserver-0.4.0/fdroidserver/update.py",> line 1146, in main
    apks, cachechanged = scan_apks(apps, apkcache, repodirs[0], knownapks)
  File "<http://localhost:18080/job/fdroidserver/ws/.testfiles/tmp.9hr9NiSmZs/fdroidserver-0.4.0/fdroidserver/update.py",> line 543, in scan_apks
    thisinfo['sig'] = getsig(os.path.join(os.getcwd(), apkfile))
  File "<http://localhost:18080/job/fdroidserver/ws/.testfiles/tmp.9hr9NiSmZs/fdroidserver-0.4.0/fdroidserver/update.py",> line 361, in getsig
    with zipfile.ZipFile(apkpath, 'r') as apk:
  File "/usr/lib/python2.7/zipfile.py", line 770, in __init__
    self._RealGetContents()
  File "/usr/lib/python2.7/zipfile.py", line 857, in _RealGetContents
    x._decodeExtra()
  File "/usr/lib/python2.7/zipfile.py", line 388, in _decodeExtra
    tp, ln = unpack('<HH', extra[:4])
struct.error: unpack requires a string argument of length 4
2016-01-21 12:26:50 +01:00
Daniel Martí
0be343273d Apparently Python cannot handle Unicode
I really hope I can revert this in the near future. Having to mutilate
my name just so that pip will work is a terrible workaround.

For better or worse, this only affects scripts defined in setup.py.
2016-01-14 23:15:05 +01:00
Daniel Martí
fff658ba19 stats: use a stable sort that doesn't randomize
Since we only sorted by count, ignoring the string, it meant that items
with the same count might be arranged in different manners. Hence the
`stats` behaviour was not predictable at all. Now it sorts first by
count, then by string.
2016-01-11 02:43:06 +01:00