1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-14 19:10:11 +01:00
Commit Graph

4200 Commits

Author SHA1 Message Date
Torsten Grote
cb942168d5 Merge branch 'nail-down-btlog-and-bugfix' into 'master'
Nail down btlog and key bugfix

See merge request !248
2017-04-10 21:21:52 +00:00
Hans-Christoph Steiner
299ba86948 server: git push for binary transparency in fdroid server update
Since `fdroid server update` is the place where all uploads to servers
happens, it makes sense to also handle the git push for the binary
transparency log here instead of `fdroid btlog`
2017-04-10 23:02:12 +02:00
Hans-Christoph Steiner
e3152e9ef0 simplify Android Observatory config
This just keeps the config key name simple and direct.
2017-04-10 23:02:12 +02:00
Hans-Christoph Steiner
f3b55ab0f0 fix mistake: lists use .append() not .add()
This was a mistake in 5ad3486741
2017-04-10 23:02:12 +02:00
Hans-Christoph Steiner
5020f70997 btlog: fix bug with first run, the etag variable needs to exist 2017-04-10 20:30:13 +02:00
Torsten Grote
4e4c6d9acf Merge branch 'morebintrans' into 'master'
`fdroid btlog` for anyone to setup a binary transparency log for any repo

See merge request !243
2017-04-05 12:45:40 +00:00
Torsten Grote
255addc53c Merge branch 'miscfixes' into 'master'
misc fixes

Closes #268

See merge request !245
2017-04-04 21:28:08 +00:00
Hans-Christoph Steiner
136d58dbae btlog: test with fdroid update without requiring ssh access
Yay git!  Just use a filesystem git remote instead of ssh.
2017-04-03 23:17:03 +02:00
Hans-Christoph Steiner
4b7084f779 btlog: if git remote is specified, push commits to that remote
This makes fdroid automatically push the new binary transparency commits if
there is a git_remote specified in either config.py or from a CLI arg.
2017-04-03 20:31:42 +02:00
Hans-Christoph Steiner
53b69f507e allow index.TestCase to work when run using any path 2017-04-03 20:24:21 +02:00
Hans-Christoph Steiner
40290fc5e0 move make_binary_transparency_log to btlog
This keeps the code more organized, and reduces the number of things that
are loaded for every command via common.
2017-04-03 20:24:21 +02:00
Hans-Christoph Steiner
896c6496b4 remove setting git name/email for binary transparency logs
The machine running the steps should include the git config, so that it is
specific to that machines.
2017-04-03 20:24:21 +02:00
Hans-Christoph Steiner
6f71465ec1 support pretty output in JSON and for binary transparency logs
This makes make_index_v1() support `fdroid update --pretty`, then also uses
pretty output for the binary transparency logs, so that the git history has
nice, readable diffs between commits.
2017-04-03 20:24:21 +02:00
Hans-Christoph Steiner
c591a4cd89 fdroid btlog external binary transparency logger
This complements the binary transparency logging that happens on the
server side !226.  Anyone can set up an efficient tracker of any
F-Droid repo which stores all index files that it sees.  It uses HEAD
requests and ETag checking to be as efficient as possible, so that
this can be automatically run at a frequent pace.
2017-04-03 20:24:21 +02:00
Hans-Christoph Steiner
eadcd13723 move make_binary_transparency_log to common for easy reuse 2017-04-03 20:24:21 +02:00
Hans-Christoph Steiner
e58ad330f4 encode filenames as bytes to handle all locale setups
This was failing on environments that did not have any LANG or LC_* locale
variables set.  This is a valid setup, and is common in headless setups, so
it needs to be handled.

This also adds a new pass of the test suite without the locale env vars set
so that this situation is also tests on gitlab-ci, not only gpjenkins.

The error this caused was:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-18: ordinal not in range(128)
2017-04-03 20:24:00 +02:00
Hans-Christoph Steiner
4d50ab9bad ignore cache on key name change, instead of crashing
In 6c2cf2ccdd, the names of some essential
data bits changed.  If those names are not in the tmp/apkcache, then
`fdroid update` shouldn't crash but instead just ignore that cache entry.
tmp/apkcache should be deleted since the metadata version was bumped, but
I guess that does not always happen.
2017-04-03 20:20:25 +02:00
Hans-Christoph Steiner
5ad3486741 make sure indexes are sorted to minimize diffs between copies
With the binary transparency log stored in git, it makes that more readable
and as small as possible if the index files are fully sorted.  That will
reduce the differences between two copies of an index file to a bare
minimum.

If in the future we implement some kind of binary diff transfer, this will
also help there.
2017-04-03 20:18:54 +02:00
Hans-Christoph Steiner
7d4176dd50 metadata: don't crash if metadata includes blank Categories list
If someone includes just 'Categories:', it shouldn't crash.
2017-04-03 20:18:54 +02:00
Hans-Christoph Steiner
baeae862ad buildserver: allow gradle to install new components
Google is making gradle automatically download Android SDK components that
are needed by the build, but not already present.  We need to support that
since it would be a lot of work to fight it.  Plus, since each build starts
from the fresh snapshot, it should not be such a big deal to let each
build install stuff during the process.

closes #268
2017-04-03 20:18:54 +02:00
Hans-Christoph Steiner
d6310ed52c Merge branch 'invalid-apk-fix' into 'master'
Fix bug when scanning invalid APKs and add a test for it

See merge request !246
2017-04-03 17:38:51 +00:00
Torsten Grote
8f6fa2955d
Fix bug when scanning invalid APKs and add a test for it 2017-04-03 13:07:49 -03:00
Hans-Christoph Steiner
2ba757e6c9 Merge branch 'index-parsing' into 'master'
Download and return repository index

See merge request !240
2017-04-03 16:04:35 +00:00
Hans-Christoph Steiner
159ca824b7 Merge branch 'local-install' into 'master'
Fix installs with pip --user

See merge request !244
2017-04-03 14:45:45 +00:00
Torsten Grote
6552c1fa49
Fix installs with pip --user 2017-04-03 09:55:20 -03:00
Torsten Grote
7c34dd96f4
Reduce code duplication
by re-using methods for extracting and verifying certificate
2017-04-03 09:23:06 -03:00
Torsten Grote
a23da47118
Add method for downloading (and verifying) a repository index
This includes some test cases to test the new code.
2017-04-03 09:02:07 -03:00
Hans-Christoph Steiner
d8ad1c78c2 Merge branch '265-fdroid-init-generates-double-entries-in-config-py' into 'master'
avoid duplicate value assignments in config files

Closes #265

See merge request !241
2017-04-03 10:13:20 +00:00
Michael Pöhn
8b51e40d63 avoid duplicate value assignments when updating config files 2017-04-02 12:08:01 +02:00
Hans-Christoph Steiner
a6b3ffeeea Merge branch 'move-index-methods' into 'master'
Move index methods into dedicated modules

See merge request !239
2017-03-31 17:42:22 +00:00
Torsten Grote
389fd7e6c9
Re-use config and options from common module in index 2017-03-31 11:00:49 -03:00
Torsten Grote
f9321f6032
Move index related methods to new index module 2017-03-29 13:10:04 -03:00
Torsten Grote
9f765ed6f7
Move index signing methods into signindex.py 2017-03-29 12:11:40 -03:00
Hans-Christoph Steiner
256ef003fc Merge branch 'patch-1' into 'master'
Update support repository.

See merge request !238
2017-03-29 14:56:37 +00:00
Jan Christian Grünhage
f637b461f0 Update support repository. 2017-03-29 10:33:38 +00:00
Hans-Christoph Steiner
8f96c9da3d Merge branch 'smaller-methods' into 'master'
Factor out code into smaller methods to be used by repomaker

See merge request !236
2017-03-27 19:59:51 +00:00
Hans-Christoph Steiner
c3492ab153 Merge branch 'notabug' into 'master'
import: add notabug.org, auto-detect gradle, add options for license and categories

See merge request !237
2017-03-27 19:55:44 +00:00
Hans-Christoph Steiner
916e8eb108 Merge branch 'build_tools_log' into 'master'
post correct build tool versions to wiki

Closes #267

See merge request !235
2017-03-27 19:54:48 +00:00
Hans-Christoph Steiner
7f08fad2c6 Merge branch 'sign-and-verify-update' into 'master'
sign and verify update

See merge request !230
2017-03-27 19:48:36 +00:00
Boris Kraut
13b276e02f import: add option to specify license and categories, auto-detect build.gradle 2017-03-26 19:00:03 +02:00
Boris Kraut
39f7429c9b import: add notabug.org 2017-03-26 18:05:37 +02:00
Torsten Grote
c9b76eb808
Return public key and fingerprint after generating repo signing key 2017-03-22 15:53:20 -03:00
Torsten Grote
c484dc33be
Factor out code from update.main() to individual functions 2017-03-22 15:17:04 -03:00
Michael Pöhn
843595eb33 moved version android tools version log header to corresponding function 2017-03-22 14:56:26 +01:00
Michael Pöhn
0fda23ef80 fix local var name overloading when iterating build flavours 2017-03-22 14:20:54 +01:00
Michael Pöhn
c15f02ed68 fix exception handling in dscanner and update.py 2017-03-22 14:07:54 +01:00
Michael Pöhn
c45c4041a5 Merge remote-tracking branch 'fdroid/master' into build_tools_log 2017-03-22 14:02:45 +01:00
Michael Pöhn
83fe2bda4d fix exception handling in build.py 2017-03-22 14:01:32 +01:00
Hans-Christoph Steiner
223c793201 prefer apksigner if installed, jarsigner sucks
Google has their own utility for verifying APK signatures on a desktop
machine since Java's jarsigner is bad for the task.  For example, it
acts as if an unsigned APK validates.  And to check whether an APK is
unsigned using jarsigner is difficult.

apksigner also does the v2 signatures, so it will have to be used
eventually anyway.  It is already in Debian/stretch and can be
available in jessie-backports if need be.

https://android.googlesource.com/platform/tools/apksig
https://packages.debian.org/apksigner
2017-03-22 10:51:12 +01:00
Hans-Christoph Steiner
ffd490d8da set_command_in_config() for finding CLI tools to run 2017-03-22 10:51:12 +01:00