1
0
mirror of https://gitlab.com/fdroid/fdroidserver.git synced 2024-11-04 14:30:11 +01:00
Commit Graph

646 Commits

Author SHA1 Message Date
Andrea Scarpino
19a83b6219 Add Liberapay support 2017-12-12 11:53:31 +01:00
Hans-Christoph Steiner
6228162cbd handle jarsigner/apksigner output cleanly for rational logging
These were both spamming the output with lots of confusing messages, even
when --verbose was not used.  Jarsigner especially has confusing messages,
since it has warnings that do not pertain to APK signatures at all, like
the ones about timestamps and missing Certificate Authority.

closes #405
2017-12-07 17:32:14 +01:00
Hans-Christoph Steiner
05abbfbabd gitlab-ci: move sdist test run to new fedora job
A full run of the test suite takes quite a bit of time.  This removes one
of the 3 runs from the main 'tests' job, and puts it into the Fedora job.
That test run is mostly to make sure the setup.py and source tarball are
correctly, so that doesn't affect merge requests very often.

This also tests `pip install --user`, which was not really being tested
before.
2017-12-06 20:20:17 +01:00
Marcus Hoffmann
bfe2c00834 common.testCase: fix find_sdk_tools when aapt is installed in /usr/bin
The testlogic was broken when having both aapt in /usr/bin and also as
part of the android sdk.
2017-12-06 12:30:47 +01:00
Hans-Christoph Steiner
1c3a4479ab add common.sign_apk() for nighly as test for using in publish
Since the MD5 migration was quite a bit of work, it makes sense to start
on moving away from SHA1 as much as possible while it is easy to do. SHA256
will only work in APK signatures on android-18 (4.3) or newer.  So if an
APK has a minSdkVersion of 18 or newer, then sign with SHA256.

https://issuetracker.google.com/issues/36956587
https://android-review.googlesource.com/c/platform/libcore/+/44491
2017-12-04 22:52:41 +01:00
Hans-Christoph Steiner
3f4e93bf86 Merge branch 'metadataFlavours' into 'master'
Add flavour to metadata

See merge request fdroid/fdroidserver!370
2017-12-02 20:51:24 +00:00
tobiasKaminsky
d5ecb34df7
add flavour to metadata 2017-12-01 15:25:00 +01:00
Hans-Christoph Steiner
5b22ff7dc6 tests: use standard dir setup so all tests start in same dir 2017-11-30 17:32:53 +01:00
Hans-Christoph Steiner
3ff4b656c6 tests: name temp test dir after test function that used it 2017-11-30 17:32:53 +01:00
Hans-Christoph Steiner
30b2f5a48a build: sort fdroid build --all by most recently changed first 2017-11-30 17:32:53 +01:00
Hans-Christoph Steiner
c7c40cb59f PEP8 fixes 2017-11-30 13:42:37 +01:00
tobiasKaminsky
33aee96ed9
added test case 2017-11-30 11:12:18 +01:00
Hans-Christoph Steiner
743ef9ba00 rename test files to work round filename length limits
It seems a lot of things like tar, Windows, some filesystems, etc. do not
handle long filenames. #416
2017-11-27 16:57:30 +01:00
Hans-Christoph Steiner
e33684fc9b tests: new tests for stacktraces caused by translated help strings 2017-11-22 11:40:16 +01:00
Hans-Christoph Steiner
cbd9669105 rename test file to fit within eCryptfs filename limits (closes #416) 2017-11-22 11:40:16 +01:00
Hans-Christoph Steiner
cd3e763953 tests: only test fdroid build if the right build-tools is installed 2017-11-09 14:07:40 +01:00
Hans-Christoph Steiner
a14124955a tests: fix fdroid build test with Debian gradle 2017-11-09 13:11:49 +01:00
Hans-Christoph Steiner
2ee32f3524 fix tests to work when only Debian Android SDK is installed
jenkins.debian.net only had `apt install android-sdk`, it does not have any
of Google's packages installed.
2017-11-08 14:05:17 +01:00
Hans-Christoph Steiner
17efa13183 tests: pre-set failfast as reminder of a handy time saver 2017-10-25 23:01:25 +02:00
Hans-Christoph Steiner
e0df6d2479 choose the most recent available version of Java
This came about testing on OSX, where there are often multiple versions of
the JDK installed.  This was choosing the oldest version.  It should
choose the most recent version.
2017-10-25 23:01:25 +02:00
Hans-Christoph Steiner
49bd5a6462 use Babel to compile localizations and include in source tarball
python3-babel was also included in the gitlab-ci docker image:
ffc08dbc1d
2017-10-19 23:28:13 +02:00
Hans-Christoph Steiner
7d4e354f25 FDroidException handle Exceptions that return things other than str
This is related to a4c4a16ed9
2017-10-19 20:38:43 +02:00
Hans-Christoph Steiner
af0d8ab84c tests: make pip3 install quieter for shorter CI logs 2017-10-19 20:23:57 +02:00
Michael Pöhn
788c8f97fd better error message in publish when repo_key is not set 2017-10-17 22:10:39 +02:00
Michael Pöhn
89d9d0c45b use OrderedDict for localization data in index-v1 2017-09-26 14:15:48 +02:00
Michael Pöhn
26192a00e2 make sure all timestamps in index-v1 are utc 2017-09-26 14:15:27 +02:00
Michael Pöhn
a2d361dec8 add test for index-v1.json 2017-09-26 14:12:59 +02:00
Michael Pöhn
b6b79edf11 store fdroid signing-key-fp-list in stats dir 2017-09-26 14:11:09 +02:00
Michael Pöhn
6930edf889 sort index-v1; publish now creates and stores a list of signature fingerprints 2017-09-26 14:11:09 +02:00
Michael Pöhn
bca07f794f added functions for storing/loading signer fingerprints to stats 2017-09-26 14:11:09 +02:00
Michael Pöhn
5a524d4d0c added some developer-signing key metadata to tests 2017-09-26 14:11:09 +02:00
Michael Pöhn
6a4a78429f add signer to index v1 2017-09-26 14:11:09 +02:00
Michael Pöhn
7c4b8bcac4 added parse release filename function 2017-09-26 14:11:09 +02:00
Michael Pöhn
04daa7a03a test for common.get_app_id_aapt 2017-09-26 14:11:09 +02:00
Michael Pöhn
efb0bf6ee5 add common functions for dealing with apk signatures 2017-09-26 14:11:09 +02:00
Michael Pöhn
c196f7dd7a common function for fetching sha256 signing-key fingerprint 2017-09-26 14:11:09 +02:00
Hans-Christoph Steiner
eaca20675c tests: test setting up app git repo like fdroid build does
I wrote this to try to find #379 but this test didn't trigger that bug.
2017-09-25 16:35:18 +02:00
Hans-Christoph Steiner
cbf7ba0414 tests: add setUp() method to common to handle standard stuff 2017-09-25 16:35:18 +02:00
Hans-Christoph Steiner
2764c43fa2 tests: use ci-test-app with current Gradle Android Plugin
This works around the gradle 2.x bug where versions newer than 2.9 cannot
run things.  This also then specifies the version of Gradle Android Plugin
that is included in Debian/stretch.
2017-09-19 20:16:08 +02:00
Hans-Christoph Steiner
dc569b9c18 tests: support Java setups where MD5 is not disabled by default
For platforms using Java < 1.8.0_133, MD5 is still enabled for JAR
signatures.  Its just too painful to manage all this, so support this
in the tests.
2017-09-19 20:16:08 +02:00
Hans-Christoph Steiner
373b46ab3f tests: don't run gpgsign tests on Travis CI's OSX, gpg always fails 2017-09-19 20:16:08 +02:00
Hans-Christoph Steiner
d1f6126dea tests: support Ubuntu/trusty's python3.4-venv 2017-09-19 20:13:36 +02:00
Hans-Christoph Steiner
bc192b6be0 tests: prefer GNU sed on BSD/OSX, BSD's sed has lame syntax
https://blog.remibergsma.com/2012/09/18/sed-inline-editing-different-on-mac-osx/
2017-09-19 20:13:36 +02:00
Hans-Christoph Steiner
176f539647 allow spaces in filenames
This fixes all the bugs I could find that prevented fdroid from
handling files with spaces in them.  This is more important now that
fdroid supports random media files, and Repomaker
2017-09-19 20:13:36 +02:00
Hans-Christoph Steiner
6adf309bef tests: move test APK with bad unicode filename to separate test
This filename has some messed up bytes related to bi-directional script
that is included (Left-to-Right and Right-to-Left).  GNU/Linux always
interprets filenames as pure byte sequences.  Windows and OSX store
filenames as Unicode strings.  So on OSX, the invalid filename gets
converted to a valid name.  That works fine, but the test fails because it
is compared to a file generated on Ubuntu, where it preserves the byte
sequence.

This includes an APK with a valid Unicode filename that includes
bi-directional script.
2017-09-19 20:13:36 +02:00
Hans-Christoph Steiner
5dcb48831f index: always use jarsigner for verifying JAR signatures
apksigner v0.7 (build-tools 26.0.1), Google made it require that the
AndroidManifest.xml was present in the archive before it verifies the
signature.  So this needs to stick with the jarsigner hack for JARs.
2017-09-19 20:13:36 +02:00
Hans-Christoph Steiner
acac15f74d Merge branch 'newcomers' into 'master'
allow starting without config.py

See merge request fdroid/fdroidserver!335
2017-09-16 08:07:56 +00:00
Hans-Christoph Steiner
e4ca82d80c tests: support running on platforms with old java
In order to test on travis-ci's OSX setup, its easiest to support the old
version of Java that is installed there by default.  It is pretty old, so
it does not include the April 2017 update that disables MD5:

https://blogs.oracle.com/java-platform-group/oracle-jre-will-no-longer-trust-md5-signed-code-by-default
https://www.bleepingcomputer.com/news/security/oracle-to-block-jar-files-signed-with-md5-starting-with-april-2017
https://support.ca.com/us/knowledge-base-articles.TEC1691042.html

#292 #323
2017-09-15 11:37:24 +02:00
Marcus Hoffmann
34f79b876c tests: don't touch config.py, no config is fine now 2017-09-14 02:15:57 +02:00
Michael Pöhn
3e6dfacf6c added signatures subcommand 2017-09-07 00:03:35 +02:00
Hans-Christoph Steiner
c67f8e349c metadata: handle empty YAML files without crashing
!311
2017-07-26 18:21:28 -07:00
Hans-Christoph Steiner
7bd171480f update: test metadata creation using internal and external templates
There is a hardcoded template in update.py, and there is also the
possibility for the user to create a template.yml.  This tests both of them
and cleans up the related code a bit.

#352 !310
2017-07-26 15:34:13 -07:00
Hans-Christoph Steiner
0a953814e2 update: always include name/summary/desc in index.xml if available
With the new localization support, the name/summary/description in the
metadata file becomes the global override.  So most apps are not going to
have those fields present in their metadata file.  This fixes the index.xml
generation to fall back to the localized versions of those fields when they
are not set in the metadata field.

https://forum.f-droid.org/t/what-has-happend-to-osmand
2017-07-19 15:07:50 +02:00
Hans-Christoph Steiner
523b5f4777 server: smooth out btlog transfer for offline signing setups
It turns out it is error prone to `git push` to a non-bare git repo. For
the offline signing machine, the git remote needs to be a regular git repo
in a directory on a thumbdrive so that once the thumbdrive is plugged into
an online machine, that git repo can be transferred to the online machine.
2017-07-17 11:38:15 +02:00
Hans-Christoph Steiner
25f96e1911 ensure that mirror URLs always include the repodir
Since the mirror URLs are per repo section (repo/archive), the mirror URLs
must include the repodir at the end.  This was missing for servergitmirrors

found by @cde when working on fdroidclient#35
2017-07-07 15:50:11 +02:00
Hans-Christoph Steiner
dfb07808d3 support configing buildserver VM per-build with sudo=
This adds the 'sudo' build field, which is just a script that is run as
root.  For more info, see the issue that this closes:

refs #318
closes #317
2017-07-06 17:29:23 +02:00
Hans-Christoph Steiner
4a15208b84 update: update openssl KnownVuln scan to handle all recent versions
Thanks to @bubu for reporting!
2017-07-06 13:31:42 +02:00
Hans-Christoph Steiner
df99c85ca6 support manually adding per-build antiFeatures in metadata
For cases like the OpenVPN vuln that was recently announced, it is useful
for fdroiddata maintainers to be able to mark builds that have known
vulnerabilities.
2017-07-06 12:25:48 +02:00
Michael Pöhn
071ad46b55 moved some yaml test files into a separate forlder to avoid conflicts with other test cases 2017-07-04 13:35:05 +02:00
Michael Pöhn
19d422da86 rewritemeta yaml: fixed boolen mapping for build flags 2017-07-04 11:51:08 +02:00
Michael Pöhn
fc070ee657 refactored yaml tests into metadata test case; fixed typo 2017-07-04 11:51:08 +02:00
Michael Pöhn
f1e0089931 prettify write_yaml metadata 2017-07-04 11:51:08 +02:00
Michael Pöhn
7aa92abebc yaml rewrite version code as int 2017-07-04 11:51:08 +02:00
Michael Pöhn
f9d4007ea6 rewrite metadata builds list to yaml 2017-07-04 11:51:08 +02:00
Michael Pöhn
735b7d0f33 fixed pyflakes/pep warings 2017-07-04 11:51:08 +02:00
Michael Pöhn
0f90ab9aac rewrite to yaml works for app data now (builds still missing) 2017-07-04 11:51:08 +02:00
Torsten Grote
0bbe0c5db0
Break up the scan_apk() method and rename it to process_apk() 2017-07-03 18:37:01 -03:00
Hans-Christoph Steiner
5f0817d7bf tests: make sure apkcache gets created 2017-07-03 10:02:51 +02:00
Hans-Christoph Steiner
746d4bd4cf update: allow_disabled_algorithms option to keep MD5 sigs in repo
The new policy is to move APKs with invalid signatures to the archive,
and only add those APKs to the archive's index if they have valid MD5
signatures.

closes #323
closes #292
2017-07-03 10:02:51 +02:00
Hans-Christoph Steiner
e1492148fa fix "Archive Policy:" field, APKs can move in/out of archive
The original logic was checking keepversions against the len() of ALL the
APKs in the repo/archive.  The correct thing is to check against the
number of APKs available for the given packageName/appid.

closes #166
2017-06-28 23:53:42 +02:00
Hans-Christoph Steiner
be2926ffc8 tests: test moving files to and from the archive
#166
2017-06-28 23:53:42 +02:00
Hans-Christoph Steiner
214c9f7a2b lint: check file extension for metadata
This helps keep fdroiddata clean, on @krt's request.

closes #222
2017-06-28 23:14:57 +02:00
lb@lb520
bab8f8ea02 Test metadata files updated. 2017-06-21 08:26:52 +02:00
Hans-Christoph Steiner
d06f05e9c8 update: do not crash on screenshots without texts/graphics
`fdroid update` crashed for apps that only had screenshots but no graphics
or localized texts because destdir was not being set in that case.  This
fixes that and adds a test case.

closes #320
!286
2017-06-12 21:25:54 +02:00
Hans-Christoph Steiner
8776221988 check signature and OpenSSL after APK has proven valid
If working with a random grabbag of APKs, there can be all sorts of
issues like corrupt entries in the ZIP, bad signatures, signatures that
are invalid since they use MD5, etc.  Moving these two checks later means
that the APKs can be renamed still.

This does change how common.getsig() works.  For years, it returned
None if the signature check failed.  Now that I've started working
with giant APK collections gathered from the wild, I can see that
`fdroid update` needs to be able to first index what's there, then
make decisions based on that information.  So that means separating
the getsig() fingerprint fetching from the APK signature verification.

This is not hugely security sensitive, since the APKs still have to
get past the Android checks, e.g. update signature checks.  Plus the
APK hash is already included in the signed index.
2017-06-01 17:45:29 +02:00
Hans-Christoph Steiner
372c8b418d strip file extension from generated name for non-APKs
With a generic file, the file name is the only guaranteed name metadata
field.  So if the name is not specified in the metadata, then the name
is set to the filename.  This changes that so that the file extension is
stripped from that generated name.
2017-06-01 16:01:05 +02:00
Hans-Christoph Steiner
9471bf2731 regexs for getting packageName and versionCode from filenames
This is useful for parsing APK files, which can include packageName,
versionCode, and optionally 7 char signing key ID (i.e. <sig>).
This also can set the packageName and versionCoe for non APK files, so
that it is easy to assign them to metadata files, and to allow for
upgrades by setting the versionCode in the filename.
2017-06-01 16:01:05 +02:00
Hans-Christoph Steiner
ceac6d25cb index.xml cannot handle APKs with the same packageName/versionCode
Really, it is the fdroidclient parser of index.xml that fails, due to the
hardcoded expectation that there will only ever be a single APK for any
given versionCode.  We keep index.xml backwards compatible for old
clients, and use index-v1.json to support new things.  Having multiple
APKs that have the same packageName and versionCode will break the client
v0.103.* since that version uses index-v1.json, but still has the hard-
coded database parsing stuff.

#153
2017-06-01 16:01:05 +02:00
Hans-Christoph Steiner
4053f03d77 update: add --rename-apks to force APK filenames to fdroid standard
uses the standard package.name_123.apk.  If that exists, it appends the
shasum.  If that exists, then its a duplicate, so its deleted. This should
help @SergeWinters with his 12,000 APKs.
2017-06-01 16:01:05 +02:00
Hans-Christoph Steiner
0f4cbc7224 allow APKs with same packageName/versionCode but different signer
There are many APKs out in the wild that claim to be the same app and
version and each other, but they are signed by different keys.  fdroid
should be able to index these, and work with them.   This supports having
the developer's signature via reproducible builds, random collections of
APKs like repomaker, etc.
2017-06-01 10:30:41 +02:00
Hans-Christoph Steiner
afac1b2efd add basic test for fdroid scanner
There was no test coverage at all for this command, this is a very basic
test that should prevent things like 2626858450
2017-05-30 21:13:49 +02:00
Hans-Christoph Steiner
114d5dc8c6 make tests pass new lint rules 2017-05-26 21:03:38 +02:00
Hans-Christoph Steiner
17cd07f1a7 fix pylint unused-argument 2017-05-23 22:34:16 +02:00
Hans-Christoph Steiner
12c80f9062 update outdated pylint setup
The logilab-astng lib is dead, replaced by python-astroid.  The crazy astng
plugin is no longer needed also.

#281
2017-05-23 22:34:16 +02:00
Hans-Christoph Steiner
c749c68486 test script for vmtools 2017-05-23 20:06:06 +02:00
Hans-Christoph Steiner
0345d2c79f update: find aapt when it is not in the PATH
This is some very messy logic built up since 2010.  This will all go away
once we have a python3 version of androguard available.

The removed imports and `dir(APK)` is to silence pyflakes

closes #303
2017-05-18 17:21:05 +02:00
Hans-Christoph Steiner
c348186ad6 update: fix fastlane scraping, it uses a subdir called 'images'
https://commons.wikimedia.org/wiki/File:GetChromium_FeatureGraphic_1024x500.png
2017-05-18 16:13:17 +02:00
thez3ro
06598ae406
use androguard if aapt isn't found 2017-05-04 23:35:17 +02:00
Torsten Grote
8d424f19ec
Support ETag when downloading repository index 2017-05-02 15:37:02 -03:00
Hans-Christoph Steiner
cdef5bcd92 downcase all 'localized' key names to match the rest of index-v1
This is a little omission.  keys that are used in metadata/*.yml all start
with an UpperCase letter, but in fdroidserver, index-v1.json, and
fdroidclient, it is all camelCase with lowercase first letter. The keys
from the 'localized' section are currently never in metadata/*.yml, so
these keys never get downcase.  This change will break fdroidclient
versions that do not also have this change, but since we're in alpha, that
should be fine.

If support for a 'localized' section is added to metadata/*.yml, then the
keys there should probably be UpperCase CamelCase to match the other keys.
2017-04-27 21:12:49 +02:00
Hans-Christoph Steiner
82095c7a9a add basic test for Triple-T Gradle Play Publisher scraping 2017-04-19 10:05:24 +02:00
Hans-Christoph Steiner
cb49f57c06 support "Author Web Site" as metadata field
Fastlane Supply, Triple-T Gradle Play Publisher, and many app stores
include the possibility to specify a website for the author, as distinct
from the website for the app.

closes #204
2017-04-19 10:05:24 +02:00
Hans-Christoph Steiner
9589d13ef2 update: include "What's New" texts when they are available
This uses the "What's New" entry for the CurrentVersionCode and includes it
as the current WhatsNew metadata for the App class.

Things like fastlane supply and Google Play support a "What's New" entry
per-APK, but fdroidclient does not current use anything but the current
version of this data.  Right now, it seems we probably only want to have
the latest WhatsNew in the index to save space.

In theory, we could make the WhatsNew data structure follow the structure
of fastlane/Play, but that would quite a bit of complexity for something
that might never be used.

fdroidclient#910
2017-04-19 10:05:24 +02:00
Hans-Christoph Steiner
31d92367ed git mirror test requires git >= 2.3 2017-04-13 00:49:36 +02:00
Hans-Christoph Steiner
17219d56f4 server: support git mirrors with the offline signing setup 2017-04-12 15:04:04 +02:00
Hans-Christoph Steiner
f75b547795 server: remove unused --sync-from-local-copy-dir
This option was not hooked up at all, and does not make sense as a command
line argument.  It should just be a config.py item.  In that case, the
presence of config.py marks the current dir as a repo, so there is no
longer a need to test for a dir called repo/ as a safety.  This makes the
setup easier, since sync_from_localcopy() now creates repo/ for the user.
2017-04-12 00:45:22 +02:00
Hans-Christoph Steiner
299ba86948 server: git push for binary transparency in fdroid server update
Since `fdroid server update` is the place where all uploads to servers
happens, it makes sense to also handle the git push for the binary
transparency log here instead of `fdroid btlog`
2017-04-10 23:02:12 +02:00
Hans-Christoph Steiner
f3b55ab0f0 fix mistake: lists use .append() not .add()
This was a mistake in 5ad3486741
2017-04-10 23:02:12 +02:00
Hans-Christoph Steiner
136d58dbae btlog: test with fdroid update without requiring ssh access
Yay git!  Just use a filesystem git remote instead of ssh.
2017-04-03 23:17:03 +02:00
Hans-Christoph Steiner
53b69f507e allow index.TestCase to work when run using any path 2017-04-03 20:24:21 +02:00
Hans-Christoph Steiner
d6310ed52c Merge branch 'invalid-apk-fix' into 'master'
Fix bug when scanning invalid APKs and add a test for it

See merge request !246
2017-04-03 17:38:51 +00:00
Torsten Grote
8f6fa2955d
Fix bug when scanning invalid APKs and add a test for it 2017-04-03 13:07:49 -03:00
Torsten Grote
a23da47118
Add method for downloading (and verifying) a repository index
This includes some test cases to test the new code.
2017-04-03 09:02:07 -03:00
Michael Pöhn
8b51e40d63 avoid duplicate value assignments when updating config files 2017-04-02 12:08:01 +02:00
Torsten Grote
9f765ed6f7
Move index signing methods into signindex.py 2017-03-29 12:11:40 -03:00
Hans-Christoph Steiner
223c793201 prefer apksigner if installed, jarsigner sucks
Google has their own utility for verifying APK signatures on a desktop
machine since Java's jarsigner is bad for the task.  For example, it
acts as if an unsigned APK validates.  And to check whether an APK is
unsigned using jarsigner is difficult.

apksigner also does the v2 signatures, so it will have to be used
eventually anyway.  It is already in Debian/stretch and can be
available in jessie-backports if need be.

https://android.googlesource.com/platform/tools/apksig
https://packages.debian.org/apksigner
2017-03-22 10:51:12 +01:00
Hans-Christoph Steiner
998b6245e9 verify: ensure only a single signature is in compared APK
The ZIP format allows multiple entries with the exact same filename, and on
top of that, it does not allow deleting or updating entries.  To make the
`fdroid verify` procedure failsafe, it needs to create a new temporary APK
that is made up on the contents of the "unsigned APK" and the signature
from the "signed APK".  Since it would be possible to give a signed APK as
in the unsigned one's position, `fdroid verify` was not able to update the
signature since it was just adding the new signature to the end of the ZIP
file.  When reading a ZIP, the first entry is used.
2017-03-22 10:51:12 +01:00
Hans-Christoph Steiner
866528de5b signindex: support signing index-v1.jar
This is a bit different than index.jar: instead of their being index.xml
and index_unsigned.jar, the presense of index-v1.json means that there is
unsigned data.  That file is then stuck into a jar and signed by the
signing process.  index-v1.json is never published to the repo.  It is
included in the binary transparency log, if that is enabled.
2017-03-17 14:12:03 +01:00
Hans-Christoph Steiner
fa657ce720 move update.signjar() to common so it can also be used in signindex 2017-03-17 13:55:40 +01:00
Hans-Christoph Steiner
713d9195c3 ensure that app.Categories is always a list of strings
This just makes it easier for people writing build recipes.  Rewriting will
output a list of strings as well.

The test index.xml and categories.txt are updated to include the new number
categories, and the changed CurrentVersionCode to 2147483647 (MAX_VALUE)
2017-03-17 13:55:40 +01:00
Hans-Christoph Steiner
7044a909ca include graphics and screenshots from repo in V1 index
If the repo has store graphics and/or screenshots, then include those in
the metadata.  This follows the possible graphics for Google Play, and the
file naming scheme of the open source 'fastlane' tool for managing those
files.

https://github.com/fastlane/fastlane/blob/1.109.0/supply/README.md#images-and-screenshots
https://support.google.com/googleplay/android-developer/answer/1078870?
https://android-developers.blogspot.com/2011/10/android-market-featured-image.html

Signed-off-by: Hans-Christoph Steiner <hans@eds.org>
2017-03-17 13:55:40 +01:00
Hans-Christoph Steiner
6c2cf2ccdd convert internal dict keys to match fdroidclient's Apk class
This syncs up the field names between the fdroiddata .yml files, the keys
used in the implementation in fdroidserver, the index data format, and the
final data structures in fdroidclient.  This makes it easier for devs to
follow, and makes the Jackson parsing library automatically handle
converting the data from the index file to Java instances.

This bumps the metadata version since the apkcache will have to be
discarded.

Here are the name changes:
* apkname --> apkName
* id --> packageName
* sha256 --> hash
* version --> versionName
* versioncode --> versionCode

tests/repo/index.xml was changed only to bump the metadata version
from 17 to 18.
2017-03-17 13:55:40 +01:00
Hans-Christoph Steiner
c9aa26d89e add index V1 format, a direct translation of internal dict
Python encode/decode libs work directly with dicts, so the internal dict
can just be passed directly to any of these libs (pyyaml, pyjson, msgpack,
simplejson, etc).  This still generates the exact same index.xml as before.

This converts the internal format for the repo timestamp to a datetime
instance, which can be easily converted to UNIX time in seconds for XML
and UNIX time in milliseconds for the new index formats.  UNIX time in
milliseconds is directly serialized into a java.util.Date instance by
Jackson.
2017-03-17 13:55:40 +01:00
Hans-Christoph Steiner
b6d8916881 Merge branch 'metadata_output_vars' into 'master'
metadata output-lines now substitute $$VERSION$$/etc.

Closes #263

See merge request !228
2017-03-17 11:37:18 +00:00
Hans-Christoph Steiner
a737255ce4 add a simple test of the binary transparency log
This makes sure it gets generated with two commits: README and first log.
2017-03-16 19:22:32 +01:00
Michael Pöhn
d2570a493e metadata output-lines now substitute $$VERSION$$/etc. 2017-03-16 09:24:05 +01:00
Hans-Christoph Steiner
6aca906863 include non-APKs that match metadata and fdroid-generated filenames
Since it is now possible to build and include arbitrary files, like OTA
update ZIP files, the update procedure needs to look for non-APK files that
match the packageName_versionCode pattern of fdroid-generated files.

!193
admin#14
privileged-extension#9
2017-03-10 18:37:18 +01:00
Hans-Christoph Steiner
d92ac1e8f6 fix metadata dump test to run on CI servers 2017-02-28 12:08:04 +01:00
Hans-Christoph Steiner
8b85c860a6 rename lastupdated to lastUpdated to match fdroid client
This lets index-v1 be parsed directly into class instances because the
field/instance var names match exactly.  The original index v0 element
must retain the 'lastupdated' name for backwards compatibility.
2017-02-28 12:08:04 +01:00
Hans-Christoph Steiner
8decd3b504 remove TeX manual, it moved to https://gitlab.com/fdroid/fdroid-website
The whole TeX manual has been folded into the new Jekyll-based website, so
this is no longer needed at all.
2017-02-24 14:01:31 +01:00
Hans-Christoph Steiner
4e39621601 provide warning if config items will not preserve order
If a group of items are enclosed in {}, then that will be a Python set,
which does not preserve order.  To preserve order, the data must be either
a tuple () or list [].
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
388c336e76 build: ensure test is running on git commit that this code works with
Since https://gitlab.com/fdroid/ci-test-app is a separate git repo, things
with incompatible changes could get out of sync.  Therefore, this test
should specify which git commit is runs against.

For example, the .fdroid.yml file is still a moving target.  Just now, the
keys had the spaces removed as part of this MR.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
593613043e normalize Build TYPE_STRING data based on .txt
Unfortunately, things like versionCode must be strings.  That should be
converted to be ints throughout.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
c178cfb843 normalize Build TYPE_LIST data based on .txt
In the future, we should have better internal datatypes for this stuff,
i.e. instead of gradle: ['yes'] for True, actually use a boolean.  For now,
make the YAML and JSON metadata produce the same internal data as .txt.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
f7ae398aae add script for mass testing metadata parsing after changes
This requires manually running it.  I suppose it would be possible to
include a snapshot of the dumped internal representation for each release,
then make the tests run automatically against that.  Right now, the dump is
17megs of YAML.  Seems large to include in this git repo.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
e0f39a7e7b rename Build fields: version -> versionName, vercode -> versionCode
Since the YAML/JSON/etc. field names are now exactly the same as the field
names used in the internal dict in the Build class, this is a global rename

This keeps with the standard names used in Android:
https://developer.android.com/guide/topics/manifest/manifest-element.html
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
c0bc3afda9 convert metadata.Build to a subclass of dict
Like with the App class in the commit before, this makes it a lot
easier to work with this data when converting between the internal
formats and external formats like YAML, JSON, MsgPack, protobuf, etc.

The one unfortunate thing here is Build.update. It becomes
dict.update(), which is a method not an attribute.
build.get('update') or build['update'] could be used, but that would
be oddly inconsistent. So instead the field is renamed to
'androidupdate', except for in the .txt v0 metadata files. This better
describes what field does anyway, since it runs `android update`.

Build.update is only referenced in two places right next to each other
for the ant builds, so this change still seems worthwhile.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
b7fc7f2228 convert App to subclass of dict to support parsing/dumping libs
Python is heavily based on its core data types, and dict is one of the more
important ones.  Even classes are basically a wrapper around a dict. This
converts metadata.App to be a subclass of dict so it can behave like a dict
when being dumped and loaded.  This makes its drastically easier to use
different data formats for build metadata and for sending data to the
client.  This approach will ultimately mean we no longer have to maintain
custom parsing and dumping code.

This also means then that the YAML/JSON field names will not have spaces in
them, and they will match exactly what it used as the dict keys once the
data is parsed, as well as matching exactly the instance attribute names:

* CurrentVersion: 1.2.6
* app['CurrentVersion'] == '1.2.6'
* app.CurrentVersion == '1.2.6'

Inspired by:
https://goodcode.io/articles/python-dict-object/
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
56ef716a4e test for original "v0" index XML compatibility
The original index.xml format needs to stay around for backwards
compatibility, but we shouldn't touch it anymore once the new format is in
place.  This is a test to make sure `fdroid update` can still generate the
correct XML.

install_list and uninstall_list should be tuples or lists in order to
ensure that the order is preserved.

These tests also check that the added and lastupdated dates are
working correct, based on the dates in tests/stats/known_apks.txt. I
could see no useful way to test the timestamp, it is just hardcoded
using a regexp search-and-replace.  Running these tests manually might
require deleting tmp/apkcache.
2017-02-24 11:01:01 +01:00
Hans-Christoph Steiner
36272656fc gitlab-ci: workaround "ImportError: No module named 'packaging'"
https://github.com/pypa/setuptools/issues/937

fdroid/ci-images#1
2017-02-06 17:28:07 +01:00
Hans-Christoph Steiner
d08bd32a35 update URL for git repo of test app 2017-01-20 12:12:20 +01:00
Hans-Christoph Steiner
b1a943d562 convert metadata test dumps to YAML for easy comparison
When making code changes related to the metadata parsing, it is useful to
see how the internal format has changed by seeing the differences in the
dump files.  Those files are currently in the binary .pickle format.  This
just straight converts them to YAML, which is a text format, so that normal
diff tools work to see changes.

The dump files are named .yaml instead of .yml since .yml is used for hand-
edited YAML files for fdroiddata/metadata, while these dump files here are
a human readable form of a Python pickle.
2016-12-05 20:16:54 +01:00
Hans-Christoph Steiner
ce3efe4168 remove support for XML app metadata, its broken
JSON and YAML are very closely related, so supporting both of them is
basically almost no extra work.  Both are also closely related to how
Python works with dicts and pickles. XML is a very different beast, and its
not popular for this kind of thing anyway, so just purge it.
2016-12-05 20:16:54 +01:00
Hans-Christoph Steiner
a4e4310803 allow metadata to be embedded in source repos via .fdroid.yml
This allows a source repo to include a complete metadata file so that it
can be built directly in place using `fdroid build`.  If that app is then
included in fdroiddata, it will first load the source repo type and URL
from fdroiddata, then read .fdroid.yml if it exists, then include the rest
of the metadata as specified in fdroiddata, so that fdroiddata has
precedence over the metadata in the source code.

This lets `fdroid build` apps without having a whole fdroiddata setup, but
instead just directly in place in the source code.  This also lets devs
optionallu maintain the fdroid metadata as part of their app, rather than
in fdroiddata without loosing any control.  This should make it easier to
spread around the maintenance load.
2016-11-16 23:28:03 +01:00
Hans-Christoph Steiner
b4a39ee272 switch import test to custom, small test app
The test project should be moved to https://gitlab.com/fdroid/ci-test-app
2016-11-16 23:28:03 +01:00
Hans-Christoph Steiner
1f55a40caa properly parse build metadata list types like gradle=
Something like `gradle: yes` in YAML will be parsed as a boolean, since
'yes' is officially defined as a boolean true in YAML.  For metadata fields
that need to be lists, this needs to be converted.  Same goes for a single
string like `gradle: customFlavor`.
2016-11-16 23:28:03 +01:00
Daniel Martí
82b1d7ad14 all: make newer pycodestyle happy
Apparently the "two empty lines" rule is now stricter.
2016-11-15 20:55:06 +00:00
Hans-Christoph Steiner
56d51fcd6b gpg-sign all valid files in the repo, including source tarballs
This makes sure there is a GPG signature on any file that is included in
the repo, including APKs, OBB, source tarballs, media files, OTA update
ZIPs, etc.  Having a GPG signature is more important on non-APK files since
they mostly do not have any signature mechanism of their own.

This also adds basic tests of adding non-APK/OBB files to a repo with
`fdroid update`.

closes #232
2016-11-07 14:53:01 +01:00
Hans-Christoph Steiner
84e09cd2a2 allow arbitrary build products, not only APKs
This makes it so that the final build product can be specified in output=
and it'll work no matter if its an APK or not.  This was developed around
the case of building the OTA update.zip for the Privileged Extension. It
should work for any build process in theory but it has not yet been tested.

https://gitlab.com/fdroid/privileged-extension/issues/9
2016-11-07 14:53:01 +01:00
Hans-Christoph Steiner
47d9fd330d remove unused 'apps' argument from update.scan_apks() 2016-11-02 16:11:52 +01:00
Hans-Christoph Steiner
008b4a31eb skip fdroid import test if gitlab is not available
This prevents CI build failures when gitlab is deploying or has other
issues that might cause a 500.
2016-09-15 23:20:18 +02:00
Hans-Christoph Steiner
6126b55136 rename server request from "delete" to "uninstall"
This matches the Android API's current Intent action for this, rather than
the deprecated one:

https://gitlab.com/fdroid/fdroidclient/blob/v0.101-alpha5/app/src/main/java/org/fdroid/fdroid/installer/DefaultInstallerActivity.java#L147
https://developer.android.com/reference/android/content/Intent.html#ACTION_UNINSTALL_PACKAGE
https://developer.android.com/reference/android/content/Intent.html#ACTION_DELETE
2016-08-24 23:05:34 +02:00
Hans-Christoph Steiner
85632ba00e set up install/delete lists for "push" commands from server
It is now possible for the server operator to specify lists of apps that
must be installed or deleted on the client (aka "push installs).  If
the user has opted in, or the device is already setup to respond to
these requests, then fdroidclient will automatically install/delete
the packageNames listed.  This is protected by the same signing key
as the app index metadata.

It generates single XML elements with the data set in the attributes. This
keeps the XML compact and easily extensible, e.g. for adding versionCode,
signingKey, etc as attributes:

    <install packageName="com.fsck.k9"/>
    <install packageName="at.bitfire.davdroid"/>
    <delete packageName="com.facebook.orca"/>

Copyright: 2016 Blue Jay Wireless
Signed-off-by: Hans-Christoph Steiner <hans@eds.org>

closes #177
2016-08-16 21:02:15 +02:00
Hans-Christoph Steiner
5f8beaa2c6 tests for finding the proper OBB files for each APK 2016-07-07 22:34:41 +02:00
Hans-Christoph Steiner
1be263e870 FDroidPopen must have a locale to support UTF-8 filenames
`fdroid update` should be able to handle any valid filename (hopefully
aapt doesn't barf on them).  To handle that, the environment where the
shell commands are run in needs to have a UTF-8 locale set. If LANG is
not set, things default to ASCII and UTF-8 filenames fail.

This also renames test APK with lots of Unicode chars as a test case.

closes #167
2016-06-27 20:30:32 +02:00
Daniel Martí
5de678e48b all: switch to jdk8 as default
Also, remove jdk7 as it will become unused. We added jdk8 for
retrolambda, and now that we will use jdk8 as the default, jdk7 is
unnecessary as retrolambda can work fine with just jdk8.

This removes it from the buildserver, and the new CI image also only has
jdk8 from jessie-backports.

Fixes #185.
2016-06-21 11:47:37 +01:00
Hans-Christoph Steiner
c352605768 add force_build_tools config option
This replaces the current default behavior of always forcing the
build_tools version and allows the user to set build-tools forcing in
config.py.

closes #147
2016-06-21 10:29:56 +02:00
Hans-Christoph Steiner
85b65f0941 tests: pyvenv --system-site-packages is too buggy on python 3.4
It always wants to install packages into /usr/lib/python3.4/site-packages

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 295, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/usr/lib/python3/dist-packages/pip/req.py", line 1436, in install
    requirement.install(install_options, global_options, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/pip/req.py", line 672, in install
    self.move_wheel_files(self.source_dir, root=root)
  File "/usr/lib/python3/dist-packages/pip/req.py", line 902, in move_wheel_files
    pycompile=self.pycompile,
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 214, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 176, in clobber
    os.makedirs(dest)
  File "/usr/lib/python3.4/os.py", line 237, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.4/site-packages'
2016-06-14 22:40:50 +02:00
Hans-Christoph Steiner
1b7a8f85fc parse targetSdkVersion from APKs
The default targetSdkVersion is minSdkVersion, according to the docs:
https://developer.android.com/guide/topics/manifest/uses-sdk-element.html#target

https://gitlab.com/fdroid/fdroidclient/issues/682
2016-06-14 11:43:07 +02:00
Hans-Christoph Steiner
547a57e693 fix "DeprecationWarning: Please use assertEqual instead."
They've been deprecated since python 3.2, which was released a long time
ago.
2016-06-14 10:06:02 +02:00
Hans-Christoph Steiner
2b6d692f06 use UTF8 as default instead of ASCII for .java .gradle pom.xml
.java .gradle and XML files all can use any encoding.  Most code is ASCII,
but authors' names, etc. can easily be non-ASCII.  UTF-8 is by far the most
common file encoding.  While UTF-8 is the default encoding inside the code
in Python 3, it still has to deal with the real world, so the encoding
needs to be explicitly set when reading and writing files. So this switches
fdroidserver to expect UTF-8 instead of ASCII when parsing these files. For
now, this commit means that we only support UTF-8 encoded *.java, pom.xml
or *.gradle files.  Ideally, the code would detect the encoding and use the
actual one, but that's a lot more work, and its something that will not
happen often. We can cross that bridge when we come to it.

One approach, which is taken in the commit when possible, is to keep the
data as `bytes`, in which case the encoding doesn't matter.

This also fixes this crash when parsing gradle and maven files with
non-ASCII chars:

ERROR: test_adapt_gradle (__main__.BuildTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/fdroidserver-eighthave/tests/build.TestCase", line 59, in test_adapt_gradle
    fdroidserver.build.adapt_gradle(testsdir)
  File "/var/lib/jenkins/workspace/fdroidserver-eighthave/fdroidserver/build.py", line 445, in adapt_gradle
    path)
  File "/var/lib/jenkins/workspace/fdroidserver-eighthave/fdroidserver/common.py", line 188, in regsub_file
    text = f.read()
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 9460: ordinal not in range(128)
2016-06-07 20:13:54 +02:00
Lode Hoste
0da94fbf58 Prefer pyvenv instead of virtualenv for Python 3.3+ 2016-04-13 08:30:41 +02:00
Hans-Christoph Steiner
9cd6b444f8 standardize on .yml as the file extension for YAML
Though the YAML people recommend .yaml for the file extension, in Android
land it seems clear that .yml has won out:

* .travis.yml
* .gitlab-ci.yml
* .circle.yml
* Ansible main.yml
2016-03-23 17:16:28 +01:00
Hans-Christoph Steiner
3768d7a4d6 refactor env handling for FDroidPopen to support .fdroid.* metadata
The start up sequence of processes that are based on the .fdroid.* metadata
is a bit different, so this ensures that the environment variables get
properly initialized in all cases.

This also creates a single function where the environment is set.  Before
it was being set in multiple places across multiple files.
2016-03-23 17:16:27 +01:00
Daniel Martí
a51d849a5b update test: replace decode('hex') with unhexlify 2016-03-11 13:27:00 +00:00
Daniel Martí
5c40e3ab99 metadata test: load pickle in bytes, not str 2016-03-11 13:27:00 +00:00
Daniel Martí
24ad0418e3 tests: switch to python3 2016-03-11 13:27:00 +00:00
Daniel Martí
ee9a296b64 Make pre-commit hook pass after python3 switch 2016-03-10 16:43:37 +00:00
Dmitriy Bogdanov
5d54b761af Add FDroidPopen usage test 2016-02-18 00:45:06 +04:00
Hans-Christoph Steiner
3967a0068b added 'mirrors' option to config for giving official mirror URLs
serverwebroot has long supported uploading to multiple servers, this bit of
metadata communicates those official mirrors to the client so that it can
automatically do something useful with that information.

closes #14 https://gitlab.com/fdroid/fdroidserver/issues/14
2016-02-17 10:40:07 +01:00
Hans-Christoph Steiner
a7a053c4f5 use utf-8 as default encoding for config.py
This makes it easy to add all languages to the description.
2016-02-17 00:26:16 +01:00
Hans-Christoph Steiner
2983f3505d support BSD and GNU sed in tests 2016-02-15 23:06:31 +01:00
Hans-Christoph Steiner
c91b715a15 run-tests: use OSX/BSD compatible mktemp -d
-d requires an arg on BSD
2016-02-15 23:06:31 +01:00
Hans-Christoph Steiner
ff30bded21 only test ./gendocs on GNU/Linux
This helps with getting the tests running on OSX and other platforms.
2016-02-15 23:06:31 +01:00
Hans-Christoph Steiner
5f1e57c7e6 complete-ci-tests: exit with error if setup is not correct 2016-02-15 23:06:31 +01:00
Hans-Christoph Steiner
751fd3fb0a common: do not crash if the java_paths are not what is expected
Many commands work without the JDK installed, and it is also possible that
someone is using only JDK 8 or 9.
2016-02-12 08:33:21 +01:00
Hans-Christoph Steiner
69c81c3817 use jarsigner and keytool from same JDK as is being set in JAVA7_HOME
Using the same JDK throughout should prevent weird bugs where a setup might
use Java8's jarsigner and Java7's keytool.  This also allows the user to
set java_paths and have jarsigner and keytool used from that specified JDK.

This incorporates almost all of the patch that is in the Debian package
that forces fdroidserver to use the default JDK on that Debian release.

closes #93 https://gitlab.com/fdroid/fdroidserver/issues/93
2016-02-11 21:17:23 +01:00
Hans-Christoph Steiner
25caa09420 run-tests: verbose output for fdroid update to aid debugging
I'm stumped by this stacktrace, hopefully debug logging will help:

Unknown exception found!
Traceback (most recent call last):
  File "./fdroidserver-0.4.0/fdroid", line 141, in <module>
    main()
  File "./fdroidserver-0.4.0/fdroid", line 119, in main
    mod.main()
  File "<http://localhost:18080/job/fdroidserver/ws/.testfiles/tmp.9hr9NiSmZs/fdroidserver-0.4.0/fdroidserver/update.py",> line 1146, in main
    apks, cachechanged = scan_apks(apps, apkcache, repodirs[0], knownapks)
  File "<http://localhost:18080/job/fdroidserver/ws/.testfiles/tmp.9hr9NiSmZs/fdroidserver-0.4.0/fdroidserver/update.py",> line 543, in scan_apks
    thisinfo['sig'] = getsig(os.path.join(os.getcwd(), apkfile))
  File "<http://localhost:18080/job/fdroidserver/ws/.testfiles/tmp.9hr9NiSmZs/fdroidserver-0.4.0/fdroidserver/update.py",> line 361, in getsig
    with zipfile.ZipFile(apkpath, 'r') as apk:
  File "/usr/lib/python2.7/zipfile.py", line 770, in __init__
    self._RealGetContents()
  File "/usr/lib/python2.7/zipfile.py", line 857, in _RealGetContents
    x._decodeExtra()
  File "/usr/lib/python2.7/zipfile.py", line 388, in _decodeExtra
    tp, ln = unpack('<HH', extra[:4])
struct.error: unpack requires a string argument of length 4
2016-01-21 12:26:50 +01:00
Dominik George
7f3434ea50
Add Author Name and Author Email fields. (Closes: #90) 2016-01-03 23:00:03 +01:00
Daniel Martí
3101ad6c39 Set all list defaults to an actual list
Iterating over the defaults will crash otherwise. Update pickle files.
2015-12-04 00:08:42 +01:00
Daniel Martí
088929711c Represent multiline fields as str, not list
Only keep lists in metadata files in the json format, since they don't
support multiline strings that are readable.

This makes the internal code easier, and a bit faster.
2015-12-03 11:36:15 +01:00
Daniel Martí
c975531d87 metadata: properly store nums as strs and bools as bools 2015-12-02 20:49:42 +01:00
Daniel Martí
bf8518ee8f Rework build into a class
This simplifies usage, goes from

    build['flag']
to
    build.flag

Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.

As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.

While at it, unify "build", "thisbuild", "info", "thisinfo", etc into
just "build".
2015-12-02 20:03:11 +01:00
Daniel Martí
ab614ab442 Rework app into a class
This simplifies usage, goes from

	app['Foo']
to
	app.Foo

Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.

As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.
2015-11-28 17:11:05 +01:00
Daniel Martí
de12cfdbe1 Handle duplicate apps in a much cleaner way
Don't log and exit in an inner metadata function. Handle it at a higher
level and do a proper exception. This also avoids unnecessary passing of
apps all around.
2015-11-28 14:04:21 +01:00
Daniel Martí
20470556f7 run-tests: Don't set -x until option parsing is done 2015-11-01 14:09:11 +01:00
Daniel Martí
c115f71640 run-tests: Don't show echo_header commands 2015-11-01 14:09:06 +01:00
Daniel Martí
27c717e764 tests: allow running ./tests/run-tests 2015-11-01 13:48:25 +01:00
Daniel Martí
16ca7f3be3 tests: we now replace "buildToolsVersion var" too
See d706cfa7d8
2015-10-24 18:47:43 +02:00
Daniel Martí
c877d4524a tests: Comment values are lists, not strings
Since there can be multiple lines.
2015-10-08 16:39:47 +02:00
Daniel Martí
5d180b8858 tests: adapt to new comments dict 2015-10-08 13:20:48 +02:00
Daniel Martí
00bd75aa8c tests: close files 2015-10-08 13:20:35 +02:00
Daniel Martí
e285a8ac31 Remove Dogecoin donation support
Not nearly as popular as bitcoin and litecoin. To the point where it hadn't
been used yet.

Bump index version to 14 to mark its removal.
2015-09-18 19:06:10 -07:00
Hans-Christoph Steiner
707930ce0a add a basic test of fdroid import 2015-09-10 11:08:40 +02:00
Hans-Christoph Steiner
e6c0be8898 move tests into common script for jenkins and gitlab-ci 2015-09-01 19:52:37 +02:00
Hans-Christoph Steiner
3fc2a99d71 fdroid --version for installed releases and running from git
This will report the version embedded in the module if it is installed, and
will report `git describe` if being run from git.  If someone installs from
git using pip, this will probably report the version in setup.py, which
will be wrong.  But that is not a documented install method, and I haven't
heard of anyone using it.  The recommended way is to run straight from git.
2015-09-01 19:45:00 +02:00
Hans-Christoph Steiner
325db90918 set "accepted formats" for metadata in config.py
For a bit repo like f-droid.org, it makes sense to standardize on a single
format for metadata files.  This adds support for enforcing a single data
format, or a reduced set of data formats.  So f-droid.org would run like
this if it changed to YAML:

  accepted_formats = ['txt', 'yaml']

Then once everything was converted to YAML, it could look like this:

  accepted_formats = ['yaml']
2015-09-01 11:53:37 +02:00
Hans-Christoph Steiner
beb217b6ae add 'gradleprops' to .pickle files for the metadata tests 2015-09-01 11:53:37 +02:00
Hans-Christoph Steiner
8d6e0aebb9 exit with error if duplicate metadata file is found
In order to prevent confusion caused by multiple metadata files for a given
app, fdroid will exit with an error if it finds any app metadata file with
the same package ID as one that has already been parsed.
2015-09-01 11:39:51 +02:00
Hans-Christoph Steiner
32e257d742 support app metadata in YAML format
YAML is a format that is quite similar to the .txt format, but is a
widespread standard that has editing modes in popular editors.  It is also
easily parsable in python.

The .pickle for testing is a lightly edited version of the real metadata
for org.videolan.vlc:

 * comments were removed
2015-09-01 11:39:51 +02:00
Hans-Christoph Steiner
0425788b7d consolidate all post-parse metadata checks
This puts all of the needed post parsing checks on the metadata into a
single method that is used by all parsing methods (.txt, JSON, XML, YAML).
This provides the single place to normalize the internal representation of
the metadata.

It would be good to also change the internal representation to use more Python
bool/int types so that less post parsing is needed for JSON, XML, and YAML.

The SMSSecure test .pickle was changed to account for the use of lstrip()
and rstrip() on all 'script' types.

This also changes the example JSON to use ints for versionCodes
2015-09-01 11:39:51 +02:00
Hans-Christoph Steiner
ab145de6bc support app metadata in XML format
While the current text metadata format is good for human readability and
editability, it is difficult to produce and parse using code.  XML is a
widespread standard format for easy automatic parsing and creating, while
having decent human readability.

The .pickle for testing is a lightly edited version of the real metadata
for net.osmand.plus:

* comments were removed
* "NonFreeNet" was added as an AntiFeature
2015-09-01 11:39:51 +02:00
Hans-Christoph Steiner
2831b3e93f convert internal representation of AntiFeatures to list
The AntiFeatures metadata is a comma-separated list of tags, like
Categories, so it should also be stored internally as a list.  This makes
parsing XML and JSON easier.

The test cases' .pickle files look like they change a lot, but they really
don't, its only the change of default AntiFeatures value from None to []
2015-09-01 11:39:51 +02:00
Hans-Christoph Steiner
dabb7e42b7 add test case for JSON parsing, based on parsed .txt metadata
The .pickle was created by dumping the output from parsing the current .txt
metadata for org.adaway.  The JSON started from that pickle dump, but was
then hand edited to be more proper JSON, e.g. using boolean values.
2015-09-01 11:39:51 +02:00
Hans-Christoph Steiner
1bbae4cd9e add test for metadata parsing based on a parse output to a pickle
This is a test to cover future modifications of the .txt metadata parsing.
The pickle file was generated by just dumping the current parsed metadata,
so this test will always succeed if the parsing is not changed.
2015-09-01 11:39:50 +02:00
Hans-Christoph Steiner
9e5dd19fc8 add test files to pre-commit hook and fix pep8 errors 2015-09-01 11:39:50 +02:00
Hans-Christoph Steiner
d53a5af715 build: improve regsub pattern for setting buildToolsVersion
This addresses the discussion in !64
https://gitlab.com/fdroid/fdroidserver/merge_requests/64

Sometimes, buildToolsVersion is a kind of gradle macro call, and other
times it is a variable assignment.  This regsub pattern now handles both of
those cases.
2015-08-19 17:55:17 +02:00
Hans-Christoph Steiner
6db6433e97 fix syntax error in manual generation and add ./gendocs.sh to the tests 2015-08-19 17:51:29 +02:00
Daniel Martí
90a926dfb0 Fix test that broke with c813a3cdbb 2015-08-13 12:44:35 -07:00
Hans-Christoph Steiner
1bb14d2f82 run-tests: fix quoting to prevent spurious execution of fdroid update 2015-08-05 13:36:21 +02:00
Hans-Christoph Steiner
64a9c93ce7 test new common.regsub_file() method that replaces sed calls 2015-08-01 00:38:41 +02:00
Alex Burka
4a478528c2 tests: short args for mktemp to support BSD
*BSD and OSX do not have compatible long args
2015-08-01 00:38:40 +02:00
Hans-Christoph Steiner
b8a3bce430 run-tests: fix copying of urzip.apk into new repo 2015-08-01 00:38:40 +02:00
Hans-Christoph Steiner
d01e814e36 run-tests: run fdroid readmeta after each --create-metadata
Just another basic check, this time for `fdroid readmeta`.
2015-07-30 22:19:10 +02:00
Hans-Christoph Steiner
ff48c6b7f9 tests: no need to dump the index.xml to the build log
This should make the build log a lot shorter.
2015-05-08 16:13:15 -04:00
Hans-Christoph Steiner
de1c80f9b4 fdroid update --create-key to create a keystore/key
This provides the final option in this series, allowing the user to just
add --create-key to `fdroid update, and thereby upgrade an unsigned repo to
a proper signed repo.  It also might be useful

closes #13 https://gitlab.com/fdroid/fdroidserver/issues/13
2015-05-08 16:13:15 -04:00
Hans-Christoph Steiner
86865faa62 make fdroid update check that it can sign the repo, or exit with error
There is no good reason to run unsigned repos any more.  It is trivially
easy to create and use a signed repo, and having to support unsigned repos
in the client makes some security-critical parts of the code a lot more
complicated.

refs #13 https://gitlab.com/fdroid/fdroidserver/issues/13
https://gitlab.com/fdroid/fdroidclient/issues/12
2015-05-08 16:13:15 -04:00
Ciaran Gultnieks
384f97e998 Ensure package names are valid 2015-01-26 18:30:01 +00:00
Hans-Christoph Steiner
2a87ae5127 fix update.TestCase: remove unused SilentPopen import, that class was removed 2015-01-21 10:21:54 +01:00
Hans-Christoph Steiner
3b3d026ee3 fix install test by making sure common.options is set 2015-01-21 10:21:54 +01:00
Hans-Christoph Steiner
fa1cc48d57 run all SDK tools commands using SdkToolsPopen 2014-12-14 13:25:20 +01:00
Hans-Christoph Steiner
9244256461 find cmds from SDK build-tools in a more flexible way, on the fly
This is a more flexible approach than testing for the complete SDK and
build-tools up front.  This will only test for the commands that are
actually being run, so that if you only have `aapt` installed, you can do
`fdroid update` without errors, but other commands will still give
appropriate errors.

This also makes the build_tools item in config.py optional, it is only
needed if you want to force a specific version of the build-tools.
2014-12-14 13:25:20 +01:00
Hans-Christoph Steiner
298a88a498 added test case for common.isApkDebuggable()
Just getting into the habit of adding tests to everything that I change...
Also, it should be useful to have an unsigned APK in the test collection,
since `fdroid update` should handle it gracefully and give a warning of
some kind.
2014-12-14 13:25:20 +01:00
Hans-Christoph Steiner
14cd835929 run-tests: properly handle zero and multiple args
The test for the help flag threw an error if there were 0 args, or if arg 1
was set to a space-separated list.  The -z tests would fail if the arg was
set to a space-separated list.
2014-11-17 17:40:48 +01:00
Hans-Christoph Steiner
49b343d024 run-tests: make sure not to include the bad APKs in tests/
These APKs are meant for specific tests, not the tests that want just a
collection of good APKs.
2014-11-17 17:24:02 +01:00
Hans-Christoph Steiner
18db382c47 include test cases for python getsig replacement
This includes the old getsig.java since that is the canonical implementation
of that algorithm.

fixes #5 https://gitlab.com/fdroid/fdroidserver/issues/5
2014-11-07 09:20:33 +00:00
Daniel Martí
6e2d0a9e1e Replace getsig.java with a pure python implementation
Special thanks to deki for helping out with the certificate encodings:
https://gitlab.com/snippets/1842

fixes #5 https://gitlab.com/fdroid/fdroidserver/issues/5
2014-11-07 09:20:14 +00:00
Daniel Martí
c99f15ac86 Remove .egg-info dir when tests succeed 2014-09-17 08:54:02 +02:00
Hans-Christoph Steiner
f34c842f55 auto-clean newlines and spaces in repo/archive descriptions
This gives us flexibility in how the blocks of text can be formatted in
config.py, but also provides a more useful format for displaying since the
client can decide where to wrap the text.
2014-07-14 15:01:57 -04:00
Daniel Martí
8b469a79f8 Fix the test suite 2014-07-03 22:33:40 +02:00
Hans-Christoph Steiner
a58a9bff18 run pre-commit hook as part of test suite 2014-06-30 23:01:48 -04:00
Hans-Christoph Steiner
4d913d646a replace redundant build_tools check that breaks fdroid init
This reverts b637568a62 since it added a
redundant check that broke `fdroid init` when the default version dir of
build_tools does not exist on the local system.  It then uses the function
that was already in place for checking the build_tools setup in a way that
does not break `fdroid init`.

Now that the fake android home version is not matching the default version,
the tests will catch this bug in the future.
2014-06-30 23:01:47 -04:00
Hans-Christoph Steiner
7a3992aa1a use 'python2' everywhere since fdroidserver has not been tested with 3.x 2014-06-30 23:01:47 -04:00
Hans-Christoph Steiner
40d4e30010 tests: create_fake_android_home should create old build-tools version
This is testing the build-tools version auto-detect in `fdroid init`, so it
should be kept as an older version.  This is not meant to test the current
version of the build tools.
2014-06-30 21:19:55 -04:00
Daniel Martí
cbdcdd1e78 Update build-tools to 20.0.0 2014-06-30 17:03:16 +02:00
Hans-Christoph Steiner
25a94dced2 tests: create a source tarball and use that to build a repo
This tests that setup.py is in working order and creating a functional
source tarball.
2014-06-27 20:31:28 -04:00
Hans-Christoph Steiner
0e47d62024 tests: by default, run on included urzip.apk
This means you can just do `cd tests/ && ./run-tests` to run the tests now.
You can still override the APK source with the first argument, like:

cd tests/ && ./run-tests /path/to/lots/of/apks/dir
2014-06-27 20:31:28 -04:00
Hans-Christoph Steiner
25f6b0c246 server: --sync-from-local-copy-dir for updating from offline signing repo
To support a fully offline build/signing machine, there is the "local copy
dir".  The repo is generated on the offline machine and then copied to a
local dir where a thumb drive or SD Card is mounted.  Then on the online
machine, using `fdroid server update --sync-from-local-copy-dir` allows
the whole server update process to happen in a single command:

0. read config.py on online machine's repo
1. rsync from the local_copy_dir to the current dir
2. copy to serverwebroot, awsbucket, etc.
2014-06-26 20:47:36 -04:00
Hans-Christoph Steiner
d73b43c5fc server: 'local_copy_dir' config/options to automate offline repo signing
This allows a dir to be specified in config.py that `fdroid server update`
will automatically rsync the repo to.  The idea is that the path would
point to an SD card on a fully offline machine that serves as the secure
repo signing machine.
2014-06-26 20:47:36 -04:00
Hans-Christoph Steiner
35ea01cbd6 run-tests: find current version of aapt in folder rather than in PATH
Not everyone adds the build-tools to their PATH, so this makes it so this
script will find aapt in the most recent build-tools version that is
installed on the local system.
2014-06-16 19:18:28 -04:00
Daniel Martí
1fcf81b1f4 Current build-tools is 19.1.0, not 19.1 2014-06-16 23:12:47 +02:00
Daniel Martí
1ca4bda4ef Fix create_fake_android_home with the new build-tools 2014-06-16 13:09:28 +02:00
Hans-Christoph Steiner
97334286ea tests: turn off unneeded debug logging during APK copying
The whole process of finding and copying APKs can be very verbose, so turn
of the bash verbose logging during that process.
2014-06-05 16:55:52 -04:00
Hans-Christoph Steiner
0272124248 tests: fix --android-home test to actually work
In this case, ANDROID_HOME is set to a fake, non-working version that will
be detected by fdroid as an Android SDK install.  It should use the path
set by --android-home over the one in ANDROID_HOME, therefore if it uses
the one in ANDROID_HOME, it won't work because it is a fake one.  Only
--android-home provides a working one.
2014-06-04 22:01:25 -04:00
Hans-Christoph Steiner
9b313e76bb tests: specify dir for APKs as cmd line arg, and set jenkins to ~
This lets people easily set whatever dir they want, while letting jenkins
search through its whole workspace for any APKs that have been built. Also,
only include the latest version of a given packageName+versionCode.
2014-06-04 22:01:25 -04:00
Hans-Christoph Steiner
78d5ebf26c use 'metadata' name throughout: --createmeta to --create-metadata
To keep the naming used within fdroid consistent, I renamed this long flag
to use the whole word 'metadata' since that is what is used everywhere else
2014-06-04 21:54:56 -04:00
Hans-Christoph Steiner
0e2f4d54d0 tests: try fdroid update after adding an APK to an existing repo
Yes, this includes a binary file, but it is only for the tests, and it is
free software since I wrote it.  The source is here:

https://github.com/eighthave/urzip
2014-06-04 21:54:55 -04:00
Hans-Christoph Steiner
ed7a0576c2 test: make sure fdroid update -c creates a full index.xml
Previously, `fdroid update -c` would only create the new metadata, but
would not add the new apps/apks to the repo.  That required a second run of
`fdroid update`.  This has been fixes, so this test makes sure it stays
fixed, in a very generic way.
2014-06-04 21:54:55 -04:00
Hans-Christoph Steiner
6ca060e10d add more SDK checks: build-tools/19.0.3 and presense of aapt
Make sure that fdroid can find aapt in the current config, otherwise exit
with an error.  Some users don't have build_tools set, and their SDK does
not include the build-tools in the default versioned dir, so this should
warn them of what is wrong.
2014-06-04 21:54:55 -04:00
Hans-Christoph Steiner
ef7c9d89d2 init: --no-prompt to skip sdk_path prompt
For running the tests and in other scripted setups, the user prompt is an
annoying. Using --no-prompt means the script can test for failure.
2014-04-23 20:10:02 -04:00
Hans-Christoph Steiner
66df02d5f8 init: --android-home for forcing the path to the Android SDK
This allows the user to set the path to their Android SDK from the command
line. This option is named after the standard env var ANDROID_HOME, as used
in the build.xml generated by `android update project`.  --android-home
takes precendence over the ANDROID_HOME env var if it is set.
2014-04-23 19:33:10 -04:00
Hans-Christoph Steiner
49387b7108 use FDroid-standard naming when copying APKs for tests
This makes sure that there are no APKs with duplicate version codes.
2014-04-10 12:37:07 -04:00
Hans-Christoph Steiner
234736f306 move run-tests to /bin/bash so we can use bashisms 2014-04-10 12:31:23 -04:00
Hans-Christoph Steiner
3829d37d34 support repo signing with a key on a smartcard
This assumes that the smartcard is already setup with a signing key.  init
does not generate a key on the smartcard, and skips genkey() if things are
configured to use a smartcard.

This also does not touch APK signing because that is a much more elaborate
question, since each app is signed by its own key.
2014-04-07 16:00:18 -04:00
Hans-Christoph Steiner
faf0c4381f add test case were init generates a keystore and uses it 2014-04-07 16:00:18 -04:00
Hans-Christoph Steiner
bfa21fb630 add script to do a test run of creating a new repo
This tests/ folder can then be used for all sorts of tests, including
standard python tests.
2014-04-02 17:49:07 -04:00