pickle can serialize executable code, while JSON is only ever pure data.
The APK cache is only ever pure data, so no need for the security risks of
pickle. For example, if some malicious thing gets write access on the
`fdroid update` machine, it can write out a custom tmp/apkcache which would
then be executed. That is not possible with JSON.
This does just ignore any existing cache and rebuilds from scratch. That is
so we don't need to maintain pickle anywhere, and to ensure there are no
glitches from a conversion from pickle to JSON.
closes#163
Debian's autopkgtest defaults to considering any output on stderr as a sign
that the tests failed. it is simple to make this achieve that standard, so
it seems worth it for now at least.
Generating a keystore is quite slow since it means a new RSA key is created.
That only needs to happen in the tests that check that it actually happened,
otherwise the test can just reuse the stored test keystore.
closes#432
This works around the gradle 2.x bug where versions newer than 2.9 cannot
run things. This also then specifies the version of Gradle Android Plugin
that is included in Debian/stretch.
This fixes all the bugs I could find that prevented fdroid from
handling files with spaces in them. This is more important now that
fdroid supports random media files, and Repomaker
This filename has some messed up bytes related to bi-directional script
that is included (Left-to-Right and Right-to-Left). GNU/Linux always
interprets filenames as pure byte sequences. Windows and OSX store
filenames as Unicode strings. So on OSX, the invalid filename gets
converted to a valid name. That works fine, but the test fails because it
is compared to a file generated on Ubuntu, where it preserves the byte
sequence.
This includes an APK with a valid Unicode filename that includes
bi-directional script.
apksigner v0.7 (build-tools 26.0.1), Google made it require that the
AndroidManifest.xml was present in the archive before it verifies the
signature. So this needs to stick with the jarsigner hack for JARs.
It turns out it is error prone to `git push` to a non-bare git repo. For
the offline signing machine, the git remote needs to be a regular git repo
in a directory on a thumbdrive so that once the thumbdrive is plugged into
an online machine, that git repo can be transferred to the online machine.
Since the mirror URLs are per repo section (repo/archive), the mirror URLs
must include the repodir at the end. This was missing for servergitmirrors
found by @cde when working on fdroidclient#35
The new policy is to move APKs with invalid signatures to the archive,
and only add those APKs to the archive's index if they have valid MD5
signatures.
closes#323closes#292
The original logic was checking keepversions against the len() of ALL the
APKs in the repo/archive. The correct thing is to check against the
number of APKs available for the given packageName/appid.
closes#166
Really, it is the fdroidclient parser of index.xml that fails, due to the
hardcoded expectation that there will only ever be a single APK for any
given versionCode. We keep index.xml backwards compatible for old
clients, and use index-v1.json to support new things. Having multiple
APKs that have the same packageName and versionCode will break the client
v0.103.* since that version uses index-v1.json, but still has the hard-
coded database parsing stuff.
#153
uses the standard package.name_123.apk. If that exists, it appends the
shasum. If that exists, then its a duplicate, so its deleted. This should
help @SergeWinters with his 12,000 APKs.
There are many APKs out in the wild that claim to be the same app and
version and each other, but they are signed by different keys. fdroid
should be able to index these, and work with them. This supports having
the developer's signature via reproducible builds, random collections of
APKs like repomaker, etc.
This option was not hooked up at all, and does not make sense as a command
line argument. It should just be a config.py item. In that case, the
presence of config.py marks the current dir as a repo, so there is no
longer a need to test for a dir called repo/ as a safety. This makes the
setup easier, since sync_from_localcopy() now creates repo/ for the user.
Since `fdroid server update` is the place where all uploads to servers
happens, it makes sense to also handle the git push for the binary
transparency log here instead of `fdroid btlog`
This is a bit different than index.jar: instead of their being index.xml
and index_unsigned.jar, the presense of index-v1.json means that there is
unsigned data. That file is then stuck into a jar and signed by the
signing process. index-v1.json is never published to the repo. It is
included in the binary transparency log, if that is enabled.
This just makes it easier for people writing build recipes. Rewriting will
output a list of strings as well.
The test index.xml and categories.txt are updated to include the new number
categories, and the changed CurrentVersionCode to 2147483647 (MAX_VALUE)
Python encode/decode libs work directly with dicts, so the internal dict
can just be passed directly to any of these libs (pyyaml, pyjson, msgpack,
simplejson, etc). This still generates the exact same index.xml as before.
This converts the internal format for the repo timestamp to a datetime
instance, which can be easily converted to UNIX time in seconds for XML
and UNIX time in milliseconds for the new index formats. UNIX time in
milliseconds is directly serialized into a java.util.Date instance by
Jackson.
If a group of items are enclosed in {}, then that will be a Python set,
which does not preserve order. To preserve order, the data must be either
a tuple () or list [].
Since https://gitlab.com/fdroid/ci-test-app is a separate git repo, things
with incompatible changes could get out of sync. Therefore, this test
should specify which git commit is runs against.
For example, the .fdroid.yml file is still a moving target. Just now, the
keys had the spaces removed as part of this MR.
