ad2b9b99c2 put this in the wrong place, it
was running it on the buildserver host rather than in the buildserver VM
itself, where the builds actually run.
refs #148
This is to clean up the code for the introduction of a new index format.
This also makes it so that only repo/categories.txt is generated, and not
archive/categories.txt.
Currently, the only thing that I can find that reads categories.txt is the
wordpress plugin, e.g. wp-fdroid. And it only reads repo/categories.txt
not archive/categories.txt.
* New command `dscanner`, enables one to scan signed APKs with Drozer
* Drozer is a dynamic vulnerability scanner for Android
* Drozer runs in a emulator or on-device, this new `dscanner` command...
* starts a docker image with Drozer and the Android Emulator pre-installed,
* loads the signed APK into the emulator
* activates Drozer automated tests for the APK
* gathers the report output and places it next to the original APK
* The Drozer docker image can be:
* cached locally for re-use (just don't run --clean*)
* retrieved from dockerhub.com for more efficient runtime
* or be built from scratch (in the new "./docker" directory)
* New "Vulnerability Scanning" documentation section (run gendocs.sh)
JSON and YAML are very closely related, so supporting both of them is
basically almost no extra work. Both are also closely related to how
Python works with dicts and pickles. XML is a very different beast, and its
not popular for this kind of thing anyway, so just purge it.
This scans all APKs for old versions of OpenSSL libraries that are known to
be vulnerable to issues, or fully unsupported.
This really should be implemented as a per-APK AntiFeature, so that it can
apply to any version that is vulnerable. Since AntiFeatures are currently
only per-App, this instead sets the AntiFeature only if the latest APK is
vulnerable.
Google also enforces this:
https://support.google.com/faqs/answer/6376725?hl=en
apk['antiFeatures'] has the first letter small, since all build fields
start with a lowercase letter. app.AntiFeatures has the first
uppercase since all App fields are that way.
Since `fdroid update` parses the output of aapt, and since aapt's output
format changes in non-compatible ways, test to make sure that the version
of aapt is new enough to prevent mystery stacktraces. This only prints a
warning since in many cases, the old version will work just fine.
When a git repo has a .fdroid.yml file in it, and `fdroid build` is run
directly in that git repo, then this uses the file path as the remote for
the git repo in build/appid that is actually built. That makes it possible
to run builds of commits that are only local, and makes things a whole lot
faster.
This allows a source repo to include a complete metadata file so that it
can be built directly in place using `fdroid build`. If that app is then
included in fdroiddata, it will first load the source repo type and URL
from fdroiddata, then read .fdroid.yml if it exists, then include the rest
of the metadata as specified in fdroiddata, so that fdroiddata has
precedence over the metadata in the source code.
This lets `fdroid build` apps without having a whole fdroiddata setup, but
instead just directly in place in the source code. This also lets devs
optionallu maintain the fdroid metadata as part of their app, rather than
in fdroiddata without loosing any control. This should make it easier to
spread around the maintenance load.
Something like `gradle: yes` in YAML will be parsed as a boolean, since
'yes' is officially defined as a boolean true in YAML. For metadata fields
that need to be lists, this needs to be converted. Same goes for a single
string like `gradle: customFlavor`.
This includes more info to help track down problems with reproducible
builds, like the specific version being built, and which exact versions of
the Android SDK and NDK were used.
Any variation in the Android tools used to build an APK can cause the build
to be unreproducible. To help troubleshoot these times, this posts the
installed versions of the Android SDK and NDK components to the lastbuild
log, for the long term record.
refs #148
This makes sure there is a GPG signature on any file that is included in
the repo, including APKs, OBB, source tarballs, media files, OTA update
ZIPs, etc. Having a GPG signature is more important on non-APK files since
they mostly do not have any signature mechanism of their own.
This also adds basic tests of adding non-APK/OBB files to a repo with
`fdroid update`.
closes#232
This makes it so that the final build product can be specified in output=
and it'll work no matter if its an APK or not. This was developed around
the case of building the OTA update.zip for the Privileged Extension. It
should work for any build process in theory but it has not yet been tested.
https://gitlab.com/fdroid/privileged-extension/issues/9
The versionName is defined as a string or string resource that can be any
arbitrary data. fdroid should not second guess the developer here, and
should just use the versionName unmodified. For anything that needs to
compare different versions of apps, versionCode should always be used since
that's what Android uses.
https://developer.android.com/guide/topics/manifest/manifest-element.html#vname
Support media files
This is the first step in supporting adding artibitrary files to fdroid repos, targeted add supporting media files and flashable _update.zip_ files like used to install the Privileged Extension. This reuses the existing metadata fields to keep compatibilty with older versions, but that means that lots of names are confusingly named since they refer to aspects of an APK rather than a generic file. We can address that later when we refactor the whole index metadata.
See merge request !177
This adds the most basic level of support for including arbitrary files in
an F-Droid repository. This is useful for things like including videos,
ebooks, update.zip files for ROM updates, and more. The aim is to have
this as generic as possible to keep it flexible for unforeseen uses.
Code-wise, this is really just a first effort. This area of code has not
been touched in a very long time, and the repo parsing is done in a giant
function that is not easy to break apart. It should be broken up to more
cleanly support arbitrary files.
Also remove the TODO line, we've decided to keep the old permission
format for now, at least until there is a major overhaul of the index
data format. And the issue tracker the proper place for TODOs.
Reproducible via `fdroid checkupdates --auto subreddit.android.appstore`
at fdroiddata HEAD (e76449ab).
WARNING: ...subreddit.android.appstore : Couldn't find package ID
CRITICAL: Unknown exception found!
Traceback (most recent call last):
File "/home/mvdan/.bin/fdroid", line 147, in <module>
main()
File "/home/mvdan/.bin/fdroid", line 124, in main
mod.main()
File "/home/mvdan/git/fsr/fdroidserver/checkupdates.py", line 571, in main
checkupdates_app(app)
File "/home/mvdan/git/fsr/fdroidserver/checkupdates.py", line 469, in checkupdates_app
if int(build.vercode) >= int(app.CurrentVersionCode):
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
In many cases, there are times where metadata errors need to be ignored, or
at least not stop the command from running. For example, there will
inevitably be new metadata fields added, in which case a packaged version
of fdroidserver will throw errors on each one. This adds a standard -W
flag to customize the response: ignore, default, or error.
* by default, the errors are still errors
* `fdroid readmeta -W` will just print errors
* `fdroid readmeta -Wignore` will not even print errors
https://gitlab.com/fdroid/fdroidserver/issues/150
urllib.parse.urljoin() will strip off the last path segment before joining
if that last path segment does not end with /. That's a "feature" to make
it easy to replace file names. Here it was stripping off the essential
'fdroid' segment, making URLs like:
https://foo.com/repo
when they should be
https://foo.com/fdroid/repo
It is now possible for the server operator to specify lists of apps that
must be installed or deleted on the client (aka "push installs). If
the user has opted in, or the device is already setup to respond to
these requests, then fdroidclient will automatically install/delete
the packageNames listed. This is protected by the same signing key
as the app index metadata.
It generates single XML elements with the data set in the attributes. This
keeps the XML compact and easily extensible, e.g. for adding versionCode,
signingKey, etc as attributes:
<install packageName="com.fsck.k9"/>
<install packageName="at.bitfire.davdroid"/>
<delete packageName="com.facebook.orca"/>
Copyright: 2016 Blue Jay Wireless
Signed-off-by: Hans-Christoph Steiner <hans@eds.org>
closes#177
Parse maxSdkVersion and -23 permission tags
Parse maxSdkVersion and uses-permission-sdk-23 tags and insert them into the index.
AndroidManifest:
```xml
<uses-permission android:name="android.permission.READ_CALENDAR" />
<uses-permission
android:name="android.permission.WRITE_EXTERNAL_STORAGE"
android:maxSdkVersion="18" />
<uses-permission-sdk-23 android:name="android.permission.ACCESS_FINE_LOCATION" />
```
new index format:
```xml
<package>
<version>1.0</version>
<versioncode>1</versioncode>
<apkname>maxmin.apk</apkname>
<hash type="sha256">526eb6d643050c3fae42fb6c001c704006046db52e98998b21f7646ecae3dae5
</hash>
<sig>893fc1f22301c902d05c938b29d21648</sig>
<size>8681</size>
<sdkver>14</sdkver>
<targetSdkVersion>24</targetSdkVersion>
<added>2016-08-01</added>
<permissions>READ_EXTERNAL_STORAGE,READ_CALENDAR,WRITE_EXTERNAL_STORAGE</permissions>
<uses-permission name="android.permission.WRITE_EXTERNAL_STORAGE"
maxSdkVersion="18" />
<uses-permission name="android.permission.READ_CALENDAR" />
<uses-permission name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission-sdk-23 name="android.permission.ACCESS_FINE_LOCATION" />
</package>
```
old format:
```xml
<permissions>READ_CALENDAR,ACCESS_FINE_LOCATION,WRITE_EXTERNAL_STORAGE</permissions>
```
This generates both formats for backward compatibility with old clients.
Also increase version of index format.
This is a proposal for the issues discussed in https://gitlab.com/fdroid/fdroidclient/issues/704
@eighthave @mvdan
See merge request !150
Parse maxSdkVersion and uses-permission-sdk-23
tags and insert them into the index, e.g.,
<package>
<permissions>READ_EXTERNAL_STORAGE,READ_CALENDAR,WRITE_EXTERNAL_STORAGE</permissions>
<uses-permission name="android.permission.WRITE_EXTERNAL_STORAGE"
maxSdkVersion="18" />
<uses-permission name="android.permission.READ_CALENDAR" />
<uses-permission name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission-sdk-23 name="android.permission.ACCESS_FINE_LOCATION" />
</package>
Also increase version of index format
Empty JDK directories can remain from previous JDK installations. For
example in RHEL/Fedora when RPM upgrades a package it can leave modified
files (usually configs) and, consequently, their directories. So we could
end up selecting a bad JDK path.
This reverts commit 82d09560c6.
It doesn't work - the setup scripts are expecting a ".bin" file (which
is apparently a 7z archive), but what's actually got is a ".zip".
Conflicts:
buildserver/provision-android-ndk
Currently, if buildjni is not used but ndk is set to an invalid value,
the build would start but $ANDROID_NDK would be empty. This is happening
in VLC, which results in very confusing errors.
If a build uses a ndk= value that is not set up, such as r11/r12 which
we do not have yet, it should error with "NDK version could not be
found". It does with this change.
Pass a date from APK to KnownApks.recordapk()
... if --use-date-from-apks option is used.
Essentially, it just expands influence of `--use-date-from-apks` option to `stats/known_apks.txt`.
See merge request !141
OBB files provide a mechanism to deliver large data blobs that do not need
to be part of the APK. For example, a game's assets do not need to change
often, so they can be shipped as an OBB, then APK updates do not need to
include all those assets for each update.
https://developer.android.com/google/play/expansion-files.html
Note that the apt packages are split into two halves, because it takes
too long (on 64 bit!) to install them all. The sensible fix would be
to simply up the timeout on the package installation section, but this
is completely broken in chef.
The ZIP format has no official encoding :-| so we have to do hacks. The
zipfile devs couldn't even sort this out:
https://bugs.python.org/issue10614closes#167
`fdroid update` should be able to handle any valid filename (hopefully
aapt doesn't barf on them). To handle that, the environment where the
shell commands are run in needs to have a UTF-8 locale set. If LANG is
not set, things default to ASCII and UTF-8 filenames fail.
This also renames test APK with lots of Unicode chars as a test case.
closes#167
This will make `vagrant ssh` and `fdroid build --server` be the same env,
so troubleshooting should be easier. !135 Here's what `man bash` says:
When bash is invoked as an interactive login shell, or as a
non-interactive shell with the --login option, it first reads and
executes commands from the file /etc/profile, if that file
exists. After reading that file, it looks for ~/.bash_profile,
~/.bash_login, and ~/.profile, in that order, and reads and
executes commands from the first one that exists and is readable.
The --noprofile option may be used when the shell is started to
inhibit this behavior. When a login shell exits, bash reads and
executes commands from the file ~/.bash_logout, if it exists.
This stops these errors:
fdroid/fdroidserver/fdroidserver/update.py:744: ResourceWarning: unclosed
file <_io.BufferedReader
name='repo/icons-320/info.guardianproject.urzip.100.png'>
fdroid/fdroidserver/fdroidserver/update.py:721: DeprecationWarning: The
'warn' function is deprecated, use 'warning' instead
This replaces the current default behavior of always forcing the
build_tools version and allows the user to set build-tools forcing in
config.py.
closes#147
By checking first, this prevents a stacktrace when the duplicate metadata
file is not valid. For example, in the tests, the duplicate is just a zero
length file, which was causing a stacktrace.
Switching to a script /etc/profile.d to set up the environment is all
well and good, except that doesn't get run when you directly execute a
command directly via ssh, which means that the buildserver didn't work
at all (at least for anything that used gradle, or relied on the
environment variables.
This fix doesn't look very nice, but it works - it just forces the
appropriate script to run before build.py is executed on the server.
(Side note, I thought we had tests for this, how did it get past them?)
switch SDK/NDK/gradle buildserver provisioning to shell scripts
This converts the SDK, NDK, and gradle chef recipes to vagrant provisioning shell scripts. Those recipes were just shell scripts anyway, forced into that ugly ruby/chef syntax. Hopefully this will make things easier to maintain since simple bash is easier for most devs than ruby/chef. Also, it might speed up the provisioning a little bit since the whole script is sent to the VM then executed, rather than sent line-by-line. Additionally, the SDK components are now installed using `android update sdk` so we do not need to duplicate Google's crazy kludges with that stuff.
See merge request !132
When the metadata changes, different things will be stored about each APK.
So invalidate the cached info parsed from APKs if the cache's metadata
version does not match the metadata version of the currently running tools.
bash provides a standard file location for a script to be run when the
shell starts: /etc/profile.d/ This converts the scattered bits of code for
making ~/.bsenv into a single provisioning script to generate
/etc/profile.d/bsenv.sh, which gets automatically executed when bash starts
.java .gradle and XML files all can use any encoding. Most code is ASCII,
but authors' names, etc. can easily be non-ASCII. UTF-8 is by far the most
common file encoding. While UTF-8 is the default encoding inside the code
in Python 3, it still has to deal with the real world, so the encoding
needs to be explicitly set when reading and writing files. So this switches
fdroidserver to expect UTF-8 instead of ASCII when parsing these files. For
now, this commit means that we only support UTF-8 encoded *.java, pom.xml
or *.gradle files. Ideally, the code would detect the encoding and use the
actual one, but that's a lot more work, and its something that will not
happen often. We can cross that bridge when we come to it.
One approach, which is taken in the commit when possible, is to keep the
data as `bytes`, in which case the encoding doesn't matter.
This also fixes this crash when parsing gradle and maven files with
non-ASCII chars:
ERROR: test_adapt_gradle (__main__.BuildTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/tests/build.TestCase", line 59, in test_adapt_gradle
fdroidserver.build.adapt_gradle(testsdir)
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/fdroidserver/build.py", line 445, in adapt_gradle
path)
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/fdroidserver/common.py", line 188, in regsub_file
text = f.read()
File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 9460: ordinal not in range(128)
This makes UTF-8 the sole supported encoding for F-Droid's files. This is
mostly codifying the already existing practice for config.py and index.xml.
The other files where always just ASCII before.
* config.py
* metadata/*.txt
* known_apks.txt
* categories.txt
* latestapps.txt
* latestapps.dat
* index.xml
Note: this does not change the read/write encoding of stats files. That is
still ASCII.
This fixes fdroid build com.umang.dashnotifier:18:
ERROR: Could not build app com.umang.dashnotifier due to unknown error: Traceback (most recent call last):
File "/home/mvdan/git/fsr/fdroidserver/build.py", line 1121, in main
options.onserver, options.refresh):
File "/home/mvdan/git/fsr/fdroidserver/build.py", line 950, in trybuild
build_local(app, build, vcs, build_dir, output_dir, srclib_dir, extlib_dir, tmp_dir, force, onserver, refresh)
File "/home/mvdan/git/fsr/fdroidserver/build.py", line 480, in build_local
extlib_dir, onserver, refresh)
File "/home/mvdan/git/fsr/fdroidserver/common.py", line 1335, in prepare_source
srclibpaths.append(getsrclib(lib, srclib_dir, build, preponly=onserver, refresh=refresh))
File "/home/mvdan/git/fsr/fdroidserver/common.py", line 1258, in getsrclib
cmd = replace_config_vars(srclib["Prepare"], None)
File "/home/mvdan/git/fsr/fdroidserver/common.py", line 1822, in replace_config_vars
cmd = cmd.replace('$$NDK$$', build.ndk_path())
AttributeError: 'NoneType' object has no attribute 'ndk_path'
get_ndk_path was removed long ago when App and Build were introduced.
Now Build.ndk_path() should be used. Trying to use the object as a dict
resulted in a nasty crash:
TypeError: 'Build' object is not subscriptable
This is probably a result from faulty rebasing and not enough test
coverage.
Though the YAML people recommend .yaml for the file extension, in Android
land it seems clear that .yml has won out:
* .travis.yml
* .gitlab-ci.yml
* .circle.yml
* Ansible main.yml
This adds a new method for `fdroid import` that will generate the fdroidserver
metadata based on a local git repo. This new mode generates the metadata in
the new .fdroid.yaml format in the git repo itself. It is intended as a quick
way to get starting building apps using the fdroidserver tools.
This changes the function name to include the format of the metadata file,
and also changes the order of the args to match the parse_*_metadata()
functions.
The default config for .fdroid.* metadata that is included in a git repo is
different than for the standard metadata/ layout because the expectations
are different. In this case, the most common user will be the app
developer working on the latest update of the app on their own machine.
Since you can have a .fdroid.* metadata file included directly in an
app's git repo, it seems annoying to force developers to also include
a blank config.py in order to run `fdroid build` from their git repo.
The next step after this is some kind of global config file for
setting paths and things that need to be customized that would apply
to all git repos, for example, to enable using a buildserver.
This also required moving all of the env setting for FDroidPopen into
a single place so that it is properly setup when using the local
metadata.
The start up sequence of processes that are based on the .fdroid.* metadata
is a bit different, so this ensures that the environment variables get
properly initialized in all cases.
This also creates a single function where the environment is set. Before
it was being set in multiple places across multiple files.
This allows app makers to include a .fdroid.(json|xml|yaml|txt) metadata
file in the root of the git repo of their app, then they can build it using
`fdroid build`. This is useful for developers who want to maintain the
fdroid build recipe themselves, and run the fdroid build process for their
own builds.
If multiple tags point at the same commit, limiting the regex search to
one tag per line would only catch one tag. This broke org.wikipedia's
update check.
File "/var/lib/jenkins/workspace/fdroidserver-makebuildserver/fdroidserver/build.py", line 1119, in main
options.onserver, options.refresh):
File "/var/lib/jenkins/workspace/fdroidserver-makebuildserver/fdroidserver/build.py", line 954, in trybuild
build_server(app, build, vcs, build_dir, output_dir, force)
File "/var/lib/jenkins/workspace/fdroidserver-makebuildserver/fdroidserver/build.py", line 403, in build_server
output += get
TypeError: Can't convert 'bytes' object to str implicitly
closes#164https://gitlab.com/fdroid/fdroidserver/issues/164
Fixes a couple errors like:
File "./makebuildserver", line 30, in vagrant
out += line
TypeError: Can't convert 'bytes' object to str implicitly
If universal_newlines=False, the default, then Popen will return bytes if
the newlines in the data do not match the system's newlines. Setting it to
true enables auto-conversion, and then guarantees that the data is always
str.
"If universal_newlines is True, the file objects stdin, stdout and stderr
are opened as text streams in universal newlines mode, as described above
in Frequently Used Arguments, otherwise they are opened as binary streams."
https://docs.python.org/3/library/subprocess.html#subprocess.Popen
Python 3
I tried to keep commits separate, so if anything causes trouble, it can be reverted or changed easily.
* pre-commit hooks pass
* all tests pass
* My use of `build`, `checkupdates`, `lint`, `import`, `publish` and `update` work as usual
* 2to3 does not report anything useful anymore (only useless parentheses and list() encapsulation of iterators)
* rewritemeta works exactly as usual
CC @eighthave
See merge request !88
For some reason, Python uses whatever encoding the system is using via a
locale. Since CI uses an ascii locale, stuff breaks. I can't find a way
to make all of fdroidserver ignore the locale and just use UTF-8, so for
now force it where it's bothering CI.
* We can now get the output bytes directly if desired
* FDroidPopen remains with str output
* Ouptut is now buffered efficiently with a BytesIO instead of
concatenating strings
This greatly speeds up checkupdates when UCM:Tags is used on git
repositories, especially on ones with lots of tags. The old method ran a
command per tag. The new method runs a single command and doesn't
require any sorting.
Test runs show that `fdroid checkupdates org.adaway` is down from ~13s
to ~10s on my laptop.
The point of removing these was to make sure that gradlew wasn't used,
and that our gradle was used instead. Removing the scripts already
accomplishes this.
Removing gradle/ should be harmless, but some apps like I2P re-use this
directory to also hold other stuff that is actually needed. So be safer
and don't remove it at all.
Fix pubkey extraction on update
Replacement of !86.
Fix pubkey extraction in case of non-empty _JAVA_OPTIONS. Fixes#133.
I didn't actually run the test suite (it looks like there are some preparations to be done for that), but I checked the commands from `test_fdroid_popen_stderr_redirect` in ipython.
See merge request !103
We weren't using ^ and $, so stuff like java-8-openjdk-i386 would match
both the Arch and the Debian regexes. A list with one regex per line in
the same format is also easier to read and maintain.
This might be why java8 isn't being detected properly on our Debian
stable buildserver.
This avoids the following incorrect error:
fdroidserver.common.FDroidException: Only one local metadata file allowed! Found: .fdroid.keystorepass.txt .fdroid.keypass.txt
Allow commonsware and gradle plugin repos
This is in a MR because I'm uneasy about adding a semi-random AWS url to the list. But seems like the commonsware guy only publishes jars there, which doesn't make any sense to me.
Any idea @eighthave?
See merge request !102
serverwebroot has long supported uploading to multiple servers, this bit of
metadata communicates those official mirrors to the client so that it can
automatically do something useful with that information.
closes#14https://gitlab.com/fdroid/fdroidserver/issues/14
Before, `fdroid build` would create some subdirs, then fail when it could
not read the metadata via metadata.read_metadata(). This checks before
whether there is any metadata available, and warns the user accordingly.
Using the same JDK throughout should prevent weird bugs where a setup might
use Java8's jarsigner and Java7's keytool. This also allows the user to
set java_paths and have jarsigner and keytool used from that specified JDK.
This incorporates almost all of the patch that is in the Debian package
that forces fdroidserver to use the default JDK on that Debian release.
closes#93https://gitlab.com/fdroid/fdroidserver/issues/93
If we don't find the version name neither in the AndroidManifest.xml nor
in the build.grade file, we fallback to the tag the user specified to
search for.
That way we have a better version name than 'Unknown'.
I was wrong - we don't check for trailing whitespaces on lines. That
would have to happen at parse time, not in the linter, so it would slow
things down and would become an error, not a warning. Not really worth
it.
1. It is packaged in modern versions of distros, update docs accordingly
2. 1.1 is hopelessly outdated - support 1.4 onwards
3. Check the version properly, and write a better config (previously it
wrote the 1.1 config for 1.1 and 1.3+
4. Ensure that the default share from later versions is not present when
building, it's only required for provisioning.
Since we only sorted by count, ignoring the string, it meant that items
with the same count might be arranged in different manners. Hence the
`stats` behaviour was not predictable at all. Now it sorts first by
count, then by string.
It is a good idea in theory, especially for extra tasks specified by the
user. But in practice, tasks specified by developers often don't have a
clean counterpart. So this just breaks the build.
We were passing the utf-8 encoded string to textwrap, which took the
bytes as characters. Hence multi-byte unicode characters (in utf-8)
would count as multiple columns, which is clearly wrong.
This breaks on newer plugin versions. Instead of complicating this logic
even further to support them, upstream devs should instead use gradle
configs that actually work.
If their build fails because of lint, it's theirs to fix, not ours.
