This greatly speeds up checkupdates when UCM:Tags is used on git
repositories, especially on ones with lots of tags. The old method ran a
command per tag. The new method runs a single command and doesn't
require any sorting.
Test runs show that `fdroid checkupdates org.adaway` is down from ~13s
to ~10s on my laptop.
The point of removing these was to make sure that gradlew wasn't used,
and that our gradle was used instead. Removing the scripts already
accomplishes this.
Removing gradle/ should be harmless, but some apps like I2P re-use this
directory to also hold other stuff that is actually needed. So be safer
and don't remove it at all.
Fix pubkey extraction on update
Replacement of !86.
Fix pubkey extraction in case of non-empty _JAVA_OPTIONS. Fixes#133.
I didn't actually run the test suite (it looks like there are some preparations to be done for that), but I checked the commands from `test_fdroid_popen_stderr_redirect` in ipython.
See merge request !103
We weren't using ^ and $, so stuff like java-8-openjdk-i386 would match
both the Arch and the Debian regexes. A list with one regex per line in
the same format is also easier to read and maintain.
This might be why java8 isn't being detected properly on our Debian
stable buildserver.
This avoids the following incorrect error:
fdroidserver.common.FDroidException: Only one local metadata file allowed! Found: .fdroid.keystorepass.txt .fdroid.keypass.txt
Allow commonsware and gradle plugin repos
This is in a MR because I'm uneasy about adding a semi-random AWS url to the list. But seems like the commonsware guy only publishes jars there, which doesn't make any sense to me.
Any idea @eighthave?
See merge request !102
serverwebroot has long supported uploading to multiple servers, this bit of
metadata communicates those official mirrors to the client so that it can
automatically do something useful with that information.
closes#14https://gitlab.com/fdroid/fdroidserver/issues/14
Before, `fdroid build` would create some subdirs, then fail when it could
not read the metadata via metadata.read_metadata(). This checks before
whether there is any metadata available, and warns the user accordingly.
Using the same JDK throughout should prevent weird bugs where a setup might
use Java8's jarsigner and Java7's keytool. This also allows the user to
set java_paths and have jarsigner and keytool used from that specified JDK.
This incorporates almost all of the patch that is in the Debian package
that forces fdroidserver to use the default JDK on that Debian release.
closes#93https://gitlab.com/fdroid/fdroidserver/issues/93
If we don't find the version name neither in the AndroidManifest.xml nor
in the build.grade file, we fallback to the tag the user specified to
search for.
That way we have a better version name than 'Unknown'.
I was wrong - we don't check for trailing whitespaces on lines. That
would have to happen at parse time, not in the linter, so it would slow
things down and would become an error, not a warning. Not really worth
it.
1. It is packaged in modern versions of distros, update docs accordingly
2. 1.1 is hopelessly outdated - support 1.4 onwards
3. Check the version properly, and write a better config (previously it
wrote the 1.1 config for 1.1 and 1.3+
4. Ensure that the default share from later versions is not present when
building, it's only required for provisioning.
Since we only sorted by count, ignoring the string, it meant that items
with the same count might be arranged in different manners. Hence the
`stats` behaviour was not predictable at all. Now it sorts first by
count, then by string.
It is a good idea in theory, especially for extra tasks specified by the
user. But in practice, tasks specified by developers often don't have a
clean counterpart. So this just breaks the build.
We were passing the utf-8 encoded string to textwrap, which took the
bytes as characters. Hence multi-byte unicode characters (in utf-8)
would count as multiple columns, which is clearly wrong.
This breaks on newer plugin versions. Instead of complicating this logic
even further to support them, upstream devs should instead use gradle
configs that actually work.
If their build fails because of lint, it's theirs to fix, not ours.
This allows us to run checks and fix types only on those that were
really modified. On our fdroiddata repo, with 2k apps with many unset
fields and flags, this reduces readmeta runtime from ~1.3s to ~1.1s.
Avoids typos, such as one I just found which was 'strsng' isntead of
'string'. The static analyzer can catch those if they are constants.
Comparing ints is also faster than strings, which adds up in readmeta.
Only keep lists in metadata files in the json format, since they don't
support multiline strings that are readable.
This makes the internal code easier, and a bit faster.
This simplifies usage, goes from
build['flag']
to
build.flag
Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.
As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.
While at it, unify "build", "thisbuild", "info", "thisinfo", etc into
just "build".
This simplifies usage, goes from
app['Foo']
to
app.Foo
Also makes static analyzers able to detect invalid attributes as the set
is now limited in the class definition.
As a bonus, setting of the default field values is now done in the
constructor, not separately and manually.
Don't log and exit in an inner metadata function. Handle it at a higher
level and do a proper exception. This also avoids unnecessary passing of
apps all around.
They all were stuck in repo/ for this reason. This resulted in apps that
were only in the archive to seem like they didn't have icons, when in
fact they did but were in the wrong dir.
If there are multiple package ids in a build.gradle file and the first
one we don't want, checkupdates would get stuck. Make it ignore any
package id that we don't want so that it can get past that.
In cases like this xml code:
<string name="app_name">"OpenKeychain"</string>
<!-- title -->
<string name="title_encrypt_text">"Encrypt"</string>
tostring() returns trailing whitespaces (including newlines). Which
aren't removed until the very end, after we try to remove enclosing
quotes. So strip right after tostring() too, since we never really care
about whitespaces anyway.
Up until now, this was done only if no overall version could be found. But in
tags, we would get stuck with an old version from an old tag and not see
subdir changes in new tags.
This finds maven repos of the format:
maven {
url 'http://foo.bar'
}
And checks if the repository is one that we allow. As usual, scanignore can be
used, or the list modified, if there are exceptions or more repositories to
allow.
* Move apks back from the archive repo (e.g. if builds are disabled thus the
number of apks on the main repo decreases)
* Also move apk signatures back and forth
This allows us to fetch strings like the following:
<string name="app_name">foo <xliff:g>bar</xliff:g></string>
Up until now, using .text would only return "foo ", but if we use .tostring()
with the text method, it converts everything into plain text for us resulting
in "foo bar".
8k is too small, e.g. when make -jN errors or when there are a lot of scanner
errors and warnings. 16k should be better. Also, use "[...]" to make it
obvious that the output is truncated.
Looks to me like we were using SHA1withRSA all along. Tested that everything
still works with a test repo.
As Hans reports, SHA1withRSA has been the default on Android tools for a long
time and it's supported on all Android versions.
Add asynchronous filereader, fix python3 lockups
with the current implementation of AsynchronousFileReader with Python 3 FDroidPopen deadlocks with 100% CPU-utilization
The code is from https://github.com/soxofaan/asynchronousfilereader
I hope the MIT-Licence makes no problems
See merge request !80
Added support for gradle flavor specific dependencies in usual suspects check
When you have flavor specific dependencies in your project, you specify them in the build.gradle file with a prefixed *compile* command:
```
compile 'normal dependency'
myflavorCompile 'only for myflavor'
```
The usual suspects check just searches for some often used library names in the whole build.gradle file. I added another condition, so dependencies for not build flavors are not checked.
See merge request !77
This is much closer to what we did before with mimetypes. Using a whitelist
turns out to be a bad idea since repositories seem to be randomly filled with
executable images and documents, which trigger the scanner.
In an ideal world the scanner would complain about all of those. For now, just
warn about the possibility of them being hidden binaries.
Initially, the scanner used libmagic which used magic numbers in the file's
content to detect what kind of file it appears to be. Since that library isn't
available on all systems, we added support for two other libraries, mimetypes
amongst them.
The issue with mimetypes is that it only uses the file's extension, not its
actual content. So this ends in variable behaviour depending on what system
you're using fdroidserver on. For example, an executable binary without
extension would be ignored if mimetypes was being used.
We now drop all libraries - mimetypes too as it depends on the system's
mime.types file - and instead check extensions ourselves. On top of that, do
a simple binary content check to find binary executables that don't have an
extension.
The new in-house code without any dependencies doesn't add any new checks, so
no builds should break. The current checks still work:
% fdroid scanner app.openconnect:1029
[...]
Found executable binary at assets/raw/armeabi/curl
Found executable binary at assets/raw/mips/curl
Found executable binary at assets/raw/x86/curl
Found JAR file at lib/XposedBridgeApi-54.jar
Found JAR file at libs/acra-4.5.0.jar
Found JAR file at libs/openconnect-wrapper.jar
Found JAR file at libs/stoken-wrapper.jar
Found shared library at libs/armeabi/libopenconnect.so
Found shared library at libs/armeabi/libstoken.so
Found shared library at libs/mips/libopenconnect.so
Found shared library at libs/mips/libstoken.so
Found shared library at libs/x86/libopenconnect.so
Found shared library at libs/x86/libstoken.so
