Fix file read/write encoding
This hopefully provides a reasonably complete set of fixes for handling file encoding when reading and writing files. It is not perfect since it does not handle _.java_, _.gradle_, or _pom.xml_ in any encoding, as is allowed. But I think it should only be an improvement of the current state, and at worst, should work only as bad as the current setup for most language setups
See merge request !129
.java .gradle and XML files all can use any encoding. Most code is ASCII,
but authors' names, etc. can easily be non-ASCII. UTF-8 is by far the most
common file encoding. While UTF-8 is the default encoding inside the code
in Python 3, it still has to deal with the real world, so the encoding
needs to be explicitly set when reading and writing files. So this switches
fdroidserver to expect UTF-8 instead of ASCII when parsing these files. For
now, this commit means that we only support UTF-8 encoded *.java, pom.xml
or *.gradle files. Ideally, the code would detect the encoding and use the
actual one, but that's a lot more work, and its something that will not
happen often. We can cross that bridge when we come to it.
One approach, which is taken in the commit when possible, is to keep the
data as `bytes`, in which case the encoding doesn't matter.
This also fixes this crash when parsing gradle and maven files with
non-ASCII chars:
ERROR: test_adapt_gradle (__main__.BuildTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/tests/build.TestCase", line 59, in test_adapt_gradle
fdroidserver.build.adapt_gradle(testsdir)
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/fdroidserver/build.py", line 445, in adapt_gradle
path)
File "/var/lib/jenkins/workspace/fdroidserver-eighthave/fdroidserver/common.py", line 188, in regsub_file
text = f.read()
File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 9460: ordinal not in range(128)
This makes UTF-8 the sole supported encoding for F-Droid's files. This is
mostly codifying the already existing practice for config.py and index.xml.
The other files where always just ASCII before.
* config.py
* metadata/*.txt
* known_apks.txt
* categories.txt
* latestapps.txt
* latestapps.dat
* index.xml
Note: this does not change the read/write encoding of stats files. That is
still ASCII.
Install version 25 now. Also use the smaller tools zip. While at it,
also remove the tools re-install - it's not worth it, as long as we keep
the initial tools zip up to date.
CI: Bump image
This is basically just an SDK update. We're now using the tools zip,
which is about 200MB smaller. Not entirely sure why, but things still
work.
See merge request !123
Since base is now 200MB smaller, this one is smaller too. The
fdroidserver-only deps are now in the client image - mainly python3-dev,
gcc and all the build deps for stuff like pillow.
Prefer pyvenv instead of virtualenv for Python 3.3+
`virtualenv` uses python2 by default on my machine.
Using `pyvenv` (see https://docs.python.org/3/library/venv.html#module-venv) instead of `virtualenv` fixes the problem and let's me perform `python3 setup.py install` in the virtual environment.
See merge request !119
Add a workaround to open apkcache created with Py2
A workaround for #163.
*May* help in some rare cases, should not break anything.
See merge request !120
This fixes fdroid build com.umang.dashnotifier:18:
ERROR: Could not build app com.umang.dashnotifier due to unknown error: Traceback (most recent call last):
File "/home/mvdan/git/fsr/fdroidserver/build.py", line 1121, in main
options.onserver, options.refresh):
File "/home/mvdan/git/fsr/fdroidserver/build.py", line 950, in trybuild
build_local(app, build, vcs, build_dir, output_dir, srclib_dir, extlib_dir, tmp_dir, force, onserver, refresh)
File "/home/mvdan/git/fsr/fdroidserver/build.py", line 480, in build_local
extlib_dir, onserver, refresh)
File "/home/mvdan/git/fsr/fdroidserver/common.py", line 1335, in prepare_source
srclibpaths.append(getsrclib(lib, srclib_dir, build, preponly=onserver, refresh=refresh))
File "/home/mvdan/git/fsr/fdroidserver/common.py", line 1258, in getsrclib
cmd = replace_config_vars(srclib["Prepare"], None)
File "/home/mvdan/git/fsr/fdroidserver/common.py", line 1822, in replace_config_vars
cmd = cmd.replace('$$NDK$$', build.ndk_path())
AttributeError: 'NoneType' object has no attribute 'ndk_path'
get_ndk_path was removed long ago when App and Build were introduced.
Now Build.ndk_path() should be used. Trying to use the object as a dict
resulted in a nasty crash:
TypeError: 'Build' object is not subscriptable
This is probably a result from faulty rebasing and not enough test
coverage.
CI: Use the new base image
Lighter, since it doesn't include all the stuff required to build and
test the client that we don't need here.
See merge request !116
