This creates a mirror of a full repo by downloading all files listed in
the index, and the ones that are generated based on that data, e.g. icons
of different resolutions. This could be useful for setting up mirrors of
small repositories, instead of having to learn and manage rsync or
something else for mirroring. This just needs a working repo.
It uses wget in a batch mode with the aim as being as efficient as
possible. wget mirroring over HTTP is always going to be less efficient
than rsync, but it shouldn't be so bad since it uses --continue to check
whether it has already downloaded a file. I suppose it could be extended
to use ETags for a little more efficiency.
I developed this creating a test mirror of f-droid.org, which is now a bit
ironic, since I added a specific check to prevent people from using this
on f-droid.org.
jenkins-build-all just sets `build_server_always = True`, these tests need
a whole keystore setup. So just delete the existing config, and let
`fdroid init` do the rest.
Sending many small files with paramiko's sftp implementation is quite
slow. There seems to be no way around this, other projects (ansible)
recommend to use rsync in this case.
Our sourcecode directories sometimes have a LOT of files, it can take
up to an hour to copy all required things inside the buildserver.
Instead we just use rsync with the ssh options we get from vagrant.
For rsync specific options we use:
* sync permissions as before
* sync symlinks (was done partially before)
* don't sync hardlinks and ownership as these don't make for things
coming from a VCS
Closesfdroid/fdroidserver#227.
We never allow git via SSH or password/key access, and right now, this
causes things to hang forever. This sets things up to fail quickly
with invalid ssh connections.
BatchMode=yes - passphrase/password querying will be disabled.
StrictHostKeyChecking=yes - never automatically prompt, or add host keys to
the ~/.ssh/known_hosts file, and refuse to connect to hosts whose host key
has changed.
If a git fetch/clone/submodule URL points to gitlab, github, bitbucket, etc
and that repo does not exist any more, those services will prompt the user
for a username/password so that the service can check if its a private
repo. Private repos show up the same as non-existent repos. This employs
two techniques for making sure that git never waits at those prompts. It
instead should just fail immediately. The buildserver has been hanging on
these prompts forever, until manually killed. This change will apply to
updates both on the buildserver host, and the buildserver guest vm.
This uses the "insteadOf" git config option to rewrite URLs to always use
HTTPS and then include a fake username/password so that git will use those
in the prompts and fail immediately. This trick has been in use on the
verification server for a long while and has been working well. It has
also been used on jenkins.debian.net in the host.
https://f-droid.org/en/docs/Verification_Server/
It also includes GIT_TERMINAL_PROMPT, which also prevents the bad behavior,
which was added in git 2.3.
https://github.com/blog/1957-git-2-3-has-been-released
If people want to create repos, they should be using `fdroid init`. This
flag is for very specific cases: migrating old unsigned repos to signed.
It shouldn't be promoted for anything else since there are better tools for
those tasks.
Now that fdroidserver requires Python 3, there's no need to tune sshd for
an old version of Paramiko because Ubuntu 14.04 does not have
python3-paramiko package in its repositories:
https://packages.ubuntu.com/search?keywords=python3-paramiko
We can safely assume that Ubuntu 14.04 users have installed a more recent
version of Paramiko from PIP.
The old hash does not work anymore, at least for platform-27. Where this
new one comes from:
1. Download command line tools from
https://developer.android.com/studio/index.html#command-tools
2. Unzip the package
3. Run ./tools/bin/sdkmanager --licenses
4. Read and accept licenses
5. Find the hash in ./licenses/android-sdk-license
Google releases new API SDKs quite rarely (about 2 times a year), but when
they do this, many apps quickly start using it. Allow downloading new API
SDKs to avoid waiting for the next buildserver rebuild.
