support cloud storage, and assorted other improvements
I just finished support for `fdroid server update` to push to Amazon AWS S3 cloud store. It uses libcloud, so there could be lots of other cloud storage services added. This would be used for alternative hosting locations for repos. For example, for the Guardian Project repo, we'd include the URLs to various cloud storage options like https://s3.amazonaws.com/guardianproject/fdroid/repo
Services like https://s3.amazonaws.com/ are often not blocked when other things are.
It does not need to be Amazon-specific. I went with libcloud because it supports like 10 cloud storage and is under active development. What is there is just the start. I'm new to cloud stuff, so I just started based on a script that Adam Prichart of psiphon gave me. I had to do a fair amount of packaging work to get the python-libcloud Debian package updated. I almost have the 0.14.1 update done, I hope that'll be in Debian tonight.
Lots more info in the commit messages.
Since it is possible to check the file size and MD5 hash of the file up on
the AWS S3 bucket, `fdroid server update` can check that a file needs to be
updated before actually deleting and uploading the new file.
fixes#3137https://dev.guardianproject.info/issues/3137
This makes the AWS S3 setup dead simple: just put in a awsbucket name of
your choosing, set the AWS credentials, and it'll do the rest, whether the
bucket exists already or not. S3 buckets are trivial to delete too, in
case of error: `s3cmd rb s3://mybadbucketname`.
apache-libcloud enables uploading to basically any cloud storage service.
This is the first implementation that allows `fdroid server` to push a repo
up to a AWS S3 'bucket'. Supporting other cloud storage services should
mostly be a matter of finding the libcloud "Provider" and setting the
access creditials.
fixes#3137https://dev.guardianproject.info/issues/3137
Right now, ssh+rsync is the only supported server upload type. Things like
cloud storage services are useful storage bins for fdroid repos since they
are often not blocked while specific websites like Google Play are.
Having serverwebroot optional in `fdroid server` means that it can support
multiple methods of hosting, like cloud storage services. `fdroid server`
can also then support multiple repo hosting options at the same time.
The .fdroid.*.txt password files are only meant to be a conduit for the
passwords, so blow them away everytime. The canonical password is stored
in config.py.
It might makes sense to replace these files with env vars using
-storepass:env and -keypass:env. I figured that the passwords are already
in a file, config.py, so adding more files in the same location with the
same perms would not increase the risk at all.
* Group apk, jar and zip files in the same case
* Use regex to support more patterns and be more flexible
* Only check for usual suspects in jar files (saves time)
* Also catch unknown zip-like files as warnings
implemeted support for using a smartcard for the repo signing key
This changeset implements using a smartcard (HSM) as the keystore for the signing key. It also fixes lots of little bugs in the `fdroid init` process.
