Python encode/decode libs work directly with dicts, so the internal dict
can just be passed directly to any of these libs (pyyaml, pyjson, msgpack,
simplejson, etc). This still generates the exact same index.xml as before.
This converts the internal format for the repo timestamp to a datetime
instance, which can be easily converted to UNIX time in seconds for XML
and UNIX time in milliseconds for the new index formats. UNIX time in
milliseconds is directly serialized into a java.util.Date instance by
Jackson.
A Binary Transparency Log is a append only log of all binaries published by
a repo. This is useful for people to find whether the binary they have
matches what F-Droid has published, and also makes it more difficult for
the published history to be changed without notice, or for a server to give
specific users custom malware binaries.
https://www.eff.org/deeplinks/2014/02/open-letter-to-tech-companies
following guidelines from:
https://docs.python.org/2/library/argparse.html#upgrading-optparse-code
except, still using option = parse.parse_args() instead of args = ...
- using the following script in folder fdroidserver:
for i in *.py; do
sed -i -e 's/optparse/argparse/' \
-e 's/OptionParser/ArgumentParser/' \
-e 's/OptionError/ArgumentError/' \
-e 's/add_option/add_argument/' \
-e 's/(options, args) = parser/options = parser/' \
-e 's/options, args = parser/options = parser/' \
-e 's/Usage: %prog/%(prog)s/' $i;
done
- use ArgumentParser argument to replace (option, args) = parser.parse()
call
- use parser.error(msg) instead of raise ArgumentException as suggested
in https://docs.python.org/2/library/argparse.html#exiting-methods
- in fdroid catch ArgumentError instead of OptionError
This keeps memory usage low because it only has to read a chunk at a time
into memory while before it read the whole file into memory before
uploading it.
This also seems to handle setting the permissions ACL better.
For devs that want to build and distribute nightly builds of their apps
using the fdroid tools. The core idea here is to make the fdroidserver
tool suite the single set of tools for all types of builds and releases.
That will hopefully drive more free software developers to make f-droid.org
an core channel for official releases.
For use cases where there is a web server running on the same machine where
the `fdroid update` is being run, allow plain paths in the serverwebroot
list. This is useful for debug repos from build servers, like:
https://dev.guardianproject.info/fdroid
In order to keep things working as much as possible during the update, the
rsync should only delete the obsolete APKs after it has finished uploading
the new APKs.
rsync's --chmod works a bit oddly, it only affects the source files. To
make it set the destintation with the perms set in --chmod, the --perms
flag must also be included.
With FAT filesystems, the user, group, and permissions will not be at all
preserved. With file systems like ext4 that have perms, the umask might
not be set to something that makes sense for the public repo files, which
are meant to be published and therefore readible by all.
If need be, it would be easy enough to add a config option for rsync's
chmod string, to address setups that have specific permissions needs.
fixes#23https://gitlab.com/fdroid/fdroidserver/issues/23
It is easier to handle programming with python rather than subprocess calls
so I replaced the subprocess call to 'ssh' with paramiko. This also makes
fdroid more portable since it no longer relies on the local system having
ssh installed.
It seems that paths for rsync must have a trailing slash in order to sync
rather than make a subdir, i.e. this makes a duplicate subdir:
rsync /tmp/fdroid/repo repo
While this syncs the dirs
rsync /tmp/fdroid/repo/ repo/
If `fdroid server update` is run with config that includes an archive, but
the 'archive' subdir does not exist, create it. This mirrors the code that
is in `fdroid update`. Seems to trivial to move to common.py.
To support a fully offline build/signing machine, there is the "local copy
dir". The repo is generated on the offline machine and then copied to a
local dir where a thumb drive or SD Card is mounted. Then on the online
machine, using `fdroid server update --sync-from-local-copy-dir` allows
the whole server update process to happen in a single command:
0. read config.py on online machine's repo
1. rsync from the local_copy_dir to the current dir
2. copy to serverwebroot, awsbucket, etc.
In `fdroid server update`, the rsync command used --update, which
`man rsync` says: "skip files that are newer on the receiver". That could
cause issues of the public repo getting out of sync with the private,
master repo. --archive is a better sync method since it aims to exactly
reproduce the sending dir to the receiving dir.
This allows a dir to be specified in config.py that `fdroid server update`
will automatically rsync the repo to. The idea is that the path would
point to an SD card on a fully offline machine that serves as the secure
repo signing machine.
This allows the SSH key used to sync with the server to be specified via
the config.py or the command line. I need it for running automated tests
and setups.
rsync uses the modification time and size of the file when deciding whether
to update a file. These are relatively easy to control in malicious code,
so instead make rsync use a full MD5 checksum when decided whether the
index needs to be updated. I suppose we could add an option to use
checksum checking on all files, but since the signed repo already provides
a checksum check, it seems not worth the added load on the process.
Also, renamed 'index' to 'indexxml' to make it clear what is the XML and
what is the JAR.
If a key 'foo' is set to None, `if config.get('foo'):` will be false while
`if 'foo' in config:` will be true. A None value is not useful here, so
config.get() is the better check.
Thanks to Adam Pritchard for the suggestion.
Since it is possible to check the file size and MD5 hash of the file up on
the AWS S3 bucket, `fdroid server update` can check that a file needs to be
updated before actually deleting and uploading the new file.
fixes#3137https://dev.guardianproject.info/issues/3137
This makes the AWS S3 setup dead simple: just put in a awsbucket name of
your choosing, set the AWS credentials, and it'll do the rest, whether the
bucket exists already or not. S3 buckets are trivial to delete too, in
case of error: `s3cmd rb s3://mybadbucketname`.
apache-libcloud enables uploading to basically any cloud storage service.
This is the first implementation that allows `fdroid server` to push a repo
up to a AWS S3 'bucket'. Supporting other cloud storage services should
mostly be a matter of finding the libcloud "Provider" and setting the
access creditials.
fixes#3137https://dev.guardianproject.info/issues/3137
Right now, ssh+rsync is the only supported server upload type. Things like
cloud storage services are useful storage bins for fdroid repos since they
are often not blocked while specific websites like Google Play are.
Having serverwebroot optional in `fdroid server` means that it can support
multiple methods of hosting, like cloud storage services. `fdroid server`
can also then support multiple repo hosting options at the same time.
For user-generated repos, the default path/URL is .*/fdroid/repo, with
fdroid/ as the root where the 'fdroid' tool operates. This makes for a URL
that is quite unique and easily matched automatically using patterns, like
in fdroidclient.
For those who don't like the standard, they can override the errror from
config.py using nonstandardwebroot = True
