fdroidserver/docs/fdroid.texi

\input texinfo   @c -*-texinfo-*-
@c %**start of header
@setfilename fdroid.info
@settitle F-Droid Server Manual
@c %**end of header

@copying
This manual is for the F-Droid repository server tools.

Copyright @copyright{} 2010, 2011, 2012 Ciaran Gultnieks
Copyright @copyright{} 2011 Henrik Tunedal, Michael Haas, John Sullivan

@quotation
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is included in the section entitled "GNU
Free Documentation License".
@end quotation
@end copying

@titlepage
@title F-Droid Server Manual
@author Ciaran Gultnieks and the F-Droid project
@page
@vskip 0pt plus 1filll
@insertcopying

@end titlepage

@contents

@ifnottex
@node Top
@top F-Droid Server

@insertcopying
@end ifnottex

@menu
* Overview::
* System Requirements::
* Setup::
* Simple Binary Repository::
* Building Applications::
* Importing Applications::
* Metadata::
* Build Server::
* GNU Free Documentation License::
* Index::
@end menu

@node Overview
@chapter Overview

The F-Droid server tools provide various scripts, tools and data that are used
to maintain the main F-Droid application repository. You can use these same
tools to create your own additional or alternative repository for publishing,
or to assist in creating, testing and submitting metadata to the main
repository.


@node System Requirements
@chapter System Requirements

@cindex installation

The system requirements for using the tools will vary depending on your
intended usage. At the very least, you'll need:

@itemize @bullet
@item
GNU/Linux
@item
Python 2.x
@item
The Android SDK
@end itemize

If you intend to build applications from source you'll also need most, if not
all, of the following:

@itemize @bullet
@item
All available SDK platforms and tools installed in the Android SDK, but *not*
the proprietary components.
@item
The Android NDK
@item
Ant
@item
Ant Contrib Tasks (Debian package ant-contrib)
@item
Maven (Debian package maven2)
@item
JavaCC (Debian package javacc)
@item
JDK (Debian package openjdk-6-jdk)
@item
VCS clients: svn, git, hg, bzr
@item
A keystore for holding release keys. (Safe, secure and well backed up!)
@end itemize

If you intend to use the 'Build Server' system, for secure and clean builds
(highly recommended), you will also need:

@itemize @bullet
@item
VirtualBox (debian package virtualbox-ose)
@item
Ruby
@item
Vagrant and Vagrant-snap
@item
Paramiko (debian package python-paramiko)
@end itemize


@node Setup
@chapter Setup

@cindex setup, installation

Because the tools and data will always change rapidly, you will almost
certainly want to work from a git clone of the tools, which are designed to
work in this way, with all associated data in a pre-defined directory
structure below the main one. To get started:

@example
git clone git://gitorious.org/f-droid/fdroidserver.git
cd fdroidserver
@end example

You will now be in the root directory of the tools. All the tasks associated
with managing the repository and data are done from here.

Regardless of the intended usage of the tools, you will always need to set
up some basic configuration details. This is done by creating a file called
@code{config.py} which you should do by copying from @code{config.sample.py}
and then editing according to the instructions within.

Once configured in this way, all the functionality of the tools is accessed
by running the @code{fdroid} script. Run it on its own to get a list of the
available commands:

@example
./fdroid
@end example

You can follow any command with @code{--help} to get a list of additional
options available for that command.

@example
./fdroid update --help
@end example


@node Simple Binary Repository
@chapter Simple Binary Repository

@cindex binary

If you want to maintain a simple repository hosting only binary APKs obtained
and compiled elsewhere, the process is quite simple:

@enumerate
@item
Make a repo directory and put APK files in it.
@item
Run @code{fdroid update.py}.
@item
If it reports that any metadata files are missing, you can create them
in the @code{metadata} directory and run it again.
@item
To ease creation of metadata files, run @code{fdroid update} with the @code{-c}
option. It will create 'skeleton' metadata files that are missing, and you can
then just edit them and fill in the details.
@item
Then, if you've changed things, run @code{fdroid update} again.
@item
Running @code{fdroid update} adds an Icons directory into the repo directory,
and also creates the repository index (index.xml, and also index.jar if you've
configured the system to use a signed index).
@end enumerate

Following the above process will result in a @code{repo} directory, which you
simply need to push to any HTTP (or preferably HTTPS) server to make it
accessible.

While some information about the applications (and versions thereof) is
retrieved directly from the APK files, most comes from the corresponding file
in the @code{metadata} directory. The metadata file covering ALL versions of a
particular application is named @code{package.id.txt} where package.id is the
unique identifier for that package.

See the Metadata chapter for details of what goes in the metadata file. All
fields are relevant for binary APKs, EXCEPT for 'Build Version' entries, which
should be omitted.


@node Building Applications
@chapter Building Applications

Instead of (or as well as) including binary APKs from external sources in a
repository, you can build them directly from the source code.

Using this method, it is is possible to verify that the application builds
correctly, corresponds to the source code, and contains only free software.
Unforunately, in the Android world, it seems to be very common for an
application supplied as a binary APK to present itself as Free Software
when in fact some or all of the following are true:

@enumerate
@item
The source code (either for a particular version, or even all versions!) is
unavailable or incomplete.
@item
The source code is not capable of producing the actual binary supplied.
@item
The 'source code' contains binary files of unknown origin, or with proprietary
licenses.
@end enumerate

For this reason, source-built applications are the preferred method for the
main F-Droid repository, although occasionally for technical or historical
reasons, exceptions are made to this policy.

When building applications from source, it should be noted that you will be
signing them (all APK files must be signed to be installable on Android) with
your own key. When an application is already installed on a device, it is not
possible to upgrade it in place to a new version signed with a different key
without first uninstalling the original. This may present an inconvenience to
users, as the process of uninstalling loses any data associated with the
previous installation.

The process for managing a repository for built-from-source applications is
very similar to that described in the Simple Binary Repository chapter,
except now you need to:

@enumerate
@item
Include Build Version entries in the metadata files.
@item
Run @code{fdroid build} to build any applications that are not already built.
@item
Run @code{fdroid publish} to finalise packaging and sign any APKs that have
been built.
@end enumerate


@section More about build.py

When run without any parameters, @code{fdroid build} will build any and all
versions of applications that you don't already have in the @code{repo}
directory (or more accurately, the @code{unsigned} directory. There are various
other things you can do. As with all the tools, the @code{--help} option is
your friend, but a few annotated examples and discussion of the more common
usage modes follows:

To build a single version of a single application, you could run the
following:

@example
./fdroid build --package=org.fdroid.fdroid --vercode 16
@end example

This attempts to build version code 16 (which is version 0.25) of the F-Droid
client. Many of the tools recognise this @code{--package} parameter, allowing
their activity to be limited to just a single package.

If the build above was succesful, two files will have been placed in the
@code{unsigned} directory:

@example
org.fdroid.fdroid_16.apk
org.fdroid.fdroid_16_src.tar.gz
@end example

The first is the (unsigned) APK. You could sign this with a debug key and push
it direct to your device or an emulator for testing. The second is a source
tarball containing exactly the source that was used to generate the binary.

If you were intending to publish these files, you could then run:

@example
./fdroid publish
@end example

The source tarball would move to the @code{repo} directory (which is the
directory you would push to your web server). A signed and zip-aligned version
of the APK would also appear there, and both files would be removed from the
@code{unsigned} directory.

If you're building purely for the purposes of testing, and not intending to
push the results to a repository, at least yet, the @code{--test} option can be
used to direct output to the @code{tmp} directory instead of @code{unsigned}.
A similar effect could by achieved by simply deleting the output files from
@code{unsigned} after the build, but with the risk of forgetting to do so!

Along similar lines (and only in conjunction with @code{--test}, you can use
@code{--force} to force a build of a Disabled application for test purposes,
where normally it would be completely ignored.


@section Direct Installation

You can also build and install directly to a connected device or emulator using
the @code{--install} switch. If you do this without using @code{--package} and
@code{--vercode} then all versions of all packages will be installed (with each
individual version overwriting the previous!). In most cases, this will not be
what you want to do, so execution will stop straight away. However, you can
override this if you're sure that's what you want, by using @code{--all}.


@node Importing Applications
@chapter Importing Applications

To help with starting work on including a new application, @code{fdroid import}
will take a URL and optionally some other parameters, and attempt to construct
as much information as possible by analysing the source code. Basic usage is:

@example
./fdroid import --url=http://address.of.project
@end example

For this to work, the URL must point to a project format that the script
understands. Currently this is limited to one of the following:

@enumerate
@item
Gitorious - @code{https://gitorious.org/PROJECTNAME/REPONAME}
@item
Github - @code{https://github.com/USER/PROJECT}
@item
Google Code - @code{http://code.google.com/p/PROJECT/}
(supports git, svn and hg repos)
@item
Bitbucket - @code{https://bitbucket.org/USER/PROJECT/}
@end enumerate

Depending on the project type, more or less information may be gathered. For
example, the license will be retrieved from a Google Code project, but not a
GitHub one.

If the import is successful, a metadata file will be created. You will need to
edit this further to check the information, and fill in the blanks.

If it fails, you'll be told why. If it got as far as retrieving the source
code, you can inspect it further by looking in @code{tmp/importer} where a full
checkout will exist.

A frequent cause of initial failure is that the project directory is actually
a subdirectory in the repository. In this case, run the importer again using
the @code{--subdir} option to tell it where. It will not attempt to determine
this automatically, since there may be several options.


@node Metadata
@chapter Metadata

@cindex metadata

Information used by update.py to compile the public index comes from two
sources:

@enumerate
@item
the APK files in the repo directory, and
@item
the metadata files in the metadata directory.
@end enumerate

The metadata files are simple, easy to edit text files, always named as the
application's package ID with '.txt' appended.

Note that although the metadata files are designed to be easily read and
writable by humans, they are also processed and written by various scripts.
They are capable of rewriting the entire file when necessary. Even so,
the structure and comments will be preserved correctly, although the order
of fields will be standardised. (In the event that the original file was
in a different order, comments are considered as being attached to the field
following them). In fact, you can standardise all the metadata in a single
command, without changing the functional content, by running:

@example
./fdroid rewritemetadata
@end example

The following sections describe the fields recognised within the file.

@menu
* License::
* Name::
* Web Site::
* Source Code::
* Issue Tracker::
* Donate::
* Summary::
* Description::
* Repo Type::
* Repo::
* Build Version::
* AntiFeatures::
* Disabled::
* Requires Root::
@end menu

@node License
@section License

@cindex license

The license for the application.

Common values: 

@itemize @bullet

@item
@samp{GPLv2}

@item
@samp{GPLv2+}

@item
@samp{GPLv3}

@item
@samp{Apache2}

@item
@samp{MIT}

@item
@samp{BSD}

@end itemize

@node Name
@section Name

@cindex Name

The name of the application. Normally, this field should not be present since the
application's correct name is retrieved from the APK file. However, in a situation
where an APK contains a bad or missing application name, it can be overridden
using this.

@node Web Site
@section Web Site

@cindex Web Site

The URL for the application's web site.

@node Source Code
@section Source Code

@cindex Source Code

The URL to view or obtain the application's source code. This should be
something human-friendly. Machine-readable source-code is covered in the
'Repo' field.

@node Issue Tracker
@section Issue Tracker

@cindex Issue Tracker

The URL for the application's issue tracker. Optional, since not all
applications have one.

@node Donate
@section Donate

@cindex Donate

The URL to donate to the project. This could be the project's donate page
if it has one, or perhaps even a direct PayPal link.

@node Summary
@section Summary

@cindex Summary

A brief summary of what the application is.

@node Description
@section Description

@cindex Description

A full description of the application. This can span multiple lines, and is
terminated by a line containing a single '.'.

@node Repo Type
@section Repo Type

@cindex Repo Type

The type of repository - for automatic building from source. If this is not
specified, automatic building is disabled for this application. Possible
values are:

@itemize @bullet
@item
@samp{git}
@item
@samp{svn}
@item
@samp{git-svn}
@item
@samp{hg}
@item
@samp{bzr}
@end itemize
@node Repo
@section Repo

@cindex Repo

The repository location. Usually a git: or svn: URL, for example.

The git-svn option connects to an SVN repository, and you specify the URL in
exactly the same way, but git is used as a back-end. This is preferable for
performance reasons, and also because a local copy of the entire history is
available in case the upstream repository disappears. (It happens!)

For a Subversion repo that requires authentication, you can precede the repo
URL with username:password@ and those parameters will be passed as @option{--username}
and @option{--password} to the SVN checkout command. (This works only for
plain svn and not for git-svn - one of the very few cases where using svn is
advisable).


@node Build Version
@section Build Version

@cindex Build Version

Any number of these fields can be present, each specifying a version to
automatically build from source. The value is a comma-separated list.
For example:

@samp{Build Version:0.12,3,651696a49be2cd7db5ce6a2fa8185e31f9a20035}

The above specifies to build version 0.12, which has a version code of 3.
The third parameter specifies the tag, commit or revision number from
which to build it in the source repository.

If the commit version starts with a !, that version is not built. Instead,
everything after the ! is used as a reason why it can't be built. The
purpose of this feature is to allow non-buildable releases (e.g. the source
is not published) to be flagged, so the scripts don't generate repeated
messages about them. (And also to record the information for review later).

In addition to the three, always required, parameters described above,
further parameters can be added (in name=value format) to apply further
configuration to the build. These are:

@table @code

@item subdir=<path>
Specifies to build from a subdirectory of the checked out source code.
Normally this directory is changed to before building,

@item bindir=<path>
Normally the build output (apk) is expected to be in the bin
subdirectory below the ant build files. If the project is configured
to put it elsewhere, that can be specified here, relative to the base
of the checked out repo.

@item oldsdkloc=yes
The sdk location in the repo is in an old format, or the build.xml is
expecting such. The 'new' format is sdk.dir while the VERY OLD format
is sdk-location. Typically, if you get a message along the lines of:
"com.android.ant.SetupTask cannot be found" when trying to build, then
try enabling this option.

@item target=<target>
Specifies a particular SDK target, when the source doesn't. This is
likely to cause the whole build.xml to be rewritten, which is fine if
it's a 'standard' android file or doesn't already exist, but not a
good idea if it's heavily customised.

@item rm=<relpath>
Specifies the relative path of file to delete before the build is
done. The path is relative to the base of the build directory - i.e.
the directory that contains AndroidManifest.xml.

@item antcommand=xxx
Specify an alternate ant command (target) instead of the default
'release'.

@item forceversion=yes
If specified, the package version in AndroidManifest.xml is replaced
with the version name for the build as specified in the metadata.

This is useful for cases when upstream repo failed to update it for
specific tag, or to build an arbitrary revision.

@item forcevercode=yes
If specified, the package version code in the AndroidManifest.xml is
replaced with the version code for the build. See also forceversion.

@item update=xxx
By default, 'android update project' is used to generate or update the
build.xml file. Specifying update=no bypasses that.

Specifiying update=force forces rebuilding of the build.xml file at the
same time - this is frequently needed with r14 of the Android platform
tools.

Be aware of any customisations in build.xml when using update=force.

@item initfun=yes
Enables a selection of mad hacks to make com.funambol.android build.
Probably not useful for any other application.

@item buildjni=[no|yes|force]
Enables building of native code via the ndk-build script before doing
the main ant build. The default is @code{no}.

The build and scan processes will complain (refuse to build) if this is set
to @code{no}, but there is a @code{jni} directory present. If the native code
is being built by other means, you can specify @code{manual} here to avoid
that. However, if the native code is actually not required, remove the
directory instead.

@item submodules=yes
Use if the project (git only) has submodules - causes git submodule
init and update to be executed after the source is cloned.

@item encoding=xxxx
Adds a java.encoding property to local.properties with the given
value. Generally the value will be 'utf-8'. This is picked up by the
SDK's ant rules, and forces the Java compiler to interpret source
files with this encoding. If you receive warnings during the compile
about character encodings, you probably need this.

@item prebuild=xxxx
Specifies a shell command (or commands - chain with &&) to run before
the build takes place. Backslash can be used as an escape character to
insert literal commas, or as the last character on a line to join that
line with the next. It has no special meaning in other contexts; in
particular, literal backslashes should not be escaped.

@item init=xxxx
As for 'prebuild', but runs on the source code BEFORE any other processing
takes place.

@item novcheck=yes 
Don't check that the version name and code in the resulting apk are
correct by looking at the build output - assume the metadata is
correct. This takes away a useful level of sanity checking, and should
only be used if the values can't be extracted.

@item fixtrans=yes
Modifies any instances of string resources that use multiple
formatting arguments, but don't use positional notation. For example,
"Hello %s, %d" becomes "Hello %1$s, %2$d". Newer versions of the
Android platform tools enforce this sensible standard. If you get
error messages relating to that, you need to enable this.

@item fixapos=yes
Like fixtrans, but deals with an even older issue relating to
'unescaped apostrophes' in translation strings.

@item maven=yes
Build with maven instead of ant

@item patch=x
Apply patch(es). 'x' names one (or more - comma-seperated)
files within a directory below the metadata, with the same
name as the metadata file but without the extension. Each of
these patches is applied to the code in turn.

@item extlibs=a;b;c
Specifies a list of external libraries (jar files) from the
@code{build/extlib} library, which will be placed in the @code{libs} directory
of the project. Separate items with semicolons.

@item srclibs=a@@r;b@@r1;
Specifies a list of source libraries (kept up to date using version control)
from a predefined set. Separate items with semicolons, and each item is of
the form name@@rev where name is the predefined source library name and rev is
the revision in source control to use. You can then also use $$name$$ in the
prebuild command to substitute the relative path to the library directory.

The available source libraries are current hard-coded in common.py. This will
later be data-driven.

@end table

Another example, using extra parameters:

@samp{Build Version:1.09.03,10903,45,subdir=Timeriffic,oldsdkloc=yes}

@node AntiFeatures
@section AntiFeatures

@cindex AntiFeatures

This is optional - if present, it contains a comma-separated list of any of
the following values, describing an AntiFeature the application has:

@itemize @bullet

@item
@samp{Ads} - the application contains advertising.

@item
@samp{Tracking} - the application tracks and reports your activity to
somewhere without your consent.

@item
@samp{NonFreeNet} - the application promotes a non-Free network service.

@item
@samp{NonFreeAdd} - the application promotes non-Free add-ons.

@item
@samp{NonFreeDep} - the application depends on a non-Free application (e.g.
Google Maps) - i.e. it requires it to be installed on the device, but does not
include it.

@end itemize

@node Disabled
@section Disabled

@cindex Disabled

If this field is present, the application does not get put into the public
index. This allows metadata to be retained while an application is temporarily
disabled from being published. The value should be a description of why the
application is disabled.

@node Requires Root
@section Requires Root

@cindex Requires Root

Set this optional field to "Yes" if the application requires root
privileges to be usable. This lets the client filter it out if the
user so desires.


@node Build Server
@chapter Build Server

The Build Server system isolates the builds for each package within a clean,
isolated and secure throwaway virtual machine environment.

@section Rationale

Building applications in this manner on a large scale, especially with the
involvement of automated and/or unattended processes, could be considered
a dangerous pastime from a security perspective. This is even more the case
when the products of the build are also distributed widely and in a
semi-automated ("you have updates available") fashion.

Assume that an upstream source repository is compromised. A small selection
of things that an attacker could do in such a situation:

@enumerate
@item
Use custom ant build steps to execute virtually anything as the user doing
the build.
@item
Access the keystore.
@item
Modify the built apk files or source tarballs for other applications in the
repository.
@item
Modify the metadata (which includes build scripts, which again, also includes
the ability to execute anything) for other applications in the repository.
@end enumerate

Through complete isolation, the repurcussions are at least limited to the
application in question.

Aside from security issues, there are some applications which have strange
requirements such as custom versions of the NDK. It would be impractical (or
at least extremely messy) to start modifying and restoring the SDK on a
multi-purpose system, but within the confines of a throwaway single-use
virtual machine, anything is possible.

@section Setting up a build server

Integrating the build server setup into the main scripts is a work in progress.
Some things may not work properly yet. Talk to CiaranG if you're trying to use
this and have problems.

In addition to the basic setup sets previously described, you will also need
a Vagrant-compatible Debian Squeeze base box called 'debian6-32'. You can
create one of these for yourself from standard Debian installation media, as
the specification for what's required to be Vagrant-compatible is very well
defined. This is the sensible and secure way to do it, since you know what's
in it. If you insist on taking a shortcut, ask CiaranG for his on the forum
or in IRC.

With this base box installed, you can then do:

@example
./makebuildserver.sh
@end example

This will take a long time - most of it spent installing the necessary parts
of the Android SDK for all the various platforms. Luckily you only need to
do it occasionally.

Once it's complete you'll have a new base box called 'buildserver' which is
what's used for the actual builds. You can then build packages as normal,
but with the addition of the @code{--server} flag to @code{fdroid build} to
instruct it to do all the hard work within the virtual machine, which is
reset to a completely clean state for every package built.


@node GNU Free Documentation License
@appendix GNU Free Documentation License

@include fdl.texi

@node Index
@unnumbered Index

@printindex cp

@bye