This manual is for the F-Droid repository server tools.
Copyright © 2010, 2011, 2012 Ciaran Gultnieks Copyright © 2011 Henrik Tunedal, Michael Haas, John Sullivan
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".
The F-Droid server tools provide various scripts, tools and data that are used to maintain the main F-Droid application repository. You can use these same tools to create your own additional or alternative repository for publishing, or to assist in creating, testing and submitting metadata to the main repository.
The system requirements for using the tools will vary depending on your intended usage. At the very least, you'll need:
If you intend to build applications from source you'll also need most, if not all, of the following:
If you intend to use the 'Build Server' system, for secure and clean builds (highly recommended), you will also need:
Because the tools and data will always change rapidly, you will almost certainly want to work from a git clone of the tools, which are designed to work in this way, with all associated data in a pre-defined directory structure below the main one. To get started:
git clone git://gitorious.org/f-droid/fdroidserver.git
cd fdroidserver
You will now be in the root directory of the tools. All the tasks associated with managing the repository and data are done from here.
Regardless of the intended usage of the tools, you will always need to set
up some basic configuration details. This is done by creating a file called
config.py which you should do by copying from config.sample.py
and then editing according to the instructions within.
If you want to maintain a simple repository hosting only binary APKs obtained and compiled elsewhere, the process is quite simple:
update.py.
metadata directory and run it again.
update.py with the -c
option. It will create 'skeleton' metadata files that are missing, and you can
then just edit them and fill in the details.
update.py again.
update.py adds an Icons directory into the repo directory, and
also creates the repository index (index.xml, and also index.jar if you've
configured the system to use a signed index).
Following the above process will result in a repo directory, which you
simply need to push to any HTTP (or preferably HTTPS) server to make it
accessible.
While some information about the applications (and versions thereof) is
retrieved directly from the APK files, most comes from the corresponding file
in the metadata directory. The metadata file covering ALL versions of a
particular application is named package.id.txt where package.id is the
unique identifier for that package.
See the Metadata chapter for details of what goes in the metadata file. All fields are relevant for binary APKs, EXCEPT for 'Build Version' entries, which should be omitted.
Instead of (or as well as) including binary APKs from external sources in a repository, you can build them directly from the source code.
Using this method, it is is possible to verify that the application builds correctly, corresponds to the source code, and contains only free software. Unforunately, in the Android world, it seems to be very common for an application supplied as a binary APK to present itself as Free Software when in fact some or all of the following are true:
For this reason, source-built applications are the preferred method for the main F-Droid repository, although occasionally for technical or historical reasons, exceptions are made to this policy.
When building applications from source, it should be noted that you will be signing them (all APK files must be signed to be installable on Android) with your own key. When an application is already installed on a device, it is not possible to upgrade it in place to a new version signed with a different key without first uninstalling the original. This may present an inconvenience to users, as the process of uninstalling loses any data associated with the previous installation.
The process for managing a repository for built-from-source applications is very similar to that described in the Simple Binary Repository chapter, except now you need to:
When run without any parameters, build.py will build any and all versions of
applications that you don't already have in the repo directory (or more
accurately, the unsigned directory. There are various other things you
can do. As with all the tools, the --help option is your friend, but a
few annotated examples and discussion of the more common usage modes follows:
To build a single version of a single application, you could run the following:
./build.py --package=org.fdroid.fdroid --vercode 16
This attempts to build version code 16 (which is version 0.25) of the F-Droid
client. Many of the tools recognise this --package parameter, allowing
their activity to be limited to just a single package.
If the build above was succesful, two files will have been placed in the
unsigned directory:
org.fdroid.fdroid_16.apk
org.fdroid.fdroid_16_src.tar.gz
The first is the (unsigned) APK. You could sign this with a debug key and push it direct to your device or an emulator for testing. The second is a source tarball containing exactly the source that was used to generate the binary.
If you were intending to publish these files, you could then run:
./publish.py
The source tarball would move to the repo directory (which is the
directory you would push to your web server). A signed and zip-aligned version
of the APK would also appear there, and both files would be removed from the
unsigned directory.
If you're building purely for the purposes of testing, and not intending to
push the results to a repository, at least yet, the --test option can be
used to direct output to the tmp directory instead of unsigned.
A similar effect could by achieved by simply deleting the output files from
unsigned after the build, but with the risk of forgetting to do so!
Along similar lines (and only in conjunction with --test, you can use
--force to force a build of a Disabled application for test purposes,
where normally it would be completely ignored.
Information used by update.py to compile the public index comes from two sources:
The metadata files are simple, easy to edit text files, always named as the application's package ID with '.txt' appended.
Note that although the metadata files are designed to be easily read and writable by humans, they are also processed and written by various scripts. They are capable of rewriting the entire file when necessary. Even so, the structure and comments will be preserved correctly, although the order of fields will be standardised. (In the event that the original file was in a different order, comments are considered as being attached to the field following them). In fact, you can standardise all the metadata in a single command, without changing the functional content, by running:
./rewritemetadata.py
The following sections describe the fields recognised within the file.
The license for the application.
Common values:
The name of the application. Normally, this field should not be present since the application's correct name is retrieved from the APK file. However, in a situation where an APK contains a bad or missing application name, it can be overridden using this.
The URL for the application's web site.
The URL to view or obtain the application's source code. This should be something human-friendly. Machine-readable source-code is covered in the 'Repo' field.
The URL for the application's issue tracker. Optional, since not all applications have one.
The URL to donate to the project. This could be the project's donate page if it has one, or perhaps even a direct PayPal link.
A brief summary of what the application is.
A full description of the application. This can span multiple lines, and is terminated by a line containing a single '.'.
The type of repository - for automatic building from source. If this is not specified, automatic building is disabled for this application. Possible values are:
The repository location. Usually a git: or svn: URL, for example.
The git-svn option connects to an SVN repository, and you specify the URL in exactly the same way, but git is used as a back-end. This is preferable for performance reasons, and also because a local copy of the entire history is available in case the upstream repository disappears. (It happens!)
For a Subversion repo that requires authentication, you can precede the repo URL with username:password and those parameters will be passed as --username and --password to the SVN checkout command. (This works only for plain svn and not for git-svn - one of the very few cases where using svn is advisable).
Any number of these fields can be present, each specifying a version to automatically build from source. The value is a comma-separated list. For example:
‘Build Version:0.12,3,651696a49be2cd7db5ce6a2fa8185e31f9a20035’
The above specifies to build version 0.12, which has a version code of 3. The third parameter specifies the tag, commit or revision number from which to build it in the source repository.
If the commit version starts with a !, that version is not built. Instead, everything after the ! is used as a reason why it can't be built. The purpose of this feature is to allow non-buildable releases (e.g. the source is not published) to be flagged, so the scripts don't generate repeated messages about them. (And also to record the information for review later).
In addition to the three, always required, parameters described above, further parameters can be added (in name=value format) to apply further configuration to the build. These are:
subdir=<path>bindir=<path>oldsdkloc=yestarget=<target>rm=<relpath>antcommand=xxxforceversion=yesThis is useful for cases when upstream repo failed to update it for
specific tag, or to build an arbitrary revision.
forcevercode=yesupdate=xxxSpecifiying update=force forces rebuilding of the build.xml file at the same time - this is frequently needed with r14 of the Android platform tools.
Be aware of any customisations in build.xml when using update=force.
initfun=yesbuildjni=[no|yes|force]no.
The build and scan processes will complain (refuse to build) if this is set
to no, but there is a jni directory present. If the native code
is being built by other means, you can specify manual here to avoid
that. However, if the native code is actually not required, remove the
directory instead.
submodules=yesencoding=xxxxprebuild=xxxxinit=xxxxnovcheck=yesfixtrans=yesfixapos=yesmaven=yespatch=xextlibs=a;b;cbuild/extlib library, which will be placed in the libs directory
of the project. Separate items with semicolons.
srclibs=a@r;b@r1;The available source libraries are current hard-coded in common.py. This will later be data-driven.
Another example, using extra parameters:
‘Build Version:1.09.03,10903,45,subdir=Timeriffic,oldsdkloc=yes’
This is optional - if present, it contains a comma-separated list of any of the following values, describing an AntiFeature the application has:
If this field is present, the application does not get put into the public index. This allows metadata to be retained while an application is temporarily disabled from being published. The value should be a description of why the application is disabled.
Set this optional field to "Yes" if the application requires root privileges to be usable. This lets the client filter it out if the user so desires.
The Build Server system isolates the builds for each package within a clean, isolated and secure throwaway virtual machine environment.
Building applications in this manner on a large scale, especially with the involvement of automated and/or unattended processes, could be considered a dangerous pastime from a security perspective. This is even more the case when the products of the build are also distributed widely and in a semi-automated ("you have updates available") fashion.
Assume that an upstream source repository is compromised. A small selection of things that an attacker could do in such a situation:
Through complete isolation, the repurcussions are at least limited to the application in question.
Aside from security issues, there are some applications which have strange requirements such as custom versions of the NDK. It would be impractical (or at least extremely messy) to start modifying and restoring the SDK on a multi-purpose system, but within the confines of a throwaway single-use virtual machine, anything is possible.
Integrating the build server setup into the main scripts is a work in progress. Some things may not work properly yet. Talk to CiaranG if you're trying to use this and have problems.
In addition to the basic setup sets previously described, you will also need a Vagrant-compatible Debian Squeeze base box called 'debian6-32'. You can create one of these for yourself from standard Debian installation media, as the specification for what's required to be Vagrant-compatible is very well defined. This is the sensible and secure way to do it, since you know what's in it. If you insist on taking a shortcut, ask CiaranG for his on the forum or in IRC.
With this base box installed, you can then do:
./makebuildserver.sh
This will take a long time - most of it spent installing the necessary parts of the Android SDK for all the various platforms. Luckily you only need to do it occasionally.
Once it's complete you'll have a new base box called 'buildserver' which is
what's used for the actual builds. You can then build packages as normal,
but with the addition of the --server flag to build.py to
instruct it to do all the hard work within the virtual machine, which is
reset to a completely clean state for every package built.
Copyright © 2000, 2001, 2002, 2007, 2008 Free Software Foundation, Inc.
http://fsf.org/
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
The purpose of this License is to make a manual, textbook, or other functional and useful document free in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.
This License is a kind of “copyleft”, which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.
We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.
This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The “Document”, below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as “you”. You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.
A “Modified Version” of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.
A “Secondary Section” is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document's overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.
The “Invariant Sections” are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.
The “Cover Texts” are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.
A “Transparent” copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not “Transparent” is called “Opaque”.
Examples of suitable formats for Transparent copies include plain ascii without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.
The “Title Page” means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, “Title Page” means the text near the most prominent appearance of the work's title, preceding the beginning of the body of the text.
The “publisher” means any person or entity that distributes copies of the Document to the public.
A section “Entitled XYZ” means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as “Acknowledgements”, “Dedications”, “Endorsements”, or “History”.) To “Preserve the Title” of such a section when you modify the Document means that it remains a section “Entitled XYZ” according to this definition.
The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.
You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.
You may also lend copies, under the same conditions stated above, and you may publicly display copies.
If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document's license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.
If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.
If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.
It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.
You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:
If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's license notice. These titles must be distinct from any other section titles.
You may add a section Entitled “Endorsements”, provided it contains nothing but endorsements of your Modified Version by various parties—for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.
You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.
The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.
You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.
The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.
In the combination, you must combine any sections Entitled “History” in the various original documents, forming one section Entitled “History”; likewise combine any sections Entitled “Acknowledgements”, and any sections Entitled “Dedications”. You must delete all sections Entitled “Endorsements.”
You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.
You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.
A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an “aggregate” if the copyright resulting from the compilation is not used to limit the legal rights of the compilation's users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.
If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document's Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.
Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.
If a section in the Document is Entitled “Acknowledgements”, “Dedications”, or “History”, the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.
You may not copy, modify, sublicense, or distribute the Document except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense, or distribute it is void, and will automatically terminate your rights under this License.
However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice.
Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, receipt of a copy of some or all of the same material does not give you any rights to use it.
The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.
Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License “or any later version” applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation. If the Document specifies that a proxy can decide which future versions of this License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Document.
“Massive Multiauthor Collaboration Site” (or “MMC Site”) means any World Wide Web server that publishes copyrightable works and also provides prominent facilities for anybody to edit those works. A public wiki that anybody can edit is an example of such a server. A “Massive Multiauthor Collaboration” (or “MMC”) contained in the site means any set of copyrightable works thus published on the MMC site.
“CC-BY-SA” means the Creative Commons Attribution-Share Alike 3.0 license published by Creative Commons Corporation, a not-for-profit corporation with a principal place of business in San Francisco, California, as well as future copyleft versions of that license published by that same organization.
“Incorporate” means to publish or republish a Document, in whole or in part, as part of another Document.
An MMC is “eligible for relicensing” if it is licensed under this License, and if all works that were first published under this License somewhere other than this MMC, and subsequently incorporated in whole or in part into the MMC, (1) had no cover texts or invariant sections, and (2) were thus incorporated prior to November 1, 2008.
The operator of an MMC Site may republish an MMC contained in the site under CC-BY-SA on the same site at any time before August 1, 2009, provided the MMC is eligible for relicensing.