doc: describe Makefile targets

With the aim to simplify development cycles, started with PR #1756 a Makefile
based boilerplate was added.  This patch adds the missing developer
documentation.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

Makefile

@@ -8,6 +8,9 @@ PYOBJECTS = searx
 DOC       = docs
 PY_SETUP_EXTRAS ?= \[test\]
 
+PYDIST=./dist/py
+PYBUILD=./build/py
+
 include utils/makefile.include
 include utils/makefile.python
 include utils/makefile.sphinx
@@ -23,6 +26,7 @@ help:
 	@echo  '  install   - developer install (./local)'
 	@echo  '  uninstall - uninstall (./local)'
 	@echo  '  gh-pages  - build docs & deploy on gh-pages branch'
+	@echo  '  clean     - drop builds and environments'
 	@echo  ''
 	@$(MAKE) -s -f utils/makefile.include make-help
 	@echo  ''

docs/conf.py

@@ -32,6 +32,7 @@ extlinks['origin'] = (GIT_URL + '/blob/master/%s', 'git://')
 extlinks['patch'] = (GIT_URL + '/commit/%s', '#')
 extlinks['search'] = (SEARX_URL + '/%s', '#')
 extlinks['docs'] = (DOCS_URL + '/%s', 'docs: ')
+extlinks['pypi'] = ('https://pypi.org/project/%s', 'PyPi: ')
 
 extensions = [
     'sphinx.ext.extlinks',

docs/dev/contribution_guide.rst

@@ -79,6 +79,8 @@ Translation currently takes place on :ref:`transifex <translation>`.
    Please, do not update translation files in the repo.
 
 
+.. _contrib docs:
+
 Documentation
 =============
 
@@ -91,7 +93,7 @@ Documentation
 
 The documentation is built using Sphinx_.  So in order to be able to generate
 the required files, you have to install it on your system.  Much easier, use
-Makefile our targets.
+our :ref:`makefile`.
 
 Here is an example which makes a complete rebuild:
 
@@ -101,6 +103,7 @@ Here is an example which makes a complete rebuild:
    ...
    The HTML pages are in dist/docs.
 
+.. _make docs-live:
 
 live build
 ----------
@@ -110,9 +113,10 @@ live build
    It is recommended to assert a complete rebuild before deploying (use
    ``docs-clean``).
 
-Live build is like WYSIWYG, If you want to edit the documentation, its
+Live build is like WYSIWYG.  If you want to edit the documentation, its
-recommended to use.  The Makefile target ``docs-live`` builds the docs, opens URL
+recommended to use.  The Makefile target ``docs-live`` builds the docs, opens
-in your favorite browser and rebuilds every time a reST file has been changed.
+URL in your favorite browser and rebuilds every time a reST file has been
+changed.
 
 .. code:: sh
 
@@ -123,12 +127,13 @@ in your favorite browser and rebuilds every time a reST file has been changed.
    ... Start watching changes
 
 
+.. _deploy on github.io:
 
 deploy on github.io
 -------------------
 
 To deploy documentation at :docs:`github.io <.>` use Makefile target
-``gh-pages``, which will builds the documentation, clones searx into a sub
+:ref:`make gh-pages`, which will builds the documentation, clones searx into a sub
 folder ``gh-pages``, cleans it, copies the doc build into and runs all the
 needed git add, commit and push:
 

docs/dev/index.rst

@@ -11,3 +11,4 @@ Developer documentation
    search_api
    plugins
    translation
+   makefile

docs/dev/makefile.rst

@@ -0,0 +1,217 @@
+.. _makefile:
+
+================
+Makefile Targets
+================
+
+.. sidebar:: build environment
+
+   Before looking deeper at the targets, first read about :ref:`makefile setup`
+   and :ref:`make pyenv`.
+
+With the aim to simplify development cycles, started with :pull:`1756` a
+``Makefile`` based boilerplate was added.
+
+The usage is simple, just type ``make {target-name}`` to *build* a target.
+Calling the ``help`` target gives a first overview::
+
+  $ make help
+    test      - run developer tests
+    docs      - build documentation
+    docs-live - autobuild HTML documentation while editing
+    run       - run developer instance
+    install   - developer install (./local)
+    uninstall - uninstall (./local)
+    gh-pages  - build docs & deploy on gh-pages branch
+    clean     - drop builds and environments
+    ...
+
+.. contents:: Contents
+   :depth: 2
+   :local:
+   :backlinks: entry
+
+
+.. _makefile setup:
+
+Setup
+=====
+
+.. _git stash: https://git-scm.com/docs/git-stash
+
+The main setup is done in the :origin:`Makefile`::
+
+  export GIT_URL=https://github.com/asciimoo/searx
+  export SEARX_URL=https://searx.me
+  export DOCS_URL=https://asciimoo.github.io/searx
+
+.. sidebar:: fork & upstream
+
+   Commit changes in your (local) branch, fork or whatever, but do not push them
+   upstream / `git stash`_ is your friend.
+
+:GIT_URL: Changes this, to point to your searx fork.
+
+:SEARX_URL: Changes this, to point to your searx instance.
+
+:DOCS_URL: If you host your own (branded) documentation, change this URL.
+
+.. _make pyenv:
+
+Python environment
+==================
+
+.. sidebar:: activate environment
+
+   ``source ./local/py3/bin/activate``
+
+With Makefile we do no longer need to build up the virualenv manually (as
+described in the :ref:`devquickstart` guide).  Jump into your git working tree
+and release a ``make pyenv``:
+
+.. code:: sh
+
+   $ cd ~/searx-clone
+   $ make pyenv
+   PYENV     usage: source ./local/py3/bin/activate
+   ...
+
+With target ``pyenv`` a development environment (aka virtualenv) was build up in
+``./local/py3/``.  To make a *developer install** of searx (:origin:`setup.py`)
+into this environment make target ``install``
+
+.. code:: sh
+
+   $ make install
+   PYENV     usage: source ./local/py3/bin/activate
+   PYENV     using virtualenv from ./local/py3
+   PYENV     install .
+
+You have never to think about intermediate targets like ``pyenv`` or
+``install``, the ``Makefile`` chains them as requisites.  Just run your main
+target.
+
+.. sidebar:: drop environment
+
+   To get rid of the existing environment before re-build use :ref:`clean target
+   <make clean>` first.
+
+If you think, something goes wrong with your ./local environment or you change
+the :origin:`setup.py` file (or the requirements listed in
+:origin:`requirements-dev.txt` and :origin:`requirements.txt`), you have to call
+:ref:`make clean`.
+
+
+.. _make run:
+
+``make run``
+============
+
+To get up a running a developer instance simply call ``make run``.  This enables
+*debug* option in :origin:`searx/settings.yml`, starts ``./searx/webapp.py``
+instance, disables *debug* option and opens the site (xdg-open):
+
+.. code:: sh
+
+  $ make run
+  PYENV     usage: source ./local/py3/bin/activate
+  PYENV     install .
+  ./local/py3/bin/python ./searx/webapp.py
+  ...
+  INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit)
+  ...
+
+.. _make clean:
+
+``make clean``
+==============
+
+Drop all intermediate files, all builds, but keep sources untouched.  Includes
+target ``pyclean`` which drops ./local environment.  Before calling ``make
+clean`` stop all processes using :ref:`make pyenv`.
+
+.. code:: sh
+
+   $ make clean
+   CLEAN     pyclean
+   CLEAN     clean
+
+.. _make docs:
+
+``make docs docs-live docs-clean``
+==================================
+
+We describe the usage of the ``doc*`` targets in the :ref:`How to contribute /
+Documentation <contrib docs>` section.  If you want to edit the documentation
+read our :ref:`make docs-live` section.  If you are working in your own brand,
+adjust your :ref:`Makefile setup <makefile setup>`.
+
+
+.. _make gh-pages:
+
+``make gh-pages``
+=================
+
+To deploy on github.io first adjust your :ref:`Makefile setup <makefile
+setup>`.  For any further read :ref:`deploy on github.io`.
+
+.. _make test:
+
+``make test``
+=============
+
+Runs a series of tests: ``test.pep8``, ``test.unit``, ``test.robot`` and does
+additional :ref:`pylint checks <make pylint>`.  You can run tests selective,
+e.g.:
+
+.. code:: sh
+
+  $ make test.pep8 test.unit
+  . ./local/py3/bin/activate; ./manage.sh pep8_check
+  [!] Running pep8 check
+  . ./local/py3/bin/activate; ./manage.sh unit_tests
+  [!] Running unit tests
+
+.. _make pylint:
+
+``make pylint``
+===============
+
+.. _Pylint: https://www.pylint.org/
+
+Before commiting its recommend to do some (more) linting.  Pylint_ is known as
+one of the best source-code, bug and quality checker for the Python programming
+language.  Pylint_ is not yet a quality gate within our searx project (like
+:ref:`test.pep8 <make test>` it is), but Pylint_ can help to improve code
+quality anyway.  The pylint profile we use at searx project is found in
+project's root folder :origin:`.pylintrc`.
+
+Code quality is a ongoing process.  Don't try to fix all messages from Pylint,
+run Pylint and check if your changed lines are bringing up new messages.  If so,
+fix it.  By this, code quality gets incremental better and if there comes the
+day, the linting is balanced out, we might decide to add Pylint as a quality
+gate.
+
+
+``make pybuild``
+================
+
+.. _PyPi: https://pypi.org/
+.. _twine: https://twine.readthedocs.io/en/latest/
+
+Build Python packages in ``./dist/py``.
+
+.. code:: sh
+
+  $ make pybuild
+  ...
+  BUILD     pybuild
+  running sdist
+  running egg_info
+  ...
+  $ ls  ./dist/py/
+  searx-0.15.0-py3-none-any.whl  searx-0.15.0.tar.gz
+
+To upload packages to PyPi_, there is also a ``upload-pypi`` target.  It needs
+twine_ to be installed.  Since you are not the owner of :pypi:`searx` you will
+never need the latter.

docs/dev/quickstart.rst

@@ -4,15 +4,23 @@
 Development Quickstart
 ======================
 
+.. sidebar:: :ref:`makefile`
+
+   For additional developer purpose there are :ref:`makefile`.
+
 This quickstart guide gets your environment set up with searx.  Furthermore, it
 gives a short introduction to the ``manage.sh`` script.
 
 How to setup your development environment
 =========================================
 
+.. sidebar:: :ref:`make pyenv <make pyenv>`
+
+   Alternatively use the :ref:`make pyenv`.
+
 First, clone the source code of searx to the desired folder.  In this case the
 source is cloned to ``~/myprojects/searx``.  Then create and activate the
-searx-ve virtualenv and install the required packages using manage.sh.
+searx-ve virtualenv and install the required packages using ``manage.sh``.
 
 .. code:: sh
 
@@ -27,6 +35,10 @@ searx-ve virtualenv and install the required packages using manage.sh.
 How to run tests
 ================
 
+.. sidebar:: :ref:`make test.unit <make test>`
+
+   Alternatively use the ``test.pep8``, ``test.unit``, ``test.robot`` targets.
+
 Tests can be run using the ``manage.sh`` script.  Following tests and checks are
 available:
 
@@ -41,7 +53,8 @@ For example unit tests are run with the command below:
 
    ./manage.sh unit_tests
 
-For further test options, please consult the help of the ``manage.sh`` script.
+For further test options, please consult the help of the ``manage.sh`` script or
+read :ref:`make test`.
 
 
 How to compile styles and javascript
@@ -97,6 +110,11 @@ After installing grunt, the files can be built using the following command:
 Tips for debugging/development
 ==============================
 
+.. sidebar:: :ref:`make run`
+
+   Makefile target ``run`` already enables debug option for your developer
+   session / see :ref:`make run`.
+
 Turn on debug logging
   Whether you are working on a new engine or trying to eliminate a bug, it is
   always a good idea to turn on debug logging.  When debug logging is enabled a
@@ -104,6 +122,10 @@ Turn on debug logging
   message. It can be turned on by setting ``debug: False`` to ``debug: True`` in
   :origin:`settings.yml <searx/settings.yml>`.
 
+.. sidebar:: :ref:`make test`
+
+   Alternatively use the :ref:`make test` targets.
+
 Run ``./manage.sh tests`` before creating a PR.
   Failing build on Travis is common because of PEP8 checks.  So a new commit
   must be created containing these format fixes.  This phase can be skipped if