From e7a4d0961b7e48ee5c3a770cb495a0bb943464ae Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Fri, 28 May 2021 19:40:29 +0200 Subject: [PATCH] [doc] add blog post about SQL servers Modified merge of [22a79a4] from searx. [22a79a4] https://github.com/searx/searx/commit/22a79a48961d07d8b7c47cebc3f51a57a5ef5cfc Signed-off-by: Markus Heiser --- docs/blog/index.rst | 1 + docs/blog/sql-engines.rst | 160 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 161 insertions(+) create mode 100644 docs/blog/sql-engines.rst diff --git a/docs/blog/index.rst b/docs/blog/index.rst index 8651cef69..a396ecaf6 100644 --- a/docs/blog/index.rst +++ b/docs/blog/index.rst @@ -13,3 +13,4 @@ Blog private-engines command-line-engines search-indexer-engines + sql-engines diff --git a/docs/blog/sql-engines.rst b/docs/blog/sql-engines.rst new file mode 100644 index 000000000..413316054 --- /dev/null +++ b/docs/blog/sql-engines.rst @@ -0,0 +1,160 @@ +=========== +SQL engines +=========== + +.. sidebar:: further read + + - `SQLite `_ + - `PostgreSQL `_ + - `MySQL `_ + +With the *SQL engines* you can bind SQL databases into SearxNG. The following +Relational Database Management System (RDBMS) are supported: + +- :ref:`engine sqlite` +- :ref:`engine postgresql` +- :ref:`engine mysql_server` + +All of the engines above are just commented out in the :origin:`settings.yml +`, as you have to set the required attributes for the +engines, e.g. ``database:`` ... + +.. code:: yaml + + - name: ... + engine: {sqlite|postgresql|mysql_server} + database: ... + result_template: {template_name} + query_str: ... + +By default, the engines use the ``key-value`` template for displaying results / +see :origin:`oscar ` & +:origin:`simple ` +themes. If you are not satisfied with the original result layout, you can use +your own template, set ``result_template`` attribute to ``{template_name}`` and +place the templates at:: + + searx/templates/{theme_name}/result_templates/{template_name} + +As mentioned in previous blog posts, if you do not wish to expose these engines +on a public instance, you can still add them and limit the access by setting +``tokens`` as described in section :ref:`private engines`. + +Configure the engines +===================== + +The configuration of the new database engines are similar. You must put a valid +SQL-SELECT query in ``query_str``. At the moment you can only bind at most one +parameter in your query. By setting the attribute ``limit`` you can define how +many results you want from the SQL server. Basically, it is the same as the +``LIMIT`` keyword in SQL. + +Please, do not include ``LIMIT`` or ``OFFSET`` in your SQL query as the engines +rely on these keywords during paging. If you want to configure the number of +returned results use the option ``limit``. + +.. _engine sqlite: + +SQLite +------ + +.. _MediathekView: https://mediathekview.de/ + +SQLite is a small, fast and reliable SQL database engine. It does not require +any extra dependency. To demonstrate the power of database engines, here is a +more complex example which reads from a MediathekView_ (DE) movie database. For +this example of the SQlite engine download the database: + +- https://liste.mediathekview.de/filmliste-v2.db.bz2 + +and unpack into ``searx/data/filmliste-v2.db``. To search the database use e.g +Query to test: ``!mediathekview concert`` + +.. code:: yaml + + - name: mediathekview + engine: sqlite + disabled: False + categories: general + result_template: default.html + database: searx/data/filmliste-v2.db + query_str: >- + SELECT title || ' (' || time(duration, 'unixepoch') || ')' AS title, + COALESCE( NULLIF(url_video_hd,''), NULLIF(url_video_sd,''), url_video) AS url, + description AS content + FROM film + WHERE title LIKE :wildcard OR description LIKE :wildcard + ORDER BY duration DESC + + +Extra Dependencies +------------------ + +For using :ref:`engine postgresql` or :ref:`engine mysql_server` you need to +install additional packages in Python's Virtual Environment of your SearxNG +instance. To switch into the environment (:ref:`searx-src`) you can use +:ref:`searx.sh`:: + + $ sudo utils/searx.sh shell + (searx-pyenv)$ pip install ... + + +.. _engine postgresql: + +PostgreSQL +---------- + +.. _psycopg2: https://www.psycopg.org/install + +.. sidebar:: requirements + + ``pip install`` psycopg2_ + +PostgreSQL is a powerful and robust open source database. Before configuring +the PostgreSQL engine, you must install the dependency ``psychopg2``. You can +find an example configuration below: + +.. code:: yaml + + - name: my_database + engine: postgresql + database: my_database + username: searx + password: password + query_str: 'SELECT * from my_table WHERE my_column = %(query)s' + +.. _engine mysql_server: + +MySQL +----- + +.. _mysql-connector-python: https://pypi.org/project/mysql-connector-python + +.. sidebar:: requirements + + ``pip install`` mysql-connector-python_ + +MySQL is said to be the most popular open source database. Before enabling MySQL +engine, you must install the package ``mysql-connector-python``. + +The authentication plugin is configurable by setting ``auth_plugin`` in the +attributes. By default it is set to ``caching_sha2_password``. This is an +example configuration for quering a MySQL server: + +.. code:: yaml + + - name: my_database + engine: mysql_server + database: my_database + username: searx + password: password + limit: 5 + query_str: 'SELECT * from my_table WHERE my_column=%(query)s' + + +Acknowledgement +=============== + +This development was sponsored by `Search and Discovery Fund +`_ of `NLnet Foundation `_ . +