mirror of
https://github.com/searxng/searxng.git
synced 2024-11-14 00:30:15 +01:00
[enh] settings.yml: add use_default_settings option
This change is backward compatible with the existing configurations. If a settings.yml loaded from an user defined location (SEARX_SETTINGS_PATH or /etc/searx/settings.yml), then this settings can relied on the default settings.yml with this option: user_default_settings:True
This commit is contained in:
parent
6ada5bac60
commit
1cfe7f2a75
17
Makefile
17
Makefile
@ -266,4 +266,19 @@ test.clean:
|
||||
travis.codecov:
|
||||
$(Q)$(PY_ENV_BIN)/python -m pip install codecov
|
||||
|
||||
.PHONY: $(PHONY)
|
||||
|
||||
# user-settings
|
||||
# -------------
|
||||
|
||||
PHONY += user-settings.create user-settings.update
|
||||
|
||||
user-settings.update: pyenvinstall
|
||||
$(Q)$(PY_ENV_ACT); pip install ruamel.yaml
|
||||
$(Q)$(PY_ENV_ACT); python utils/update_user_settings.py ${SEARX_SETTINGS_PATH}
|
||||
|
||||
user-settings.update.engines: pyenvinstall
|
||||
$(Q)$(PY_ENV_ACT); pip install ruamel.yaml
|
||||
$(Q)$(PY_ENV_ACT); python utils/update_user_settings.py --add-engines ${SEARX_SETTINGS_PATH}
|
||||
|
||||
|
||||
.PHONY: $(PHONY)
|
@ -206,3 +206,97 @@ Engine settings
|
||||
|
||||
A few more options are possible, but they are pretty specific to some
|
||||
engines, and so won't be described here.
|
||||
|
||||
|
||||
.. _settings location:
|
||||
|
||||
settings.yml location
|
||||
=====================
|
||||
|
||||
First, searx will try to load settings.yml from these locations:
|
||||
|
||||
1. the full path specified in the ``SEARX_SETTINGS_PATH`` environment variable.
|
||||
2. ``/etc/searx/settings.yml``
|
||||
|
||||
If these files don't exist (or are empty or can't be read), searx uses the :origin:`searx/settings.yml` file.
|
||||
|
||||
.. _ settings use_default_settings:
|
||||
|
||||
use_default_settings
|
||||
====================
|
||||
|
||||
.. note::
|
||||
|
||||
If searx is cloned from a git repository, most probably there is no need to have an user settings.
|
||||
|
||||
The user defined settings.yml can relied on the default configuration :origin:`searx/settings.yml` using ``use_default_settings: True``.
|
||||
|
||||
In the following example, the actual settings are the default settings defined in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and the ``bind_address``:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
use_default_settings: true
|
||||
server:
|
||||
secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
|
||||
server:
|
||||
bind_address: "0.0.0.0"
|
||||
|
||||
With ``use_default_settings: True``, each settings can be override in a similar way with one exception, the ``engines`` section:
|
||||
|
||||
* If the ``engines`` section is not defined in the user settings, searx uses the engines from the default setttings (the above example).
|
||||
* If the ``engines`` section is defined then:
|
||||
|
||||
* searx loads only the engines declare in the user setttings.
|
||||
* searx merges the configuration according to the engine name.
|
||||
|
||||
In the following example, only three engines are available. Each engine configuration is merged with the default configuration.
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
use_default_settings: true
|
||||
server:
|
||||
secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
|
||||
engines:
|
||||
- name: wikipedia
|
||||
- name: wikidata
|
||||
- name: ddg definitions
|
||||
|
||||
Another example where four engines are available. The arch linux wiki engine has a :ref:`token<private engines>`.
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
use_default_settings: true
|
||||
server:
|
||||
secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
|
||||
engines:
|
||||
- name: arch linux wiki
|
||||
tokens: ['$ecretValue']
|
||||
- name: wikipedia
|
||||
- name: wikidata
|
||||
- name: ddg definitions
|
||||
|
||||
automatic update
|
||||
----------------
|
||||
|
||||
The following comand creates or updates a minimal user settings (a secret key is defined if it is not already the case):
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
make SEARX_SETTINGS_PATH=/etc/searx/settings.yml user-settings.update
|
||||
|
||||
Set ``SEARX_SETTINGS_PATH`` to your user settings path.
|
||||
|
||||
As soon the user settings contains an ``engines`` section, it becomes difficult to keep the engine list updated.
|
||||
The following command creates or updates the user settings including the ``engines`` section:
|
||||
|
||||
.. code-block:: sh
|
||||
|
||||
make SEARX_SETTINGS_PATH=/etc/searx/settings.yml user-settings.update.engines
|
||||
|
||||
After that ``/etc/searx/settings.yml``
|
||||
|
||||
* has a ``secret key``
|
||||
* has a ``engine`` section if it is not already the case, moreover the command:
|
||||
|
||||
* has deleted engines that do not exist in the default settings.
|
||||
* has added engines that exist in the default settings but are not declare in the user settings.
|
||||
|
@ -7,6 +7,8 @@ enabled engines on their instances. It might be because they do not want to
|
||||
expose some private information through an offline engine. Or they
|
||||
would rather share engines only with their trusted friends or colleagues.
|
||||
|
||||
.. _private engines:
|
||||
|
||||
Private engines
|
||||
===============
|
||||
|
||||
|
@ -16,39 +16,15 @@ along with searx. If not, see < http://www.gnu.org/licenses/ >.
|
||||
'''
|
||||
|
||||
import logging
|
||||
import searx.settings
|
||||
from os import environ
|
||||
from os.path import realpath, dirname, join, abspath, isfile
|
||||
from io import open
|
||||
from yaml import safe_load
|
||||
|
||||
|
||||
searx_dir = abspath(dirname(__file__))
|
||||
engine_dir = dirname(realpath(__file__))
|
||||
static_path = abspath(join(dirname(__file__), 'static'))
|
||||
|
||||
|
||||
def check_settings_yml(file_name):
|
||||
if isfile(file_name):
|
||||
return file_name
|
||||
else:
|
||||
return None
|
||||
|
||||
|
||||
# find location of settings.yml
|
||||
if 'SEARX_SETTINGS_PATH' in environ:
|
||||
# if possible set path to settings using the
|
||||
# enviroment variable SEARX_SETTINGS_PATH
|
||||
settings_path = check_settings_yml(environ['SEARX_SETTINGS_PATH'])
|
||||
else:
|
||||
# if not, get it from searx code base or last solution from /etc/searx
|
||||
settings_path = check_settings_yml(join(searx_dir, 'settings.yml')) or check_settings_yml('/etc/searx/settings.yml')
|
||||
|
||||
if not settings_path:
|
||||
raise Exception('settings.yml not found')
|
||||
|
||||
# load settings
|
||||
with open(settings_path, 'r', encoding='utf-8') as settings_yaml:
|
||||
settings = safe_load(settings_yaml)
|
||||
settings, settings_load_message = searx.settings.load_settings()
|
||||
|
||||
if settings['ui']['static_path']:
|
||||
static_path = settings['ui']['static_path']
|
||||
@ -58,7 +34,6 @@ enable debug if
|
||||
the environnement variable SEARX_DEBUG is 1 or true
|
||||
(whatever the value in settings.yml)
|
||||
or general.debug=True in settings.yml
|
||||
|
||||
disable debug if
|
||||
the environnement variable SEARX_DEBUG is 0 or false
|
||||
(whatever the value in settings.yml)
|
||||
@ -78,7 +53,7 @@ else:
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
|
||||
logger = logging.getLogger('searx')
|
||||
logger.debug('read configuration from %s', settings_path)
|
||||
logger.info(settings_load_message)
|
||||
logger.info('Initialisation done')
|
||||
|
||||
if 'SEARX_SECRET' in environ:
|
||||
|
@ -31,3 +31,11 @@ class SearxParameterException(SearxException):
|
||||
self.message = message
|
||||
self.parameter_name = name
|
||||
self.parameter_value = value
|
||||
|
||||
|
||||
class SearxSettingsException(SearxException):
|
||||
|
||||
def __init__(self, message, filename):
|
||||
super().__init__(message)
|
||||
self.message = message
|
||||
self.filename = filename
|
||||
|
91
searx/settings.py
Normal file
91
searx/settings.py
Normal file
@ -0,0 +1,91 @@
|
||||
import collections.abc
|
||||
|
||||
import yaml
|
||||
from searx.exceptions import SearxSettingsException
|
||||
from os import environ
|
||||
from os.path import dirname, join, abspath, isfile
|
||||
|
||||
|
||||
searx_dir = abspath(dirname(__file__))
|
||||
|
||||
|
||||
def check_settings_yml(file_name):
|
||||
if isfile(file_name):
|
||||
return file_name
|
||||
else:
|
||||
return None
|
||||
|
||||
|
||||
def load_yaml(file_name):
|
||||
try:
|
||||
with open(file_name, 'r', encoding='utf-8') as settings_yaml:
|
||||
settings = yaml.safe_load(settings_yaml)
|
||||
if not isinstance(settings, dict) or len(settings) == 0:
|
||||
raise SearxSettingsException('Empty file', file_name)
|
||||
return settings
|
||||
except IOError as e:
|
||||
raise SearxSettingsException(e, file_name)
|
||||
except yaml.YAMLError as e:
|
||||
raise SearxSettingsException(e, file_name)
|
||||
|
||||
|
||||
def get_default_settings_path():
|
||||
return check_settings_yml(join(searx_dir, 'settings.yml'))
|
||||
|
||||
|
||||
def get_user_settings_path():
|
||||
# find location of settings.yml
|
||||
if 'SEARX_SETTINGS_PATH' in environ:
|
||||
# if possible set path to settings using the
|
||||
# enviroment variable SEARX_SETTINGS_PATH
|
||||
return check_settings_yml(environ['SEARX_SETTINGS_PATH'])
|
||||
else:
|
||||
# if not, get it from searx code base or last solution from /etc/searx
|
||||
return check_settings_yml('/etc/searx/settings.yml')
|
||||
|
||||
|
||||
def update_dict(d, u):
|
||||
for k, v in u.items():
|
||||
if isinstance(v, collections.abc.Mapping):
|
||||
d[k] = update_dict(d.get(k, {}), v)
|
||||
else:
|
||||
d[k] = v
|
||||
return d
|
||||
|
||||
|
||||
def update_settings(default_settings, user_settings):
|
||||
for k, v in user_settings.items():
|
||||
if k == 'use_default_settings':
|
||||
continue
|
||||
elif k == 'engines':
|
||||
default_engines = default_settings[k]
|
||||
default_engines_dict = dict((definition['name'], definition) for definition in default_engines)
|
||||
default_settings[k] = [update_dict(default_engines_dict[definition['name']], definition)
|
||||
for definition in v]
|
||||
else:
|
||||
update_dict(default_settings[k], v)
|
||||
|
||||
return default_settings
|
||||
|
||||
|
||||
def load_settings(load_user_setttings=True):
|
||||
default_settings_path = get_default_settings_path()
|
||||
user_settings_path = get_user_settings_path()
|
||||
if user_settings_path is None or not load_user_setttings:
|
||||
# no user settings
|
||||
return (load_yaml(default_settings_path),
|
||||
'load the default settings from {}'.format(default_settings_path))
|
||||
|
||||
# user settings
|
||||
user_settings = load_yaml(user_settings_path)
|
||||
if user_settings.get('use_default_settings'):
|
||||
# the user settings are merged with the default configuration
|
||||
default_settings = load_yaml(default_settings_path)
|
||||
update_settings(default_settings, user_settings)
|
||||
return (default_settings,
|
||||
'merge the default settings ( {} ) and the user setttings ( {} )'
|
||||
.format(default_settings_path, user_settings_path))
|
||||
|
||||
# the user settings, fully replace the default configuration
|
||||
return (user_settings,
|
||||
'load the user settings from {}'.format(user_settings_path))
|
98
utils/update_user_settings.py
Normal file
98
utils/update_user_settings.py
Normal file
@ -0,0 +1,98 @@
|
||||
#!/usr/bin/env python
|
||||
|
||||
# set path
|
||||
from sys import path
|
||||
from os.path import realpath, dirname, join
|
||||
path.append(realpath(dirname(realpath(__file__)) + '/../'))
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
import string
|
||||
import ruamel.yaml
|
||||
import secrets
|
||||
import collections
|
||||
from ruamel.yaml.scalarstring import SingleQuotedScalarString, DoubleQuotedScalarString
|
||||
from searx.settings import load_settings, check_settings_yml, get_default_settings_path
|
||||
from searx.exceptions import SearxSettingsException
|
||||
|
||||
|
||||
RANDOM_STRING_LETTERS = string.ascii_lowercase + string.digits + string.ascii_uppercase
|
||||
|
||||
|
||||
def get_random_string():
|
||||
r = [secrets.choice(RANDOM_STRING_LETTERS) for _ in range(64)]
|
||||
return ''.join(r)
|
||||
|
||||
|
||||
def main(prog_arg):
|
||||
yaml = ruamel.yaml.YAML()
|
||||
yaml.preserve_quotes = True
|
||||
yaml.indent(mapping=4, sequence=1, offset=2)
|
||||
user_settings_path = prog_args.get('user-settings-yaml')
|
||||
|
||||
try:
|
||||
default_settings, _ = load_settings(False)
|
||||
if check_settings_yml(user_settings_path):
|
||||
with open(user_settings_path, 'r', encoding='utf-8') as f:
|
||||
user_settings = yaml.load(f.read())
|
||||
new_user_settings = False
|
||||
else:
|
||||
user_settings = yaml.load('use_default_settings: True')
|
||||
new_user_settings = True
|
||||
except SearxSettingsException as e:
|
||||
sys.stderr.write(str(e))
|
||||
return
|
||||
|
||||
if not new_user_settings and not user_settings.get('use_default_settings'):
|
||||
sys.stderr.write('settings.yml already exists and use_default_settings is not True')
|
||||
return
|
||||
|
||||
user_settings['use_default_settings'] = True
|
||||
use_default_settings_comment = "settings based on " + get_default_settings_path()
|
||||
user_settings.yaml_add_eol_comment(use_default_settings_comment, 'use_default_settings')
|
||||
|
||||
if user_settings.get('server', {}).get('secret_key') in [None, 'ultrasecretkey']:
|
||||
user_settings.setdefault('server', {})['secret_key'] = DoubleQuotedScalarString(get_random_string())
|
||||
|
||||
user_engines = user_settings.get('engines')
|
||||
if user_engines:
|
||||
has_user_engines = True
|
||||
user_engines_dict = dict((definition['name'], definition) for definition in user_engines)
|
||||
else:
|
||||
has_user_engines = False
|
||||
user_engines_dict = {}
|
||||
user_engines = []
|
||||
|
||||
# remove old engines
|
||||
if prog_arg.get('add-engines') or has_user_engines:
|
||||
default_engines_dict = dict((definition['name'], definition) for definition in default_settings['engines'])
|
||||
for i, engine in enumerate(user_engines):
|
||||
if engine['name'] not in default_engines_dict:
|
||||
del user_engines[i]
|
||||
|
||||
# add new engines
|
||||
if prog_arg.get('add-engines'):
|
||||
for engine in default_settings.get('engines', {}):
|
||||
if engine['name'] not in user_engines_dict:
|
||||
user_engines.append({'name': engine['name']})
|
||||
user_settings['engines'] = user_engines
|
||||
|
||||
# output
|
||||
if prog_arg.get('dry-run'):
|
||||
yaml.dump(user_settings, sys.stdout)
|
||||
else:
|
||||
with open(user_settings_path, 'w', encoding='utf-8') as f:
|
||||
yaml.dump(user_settings, f)
|
||||
|
||||
|
||||
def parse_args():
|
||||
parser = argparse.ArgumentParser(description='Update user settings.yml')
|
||||
parser.add_argument('--add-engines', dest='add-engines', default=False, action='store_true', help='Add new engines')
|
||||
parser.add_argument('--dry-run', dest='dry-run', default=False, action='store_true', help='Dry run')
|
||||
parser.add_argument('user-settings-yaml', type=str)
|
||||
return vars(parser.parse_args())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
prog_args = parse_args()
|
||||
main(prog_args)
|
Loading…
Reference in New Issue
Block a user