LTS distros like Ubuntu 18.04 do not ship a up-to-date version of git.::
$ sudo -H ./utils/lxc.sh cmd searx-ubu1804 git --version
...
git version 2.17.1
The option `--show-current` was added in git v2.22, the alternative to this
option is::
git rev-parse --abbrev-ref HEAD
Issue when using option `--show-current`::
[searx-ubu1804] Clone searx sources
[searx-ubu1804] -------------------
[searx-ubu1804]
[searx-ubu1804] error: unknown option `show-current'
[searx-ubu1804] usage: git branch [<options>] [-r | -a] [--merged | --no-merged]
....
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
EOL of Ubuntu 16.04 (Xenial Xerus) is in April 2021 but we do not support 16.04
since a longer time. Issues are comming from dependencies (e.g. pip)::
[searx-ubu1604] |searx| SyntaxError: invalid syntax
[searx-ubu1604] |searx| Traceback (most recent call last):
[searx-ubu1604] |searx| File "/usr/local/searx/searx-pyenv/bin/pip", line 7, in <module>
[searx-ubu1604] |searx| from pip._internal.cli.main import main
[searx-ubu1604] |searx| File "/usr/local/searx/searx-pyenv/lib/python3.5/site-packages/pip/_internal/cli/main.py", line 60
[searx-ubu1604] |searx| sys.stderr.write(f"ERROR: {exc}")
[searx-ubu1604] |searx| ^
[searx-ubu1604] |searx| SyntaxError: invalid syntax
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
use a sparql request on wikidata to get the list of currencies.
currencies.json contains the translation for all supported searx languages.
Supersede #993
Target pip-exe is a prerequisite of the targets:
- pyinstall
- pyuninstall
and was accidentally deleted in commit 9b48ae47.
HINT:
do not confuse pyinstall with penvinstall
pyinstall & pyuninstall
Installing into user's HOME using pip from OS,
therefore the message is needed.
pyenvinstall & pyenvuninstall
Installing into virtualenv (./local) using pip which is provided by
prerequisite 'pyenv' in the virtualenv.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1. This patch fixes error:
rm -rf gh-pages/
make V=1 gh-pages
make[1]: Leaving directory '/800GBPCIex4/share/searx'
[ -d "gh-pages/.git" ] || git clone gh-pages
fatal: repository 'gh-pages' does not exist
2. The gh-page build has been moved to ./build/gh-pages this also affects
'travis-gh-pages'
3. The gh-pages commit messages now includes a ref to the repository and commit
4. Since a gh-pages history has only the drawback that the reposetory grows
fast, this patch also flattens the history:
cd build/gh-pages/; git log --oneline
bash: cd: build/gh-pages/: Datei oder Verzeichnis nicht gefunden
026126be (HEAD -> gh-pages, origin/gh-pages) make gh-pages: from https://github.com/return42/searx.git@71d66979c2935312e0aed7fc7c3cf6199fbe88a2
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
$ make test.sh
In utils/lxc.sh line 42:
ubu2010_boilerplate="$ubu1904_boilerplate"
^-----------------^ SC2034: ubu2010_boilerplate appears unused. Verify use (or export if used externally).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The fabfile.py has not been updated since 5 years. I also asked [1] if someone
still use Fabric wtihout any response. Lets drop outdated Fabric file.
[1] https://github.com/searx/searx/discussions/2400
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
pip install -e .
...
Obtaining file:///usr/local/searx/searx-src
ERROR: Command errored out with exit status 1:
command: /usr/local/searx/searx-pyenv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/usr/local/searx/searx-src/setup.py'"'"'; __file__='"'"'/usr/local/searx/searx-src/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-vzer91m2
cwd: /usr/local/searx/searx-src/
Complete output (9 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/searx/searx-src/setup.py", line 10, in <module>
from searx.version import VERSION_STRING
File "/usr/local/searx/searx-src/searx/__init__.py", line 19, in <module>
import searx.settings_loader
File "/usr/local/searx/searx-src/searx/settings_loader.py", line 8, in <module>
import yaml
ModuleNotFoundError: No module named 'yaml'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Removes module searx/brand.py and creates a namespace at searx.brand.
This patch is a first 'proof of concept'. Later we can decide to remove the
brand namespace entirely or not.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Ubuntu 20.10 image [1] is avialable from https://images.linuxcontainers.org
BTW: replace builhost package 'ttf-dejavu' by more common meta package
'fonts-dejavu' ('ttf-dejavu' is no longer available in 20.10).
[1] d161de601b
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Official support for Ubuntu 19.10 'Eoan Ermine' ended on July 17, 2020.
The image has been dropped [1] from https://images.linuxcontainers.org
[1] d161de601b
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Without this commit the module searx checks the secret_key value.
With this commit, make docs, utils/standalone_searx.py,
utils/fetch_firefox_version.py works without SEARX_DEBUG=1
For reference see https://github.com/searx/searx/pull/2386
Create a (PY_ENV)/requirements.sha256 file with sha256 sums of the
./requirements*.txt files and reinstall only if the sha256 sums change.
Suggested-by: @dalf / https://github.com/searx/searx/pull/2418#pullrequestreview-558426579
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Install boilerplate packages (pip, wheel, setuptools and requirements.txt) only
once, when virtualenv is created.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In commit c2a6f145 the sphinx-doc and sphinx-live targets has been removed,
since Sphinx is already installed by the pyenvinstall target.
This commit removes the targets from the remaining prerequisites of the 'BOOK'
targets
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Makefile targets 'pyenv-install' and 'pyenv-uninstall' are unused duplicates of
'pyenvinstall' and 'pyenvuninstall'.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch keeps an eye on:
1. When installing searx, the branch to install needs to be a local branch.
2. The branch to install can be different from the current branch.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In commit a70b9b9f the SEARX_SETTINGS_TEMPLATE environment was added to the
.config file, but was not use in utils/searx.sh.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Installation and use of the command 'virtualenv' was only needed in py2 and py2
is no longer suported by searx. In py3 the command is replaced by 'python -m
venv'.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The path to the makefile.lxc is not always static and
'/share/searx/utils/makefile.lxc' can be wrong.
When target 'lxc.activate' is called within the LXC container the path to
makefile.lxc depends on LXC_SHARE_FOLDER and the basename of the folder where
searx was cloned inital on the HOST system.
This patch determines the path of 'makefile.lxc' next to 'makefile.include'
itself.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
see searx.search.processors.abstract.EngineProcessor
First the method searx call the get_params method.
If the return value is not None, then the searx call the method search.
Since #2291 is merged, it is recommend to use::
use_default_settings=True
1. Add a template file use_default_settings.yml::
SEARX_SETTINGS_TEMPLATE="${REPO_ROOT}/utils/templates/etc/searx/use_default_settings.yml"
2. In Chapter "Configuration" recommend to make use of
'use_default_settings=True' and describe it
3. Rewrite of docs/admin/settings.rst
- move chapter 'settings.yml location' to the top
- update and split chapter 'Global Settings'
4. Add environment SEARX_SETTINGS_TEMPLATE to .config.sh
5. Use environment $SEARX_SETTINGS_TEMPLATE in the utils/searx.sh script
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* Update call to sphinx-autobuild (remove --poll -B options, add --open-browser)
* Upgrade the Sphinx and sphinx-tabs versions.
* utils/makefile.sphinx: remove the SPHINXVERS variable.
* utils/makefile.sphinx: remove the sphinx-doc and sphinx-live targets, Sphinx is already installed by the pyenvinstall target.
Since there is no centos-7 documentation 'searx.sh doc' produce an error
message::
ERROR: centos-: uWSGI not yet implemented
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
## What does this PR do?
To test centos-7 on non centos systems add centos-7 image to the LXC suite.
## Why is this change important?
PR #2112 adds centos-7 support to the install scripts
## How to test this PR locally?
`sudo ./utils/lxc.sh build searx-centos7`
Python's default encoding depends on the platform, set (python) default encoding
UTF-8 in uwsgi ini files:
LANG=C.UTF-8
LANGUAGE=C.UTF-8
LC_ALL=C.UTF-8
Error pattern:
Traceback (most recent call last):
File "/usr/local/searx/searx-src/searx/webapp.py", line 74, in <module>
from searx.search import SearchWithPlugins, get_search_query_from_webapp
File "/usr/local/searx/searx-src/searx/search.py", line 32, in <module>
from searx.external_bang import get_bang_url
File "/usr/local/searx/searx-src/searx/external_bang.py", line 13, in <module>
for bang in json.load(json_file)['bang']:
File "/usr/lib/python3.8/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/usr/lib/python3.8/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 31341: ordinal not in range(128)
close: https://github.com/asciimoo/searx/issues/2041
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The sphinx-doc-prebuilds is the single point of definition for prerequisites
each doc build needs
BTW: remove $(LXC_ENV_FOLDER) offset from BOOKS_FOLDER and DOCS_FOLDER path.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In the past we did not really install searx into the virtualenv. This is a
problem, since entry-points and other python installation stuff is not trigger.
See discussion: https://github.com/asciimoo/searx/pull/1938#issuecomment-632056508
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Even it is a core component, some hoster do not have pre installed the
'inetutils' package. We do need this package for the 'hostname' command.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
A command to set new morty key in morty & searx service (settings.yml) and to
restart both services with new key::
utils/morty.sh option new-key
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This is an addittion to PR #1934:
The .0 change really counts on some engines like Bing which seems to allow
Windows NT 10.0 but not Windows NT 10.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- add installation method for nginx sites, morty and filtron
- clean up PUBLIC_URL environment in and outside of containers
- clean up comand lines
- handle uWSGI quirks on fedora (emperor mode)
- handle Python quirks on debian (there is no 'python' command anymore)
- lib.sh: add die and die_caller functions
- lxc_suite_install_info is now a function
- lint: shellcheck
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This is the revision of the documentation about the varous nginx installation
variants. It also implements the nginx installation scripts for morty and
filtron.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In utils/lib.sh there are two functions with the same name, but different tasks.
Rename one of them from lxc_init_container() into lxc_init_container_env().
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Commit 09a40625 adds virtualenv dependency. BTW remove deprecated
--no-site-packages. Not having access to global site-packages is now the
default behavior.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
revert "makefile.python: ... drop build support py2" to get back Py2 support.
TPy2 support need virtualenv installed by the OS.
BTW: log environment and python version in travis's install phase
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
From py3.3 on a subset of virtualenv is built-in, so you can run '-m venv' ot of
the box.
- replace: $(PY_ENV_BIN)/pip --> $(PY_ENV_BIN)/python -m pip
- remove obsolete virtualenv-exe target and adjust VTENV_OPTS
- remove obsolete msg-pip-exe target
- print list of py launchers available from $(PY_ENV_BIN) to the log
- fix hard coded ./local
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- no more need for a .config.mk
- docs: use searx.brands environment
- searx.sh, filtron.sh & morty.sh are sourcing utils/brand.env
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To Makefile target brand, add creation of bash environment in::
utils/brand.env
In bash scripts (manage.sh) source env by::
. utils/brand.env
manage.sh help: show GIT_URL and more environment
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
checked with::
sudo ./utils/lxc.sh cmd -- make test.sh
on ubu1804, ubu1910, fedora31 and archlinux. All checks OK except:
ubu1604: uses shellcheck v0.3.7 (from 04/2015) which is no longer supported!
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To get meaningfull diffs, the json file has to be sorted. Before applying any
further content patch, the json file needs a inital sort (without changing any
content).
Sorted by::
import sys, json
with open('engines_languages.json') as f:
j = json.load(f)
with open('engines_languages.json', 'w') as f:
json.dump(j, f, indent=2, sort_keys=True)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
By isolating the environment of makefiles and bash scripts into .config.mk and
.config.sh it is simple to maintain searx brands by setting some central
environments.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Add *Makefile* boilerplate useful for python projects. All python tasks are
using a virtualenv from ./local/py3
$ make help
run - run developer instance
install - developer install (./local)
uninstall - uninstall (./local)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Instead of a single line with 500000 characters use nicely formatted JSON.
Sort the lists in engine_languages.py so when updating it is possible to
more easily see the differences (search engines do change the order their
languages are listed in)
Add match_language function in utils to match any user given
language code with a list of engine's supported languages.
Also add language_aliases dict on each engine to translate
standard language codes into the custom codes used by the engine.
