In 2a12637 an ASCII escape code was added, such escape codes should not be
written to pipes (when the output is not a terminal).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Some piped commands hide the cursor, show cursory when the stream ends.
Most often this is a bug of the command which piped. The command should not
hide the cursor when it writes to a pipe. I have seen this bug with the package
manager (pacman) from ArchLinux.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The functions:
- static.build.commit
- static.build.commit.drop
- static.build.restore
are imported into the ./manage script. To avoid name collisions some variables
and fucntions has been renamed by adding the prefix *static_*.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Marginal simplification of the procedure to get only the last local commit which
is not in remotes.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- add script to shellcheck,
- fix error messages from shellcheck and
- moved global variables to local variables (lower case)
No functional change!
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The idea is to avoid conflict when there are differents branches with changes are made on the static files.
A solution is to ask the administrators to build the files from the sources, but it requires to install the npm dependencies.
So the solution in this commit keep the sources and the build files in the same git repository.
In one branch, the modification of the source (*.less, *.js) are commited without the built files.
The built files are commited in a uniq commit, with a commit message "Static build"
In case of merge or rebase, this commit can be dropped.
New make targets:
* static.build.commit.drop:
drop the last "Static build".
The command checks that there are only build files in the commit.
* static.build.commit :
call "make static.build.commit.drop"
call "make themes.all"
commit the files
* static.git.restore.staged:
git restore --staged <build files>
* static.git.restore:
git restore <build files>
Related to https://github.com/searxng/searxng/issues/137
Docker is blocking network of existing LXC containers / there is a conflict in
the iptables setup of Docker & LXC. With this patch:
- utils/lxc.sh checks internet connectivity (instead of silently hang)
- Chapter "Internet Connectivity & Docker" describes the problem and made a
suggestion for a solution a solution
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In 5a7b12ee we changed the signature of the YAML settings, this patch takes this
into account.
Related-to: 5a7b12ee [yamllint] searx/settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The node.env build environment require npm, ttfautohint and fontforge installed
in the OS. These tools can be installed by::
sudo -H ./utils/searx.sh install buildhost
If one of the tools is not installed, the script node.env stops with a
appropriate message.
BTW: We ignore CentOS-7 as developer & build platform
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This file is generated by webfont.
* It is now generated as searx/static/themes/simple/ion.less
* It is generated before the .less compilation.
* .gitignore includes this file
Add two new package depedencies: fontforge ttfautohint
See utils/searx.sh
The public URL of the searx instance: PUBLIC_URL="https://mydomain.xy/searx"
The default is SEARX_URL which is taken from ./utils/brand.env.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Access to formats can be denied by settings configuration::
search:
formats: [html, csv, json, rss]
Closes: https://github.com/searxng/searxng/issues/95
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To select a different file with filtron rules, set environment
FILTRON_RULES_TEMPLATE
the default is
utils/templates/etc/filtron/rules.json
The installation is done by the new function install_rules() which offers a
multiple choice in case of collisions (known from searx.sh install setup).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Implement a boilerplate to manage performance optimized virtualenv builds.
Shell scripts can use (e.g.) 'pyenv.cmd' to execute command in the virtualenv
without having to worry about whether and how the environment is provided. ::
pyenv.cmd which python
..../local/py3/bin/python
pyenv.cmd which pip
..../local/py3/bin/pip
If pyenv.cmd released multiple times the installation will only rebuild if the
function 'pyenv.OK' fails. Function 'pyenv.OK' make some test to validate that
the virtualenv exists and works as expected. The check also fails if
requirements listed requirements-dev.txt and requirements.txt has been edited.
Among these tests 'pyenv.OK' calls 'pyenv.check' which implements a python
script that validate the python installation. Here is an example how a
'pyenv.check' implementation could look like::
pyenv.check() {
cat <<EOF
import yaml
print('import yaml --> OK')
EOF
}
Signed-off-by: Markus Heiser <markus@darmarit.de>
Newer CentOS-7 images from https://images.linuxcontainers.org do no longer
include the which command.
Issue:
$ sudo -H ./utils/lxc.sh cmd searx-centos7 ./utils/filtron.sh install all
INFO: [searx-centos7] ./utils/filtron.sh install all
...
Install Go in user's HOME
-------------------------
download and install go binary ..
...
-bash: line 1: which: command not found
-->|ERROR - Go Installation not found in PATH!?!
-bash: line 2: which: command not found
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The wrong and unnecessary `pip install .` is executed in /usr/local/searx and is
responsible for the error message:
ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /usr/local/searx
The correct pip-install comes right after changing to `cd ${SEARX_SRC}`.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
LTS distros like Ubuntu 18.04 do not ship a up-to-date version of git.::
$ sudo -H ./utils/lxc.sh cmd searx-ubu1804 git --version
...
git version 2.17.1
The option `--show-current` was added in git v2.22, the alternative to this
option is::
git rev-parse --abbrev-ref HEAD
Issue when using option `--show-current`::
[searx-ubu1804] Clone searx sources
[searx-ubu1804] -------------------
[searx-ubu1804]
[searx-ubu1804] error: unknown option `show-current'
[searx-ubu1804] usage: git branch [<options>] [-r | -a] [--merged | --no-merged]
....
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
EOL of Ubuntu 16.04 (Xenial Xerus) is in April 2021 but we do not support 16.04
since a longer time. Issues are comming from dependencies (e.g. pip)::
[searx-ubu1604] |searx| SyntaxError: invalid syntax
[searx-ubu1604] |searx| Traceback (most recent call last):
[searx-ubu1604] |searx| File "/usr/local/searx/searx-pyenv/bin/pip", line 7, in <module>
[searx-ubu1604] |searx| from pip._internal.cli.main import main
[searx-ubu1604] |searx| File "/usr/local/searx/searx-pyenv/lib/python3.5/site-packages/pip/_internal/cli/main.py", line 60
[searx-ubu1604] |searx| sys.stderr.write(f"ERROR: {exc}")
[searx-ubu1604] |searx| ^
[searx-ubu1604] |searx| SyntaxError: invalid syntax
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
use a sparql request on wikidata to get the list of currencies.
currencies.json contains the translation for all supported searx languages.
Supersede #993
Target pip-exe is a prerequisite of the targets:
- pyinstall
- pyuninstall
and was accidentally deleted in commit 9b48ae47.
HINT:
do not confuse pyinstall with penvinstall
pyinstall & pyuninstall
Installing into user's HOME using pip from OS,
therefore the message is needed.
pyenvinstall & pyenvuninstall
Installing into virtualenv (./local) using pip which is provided by
prerequisite 'pyenv' in the virtualenv.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
1. This patch fixes error:
rm -rf gh-pages/
make V=1 gh-pages
make[1]: Leaving directory '/800GBPCIex4/share/searx'
[ -d "gh-pages/.git" ] || git clone gh-pages
fatal: repository 'gh-pages' does not exist
2. The gh-page build has been moved to ./build/gh-pages this also affects
'travis-gh-pages'
3. The gh-pages commit messages now includes a ref to the repository and commit
4. Since a gh-pages history has only the drawback that the reposetory grows
fast, this patch also flattens the history:
cd build/gh-pages/; git log --oneline
bash: cd: build/gh-pages/: Datei oder Verzeichnis nicht gefunden
026126be (HEAD -> gh-pages, origin/gh-pages) make gh-pages: from https://github.com/return42/searx.git@71d66979c2935312e0aed7fc7c3cf6199fbe88a2
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
$ make test.sh
In utils/lxc.sh line 42:
ubu2010_boilerplate="$ubu1904_boilerplate"
^-----------------^ SC2034: ubu2010_boilerplate appears unused. Verify use (or export if used externally).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The fabfile.py has not been updated since 5 years. I also asked [1] if someone
still use Fabric wtihout any response. Lets drop outdated Fabric file.
[1] https://github.com/searx/searx/discussions/2400
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
pip install -e .
...
Obtaining file:///usr/local/searx/searx-src
ERROR: Command errored out with exit status 1:
command: /usr/local/searx/searx-pyenv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/usr/local/searx/searx-src/setup.py'"'"'; __file__='"'"'/usr/local/searx/searx-src/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-vzer91m2
cwd: /usr/local/searx/searx-src/
Complete output (9 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/searx/searx-src/setup.py", line 10, in <module>
from searx.version import VERSION_STRING
File "/usr/local/searx/searx-src/searx/__init__.py", line 19, in <module>
import searx.settings_loader
File "/usr/local/searx/searx-src/searx/settings_loader.py", line 8, in <module>
import yaml
ModuleNotFoundError: No module named 'yaml'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Removes module searx/brand.py and creates a namespace at searx.brand.
This patch is a first 'proof of concept'. Later we can decide to remove the
brand namespace entirely or not.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Ubuntu 20.10 image [1] is avialable from https://images.linuxcontainers.org
BTW: replace builhost package 'ttf-dejavu' by more common meta package
'fonts-dejavu' ('ttf-dejavu' is no longer available in 20.10).
[1] d161de601b
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Official support for Ubuntu 19.10 'Eoan Ermine' ended on July 17, 2020.
The image has been dropped [1] from https://images.linuxcontainers.org
[1] d161de601b
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Without this commit the module searx checks the secret_key value.
With this commit, make docs, utils/standalone_searx.py,
utils/fetch_firefox_version.py works without SEARX_DEBUG=1
For reference see https://github.com/searx/searx/pull/2386
Create a (PY_ENV)/requirements.sha256 file with sha256 sums of the
./requirements*.txt files and reinstall only if the sha256 sums change.
Suggested-by: @dalf / https://github.com/searx/searx/pull/2418#pullrequestreview-558426579
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Install boilerplate packages (pip, wheel, setuptools and requirements.txt) only
once, when virtualenv is created.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In commit c2a6f145 the sphinx-doc and sphinx-live targets has been removed,
since Sphinx is already installed by the pyenvinstall target.
This commit removes the targets from the remaining prerequisites of the 'BOOK'
targets
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Makefile targets 'pyenv-install' and 'pyenv-uninstall' are unused duplicates of
'pyenvinstall' and 'pyenvuninstall'.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This patch keeps an eye on:
1. When installing searx, the branch to install needs to be a local branch.
2. The branch to install can be different from the current branch.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
In commit a70b9b9f the SEARX_SETTINGS_TEMPLATE environment was added to the
.config file, but was not use in utils/searx.sh.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Installation and use of the command 'virtualenv' was only needed in py2 and py2
is no longer suported by searx. In py3 the command is replaced by 'python -m
venv'.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The path to the makefile.lxc is not always static and
'/share/searx/utils/makefile.lxc' can be wrong.
When target 'lxc.activate' is called within the LXC container the path to
makefile.lxc depends on LXC_SHARE_FOLDER and the basename of the folder where
searx was cloned inital on the HOST system.
This patch determines the path of 'makefile.lxc' next to 'makefile.include'
itself.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
see searx.search.processors.abstract.EngineProcessor
First the method searx call the get_params method.
If the return value is not None, then the searx call the method search.
Since #2291 is merged, it is recommend to use::
use_default_settings=True
1. Add a template file use_default_settings.yml::
SEARX_SETTINGS_TEMPLATE="${REPO_ROOT}/utils/templates/etc/searx/use_default_settings.yml"
2. In Chapter "Configuration" recommend to make use of
'use_default_settings=True' and describe it
3. Rewrite of docs/admin/settings.rst
- move chapter 'settings.yml location' to the top
- update and split chapter 'Global Settings'
4. Add environment SEARX_SETTINGS_TEMPLATE to .config.sh
5. Use environment $SEARX_SETTINGS_TEMPLATE in the utils/searx.sh script
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
