Since [PR 2600] is merged the update scripts for languages and useragent has
been moved to folder:
searx_extra/update/
[PR 2600] https://github.com/searx/searx/pull/2600
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Many things have been changed since last review of this engine. This patch fix
xpath selectors, implements suggestion and is a complete review / rewrite of the
engine.
Signed-off-by: Markus Heiser <markus@darmarit.de>
The old xpath configuration for google scholar did not work and is replaced by a
python implementation.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
aka: ensure that 'make test' works as expected
The cache contains a copy './local' which is - under some circumstance -
corrupted. It is not possible to clear the cache [1] (see the top of the page).
Ensure that 'make test' works as expected [2] even if
- the python interpreter is missing
- the virtualenv exists but pyyaml is missing
To hardening when the workflow cache fails, this patch adds the new target
'travis.test' into the workflow. This target probes to import a python module
'yaml'. If this fails the virtualenv will be completely new build.
[1] https://github.com/actions/cache/issues/2#issuecomment-673493515
[2] https://github.com/searx/searx/pull/2517#discussion_r567240235
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To check all engines:
make search.checker
To check a engine 'google news' replace space by underline:
make search.checker.google_news
To see HTTP requests and more use SEARX_DEBUG:
make SEARX_DEBUG=1 search.checker.google_news
To filter out HTTP redirects:
make SEARX_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]"
...
Engine google news Checking
https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
--
https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
--
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This revise is based on the methods developed in the revise of the google engine
(see commit 410c2f9).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
This revise is based on the methods developed in the revise of the google engine
(see commit 410c2f9).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
There are 3 invocations:
* one with the default .pylintrc file
* one for searx/engines with some disabled checks and some additional-builtins
* one for the all the code except the engines with some disabled checks
Devian's request and response forms has been changed.
- fixed title
- fixed time_range_dict to 'popular-*-***'
- use image from <noscript> if exists
- drop obsolete "http to https, remove domain sharding"
- use query URL https://www.deviantart.com/search/deviations?page=5&q=foo
- add searx/engines/deviantart.py to pylint check (test.pylint)
Error pattern::
There DEBUG:searx:result: invalid title: {'url': 'https://www.deviantart.com/ ...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Reloading webapp.py when a file is changed exit with error::
$ make run
....
INFO:werkzeug: * Detected change in 'searx/searx/engines/deviantart.py', reloading
INFO:werkzeug: * Restarting with stat
ERROR:searx:server.secret_key is not changed. Please use something else instead of ultrasecretkey.
make: *** [Makefile:69: run] Error 1
Patch suggested by @dalf [1]
[1] https://github.com/searx/searx/issues/2278#issuecomment-719732579
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* Update call to sphinx-autobuild (remove --poll -B options, add --open-browser)
* Upgrade the Sphinx and sphinx-tabs versions.
* utils/makefile.sphinx: remove the SPHINXVERS variable.
* utils/makefile.sphinx: remove the sphinx-doc and sphinx-live targets, Sphinx is already installed by the pyenvinstall target.
The sphinx-doc-prebuilds is the single point of definition for prerequisites
each doc build needs
BTW: remove $(LXC_ENV_FOLDER) offset from BOOKS_FOLDER and DOCS_FOLDER path.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
fix travis build error [1]::
The command "local/py3/bin/pip install codecov" failed and exited with 127
Use the correct pip (python environment) from build environment::
$(PY_ENV_BIN)/python -m pip
[1] https://travis-ci.org/github/asciimoo/searx/jobs/669701405#L590
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
And add 'buildenv' as an first order prerequisite to the main targets:
- install
- run
- docs
- docs-live
- project
- node.env
- docker
- test
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- no more need for a .config.mk
- docs: use searx.brands environment
- searx.sh, filtron.sh & morty.sh are sourcing utils/brand.env
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To Makefile target brand, add creation of bash environment in::
utils/brand.env
In bash scripts (manage.sh) source env by::
. utils/brand.env
manage.sh help: show GIT_URL and more environment
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
style.legacy could be renamed themes.legacy : it actually builds the files for
the legacy theme, then themes.legacy can be a dependency of themes. Same for
the other styles.*
Debatable: about style.bootstrap, same convention : theme.bootstrap (even it is
more a toolbox for the oscar theme).
So there is no need to add the missing make styles in the help target.
thanks @dalf:
- https://github.com/asciimoo/searx/pull/1900#discussion_r399160355
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To build all styles use:
make styles
To build individual styles use one of:
make style.legacy
make style.courgette
make style.pixart
make style.bootstrap
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Update searx/data/useragents.json with the most recent versions of Firefox.
BTW: add 'useragents.update' to 'project' target and clean up the Makefile and
remove it from the manage.sh script.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
closes: https://github.com/asciimoo/searx/issues/1882
A *brand* of searx is a fork which might have its own design and some special
functions which might bee reasonable in a special context.
In this sense, the fork might have its own documentation but not its own issue
tracker. The *upstream* of a brand is always https://github.com/asciimoo from
where the brand-fork pulls the master branch regularly. A fork which has its
own issue tracker is a spin-off and out of the scope of the searx project
itself. The conclusion is:
- hard code ISSUE_URL (in the Makefile)
- always refer to DOCS_URL
- links in the about page refer to the *upstream* (searx project)
except DOCS_URL
- "fork me on github" ribbons refer to the *upstream*
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
We have some variables in the build environment which are also needed in the
grunt process when building themes. Theses variables are relavant if one
creates a fork with its own branding. We treat these variables under the term
'brands'.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
We have some variables in the build environment which are also needed in the
templating process. Theses variables are relavant if one creates a fork with
its own branding. We treat these variables under the term 'brands'.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
checked with::
sudo ./utils/lxc.sh cmd -- make test.sh
on ubu1804, ubu1910, fedora31 and archlinux. All checks OK except:
ubu1604: uses shellcheck v0.3.7 (from 04/2015) which is no longer supported!
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
To get meaningfull diffs, the json file has to be sorted. Before applying any
further content patch, the json file needs a inital sort (without changing any
content).
Sorted by::
import sys, json
with open('engines_languages.json') as f:
j = json.load(f)
with open('engines_languages.json', 'w') as f:
json.dump(j, f, indent=2, sort_keys=True)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
By isolating the environment of makefiles and bash scripts into .config.mk and
.config.sh it is simple to maintain searx brands by setting some central
environments.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
With the aim to simplify development cycles, started with PR #1756 a Makefile
based boilerplate was added. This patch adds the missing developer
documentation.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
- add sphinx extensions
- patch documentation to make use of
These modules help to simplify the reST markup of external references. BTW it
helps to write more readable reST and form custom brands.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Add *Makefile* boilerplate useful for python projects. All python tasks are
using a virtualenv from ./local/py3
$ make help
run - run developer instance
install - developer install (./local)
uninstall - uninstall (./local)
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
