kamp/searxng
1
0
Fork 0
mirror of https://github.com/searxng/searxng.git synced 2024-11-05 21:00:13 +01:00
Code
4,351 Commits 7 Branches 17 Tags 128 MiB
703f8c4a8b
Commit Graph

2842 Commits

Author SHA1 Message Date
Markus Heiser
8cd544b2a6 [doc] add documentation about the XPath engine 
- pylint searx/engines/xpath.py
- fix indentation of some long lines
- add logging
- add doc-strings

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-23 11:48:21 +02:00
Markus Heiser
7728e25b11 [refactor] metrics.get_reliabilities() - make code more readable 
- init stat values by None
- drop round_or_none
- don't try to get percentage if base is 'None'

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-22 15:17:18 +02:00
Alexandre Flament
99aaf86b73 [fix] offline engines: fix templates /stats and /preferences 2021-05-22 15:17:18 +02:00
Alexandre Flament
3014463fed [fix] metrics: processing time = total time if there is no http time 
It was previsouly None

Fix /stats
 2021-05-22 15:17:18 +02:00
Alexandre Flament
ec83493538 [fix] offline engine: don't crash on time recording 2021-05-22 15:17:18 +02:00
Markus Heiser
6f1446d55f [pylint] searx/search/__init__.py & replace lic-text by SPDX tag 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-21 17:31:22 +02:00
Alexandre Flament
426fadccb3 [mod] remove gc.collect() after each user request 2021-05-21 17:23:18 +02:00
Alexandre Flament
d1cffc55cb [fix] unit test: don't load /etc/searx/settings.yml 
Add a new environment variable SEARX_DISABLE_ETC_SETTINGS
to disable loading of /etc/searx/settings.yml

unit tests:
* set SEARX_DISABLE_ETC_SETTINGS to 1
* remove SEARX_SETTINGS_PATH if it exists
 2021-05-18 17:23:21 +02:00
Alexandre Flament
d255e5637b
Merge pull request #81 from return42/csp-bar-graph 
[fix] make /stats more CSP compliant
 2021-05-18 08:55:56 +02:00
Markus Heiser
73d73dcf9b [static] make /stats more CSP compliant- make themes.all 
Based on commits

- 0507e185 [fix] bar graph and rename CSS class engine-scores -> engine-score
- 3e9ad7ae [fix] make /stats more CSP compliant - github issue form
- 34859d0e [fix] make /stats more CSP compliant - oscar theme
- 0a6c4884 [fix] make /stats more CSP compliant - simple theme
- cdfb4b7f [fix] make /stats more CSP compliant - bar graph
- 965817f2 [fix] simple theme - generate missing sourceMap file

this patch is generated by::

     make themes.all

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-17 21:25:32 +02:00
Markus Heiser
0507e185a5 [fix] bar graph and rename CSS class engine-scores -> engine-score 
- drop #main_stats selector in stats.less
- 'engine-score' exists before this PR.
- untabify searx/static/themes/__common__/less/stats.less

for details see comment at: d93bec7638..1204e4f07e (r633571496)

Suggested-by: @dalf in commit 1204e4f0
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-17 21:24:19 +02:00
Markus Heiser
3e9ad7ae0c [fix] make /stats more CSP compliant - github issue form 
Hide textarea from github issue form::

    ./__common__/new_issue.html:6:    <textarea name="body" style="display: none;">{{- '' -}}

BTW: fix indentation.

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-17 21:23:27 +02:00
Markus Heiser
34859d0e58 [fix] make /stats more CSP compliant - oscar theme 
Replace oscar theme's *styles* (see below) by CSP compliant implementation in
``searx/static/themes/__common__/less/stats.less`` ::

    ./oscar/stats.html:29:  <th scope="col" style="width:20rem;">{{ th_sort('name', _("Engine name")) }}</th>
    ./oscar/stats.html:30:  <th scope="col" style="width:7rem; text-align: right;">{{ th_sort('score', _('Scores')) }}</th>
    ./oscar/stats.html:33:  <th scope="col" style="text-align: right;">{{ th_sort('reliability', _('Reliability')) }}</th>
    ./oscar/stats.html:38:  <td style="text-align: right;">
    ./oscar/stats.html:91:  <td style="text-align: right;"> {{ engine_reliabilities.get(engine_stat.name, {}).get('reliablity') }}</td>
    ./oscar/stats.html:109: <tbody style="padding-top: 1rem;">
    ./oscar/stats.html:112: <th scope="row" style="width: 10rem">{{ _('Exception') }}</th><td>{{ error.exception_classname }}</td>
    ./oscar/stats.html:114: <th scope="row" style="width: 10rem">{{ _('Message') }}</th><td>{{ error.log_message }}</td>
    ./oscar/stats.html:116: <th scope="row" style="width: 10rem">{{ _('Percentage') }}</th><td style="width: 10rem">{{ error.percentage }}</td>
    ./oscar/stats.html:121: <span style="border-right: 1px solid gray; padding: 0 1rem 0 0; margin: 0 0 0 0.5rem;">{{ param }}</span>
    ./oscar/stats.html:138: <th scope="col" style="width: 10rem">{{ _('Failed test') }}</th>

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-17 17:41:22 +02:00
Markus Heiser
0a6c488417 [fix] make /stats more CSP compliant - simple theme 
Replace simple theme's *styles* (see below) by CSP compliant implementation in
``searx/static/themes/simple/less/stats.less`` ::

    ./simple/stats.html:26:  <table style="max-width: 1280px; margin: 0 auto 0 0;">
    ./simple/stats.html:28:  <th scope="col" style="width:20rem;">{{ th_sort('name', _("Engine name")) }}</th>
    ./simple/stats.html:29:  <th scope="col" style="width:7rem; text-align: right;">{{ th_sort('score', _('Scores')) }}</th>
    ./simple/stats.html:32:  <th scope="col" style="text-align: right;">{{ th_sort('reliability', _('Reliability')) }}</th>
    ./simple/stats.html:37:  <td style="text-align: right;">
    ./simple/stats.html:90:  <td style="text-align: right;"> {{ engine_reliabilities.get(engine_stat.name, {}).get('reliablity') }}</td>
    ./simple/stats.html:106: <table style="max-width: 1280px; margin: 1rem; border: 1px solid gray;">
    ./simple/stats.html:107: <tbody style="padding-top: 1rem;">
    ./simple/stats.html:110: <th scope="row" style="width: 10rem">{{ _('Exception') }}</th><td>{{ error.exception_classname }}</td>
    ./simple/stats.html:112: <th scope="row" style="width: 10rem">{{ _('Message') }}</th><td>{{ error.log_message }}</td>
    ./simple/stats.html:114: <th scope="row" style="width: 10rem">{{ _('Percentage') }}</th><td style="width: 10rem">{{ error.percentage }}</td>
    ./simple/stats.html:119: <span style="border-right: 1px solid gray; padding: 0 1rem 0 0; margin: 0 0 0 0.5rem;">{{ param }}</span>
    ./simple/stats.html:136: <th scope="col" style="width: 10rem">{{ _('Failed test') }}</th>

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-17 17:41:22 +02:00
Markus Heiser
cdfb4b7ff9 [fix] make /stats more CSP compliant - bar graph 
Replace bar graph's *styles* (see below) by CSP compliant implementation in
``searx/static/themes/__common__/less/stats.less`` ::

    ./simple/stats.html:49: <span style="width: calc(max(2px, 100%*{{ (engine_stat.result_count / engine_stats.max_result_count )|round(3) }}))" class="stacked-bar-chart-serie1"></span>{{- "" -}}
    ./simple/stats.html:57: <span style="width: calc(max(2px, 100%*{{ (engine_stat.http / engine_stats.max_time )|round(3) }}))" class="stacked-bar-chart-serie1"></span>{{- "" -}}
    ./simple/stats.html:58: <span style="width: calc(100%*{{ engine_stat.processing / engine_stats.max_time |round(3) }})" class="stacked-bar-chart-serie2"></span>{{- "" -}}
    ./oscar/stats.html:50:  <span style="width: calc(max(2px, 100%*{{ (engine_stat.result_count / engine_stats.max_result_count )|round(3) }}))" class="stacked-bar-chart-serie1"></span>{{- "" -}}
    ./oscar/stats.html:58:  <span style="width: calc(max(2px, 100%*{{ (engine_stat.http / engine_stats.max_time )|round(3) }}))" class="stacked-bar-chart-serie1"></span>{{- "" -}}
    ./oscar/stats.html:59:  <span style="width: calc(100%*{{ engine_stat.processing / engine_stats.max_time |round(3) }})" class="stacked-bar-chart-serie2"></span>{{- "" -}}

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-17 17:41:22 +02:00
Markus Heiser
965817f294 [fix] simple theme - generate missing sourceMap file 
C&P from searx/static/themes/oscar/gruntfile.js

Suggested-by: @dalf in commit 1204e4f0
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-17 17:41:22 +02:00
Markus Heiser
93594a7b42 [less] update grunt-contrib-less v3.3.0 
Upgraded [v3.3.0] otherwise::

`  width: calc(100% - 5rem);`

becomes `width: 95%` once compiled by less version 1.4.1.

[v3.3.0] https://github.com/gruntjs/grunt-contrib-less/releases/tag/v3.0.0

Suggested-by: @dalf in commit 1204e4f0
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-17 17:41:22 +02:00
Markus Heiser
57cdf180fe [fix] engine archive is - search_url has been changed 
BTW: set soft_max_redirects from *archive is* by 1, to prevent logging::

    DEBUG:httpx._client:HTTP Request: GET https://archive.is/search/?q=www.python.org "HTTP/2 302 Found"
    DEBUG:httpx._client:HTTP Request: GET https://archive.is/www.python.org "HTTP/2 200 OK"
    DEBUG:searx:archive is: ErrorContext('searx/search/processors/online.py', 110, 'count_error(self.engine_name,', None, '1 redirects, maximum: 0', ('200', 'OK', 'archive.is')) True

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-17 15:21:53 +02:00
Markus Heiser
ffcebf5e12 [enh] xpath engine - add request parameter 'soft_max_redirects' 
Make 'soft_max_redirects' configurable per Xpath engine::

    - name : <engine-name>
      engine : xpath
      soft_max_redirects: 1
      ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-17 15:04:55 +02:00
Alexandre Flament
90fe232e08 [fix] webapp.py: pylint: disable=assigning-non-slot for request.* 
Note: actually, the code should use flask.g.* instead of request.*
 2021-05-14 11:15:35 +02:00
Alexandre Flament
d12033e166 [mod] Server-Timing header: add render field 
Add a "render" field to the Server-Timing header
This field is the time spent in the searx.webapp.render function (Jinja2 rendering)

See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Server-Timing
 2021-05-06 09:39:52 +02:00
Alexandre Flament
0f4e995ab4 [mod] searx.network.client: the same configuration reuses the same ssl.SSLContext 
before there was one ssl.SSLContext per client.

see https://github.com/encode/httpx/issues/978
 2021-05-05 20:36:37 +02:00
Markus Heiser
fa0d05c313 [pylint] checker/__main__.py & checker/background.py 
Lint files that has been touched by [PR #58]

[PR #58] https://github.com/searxng/searxng/pull/58

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-05-05 16:47:02 +02:00
Alexandre Flament
8c1a65d32f [mod] multithreading only in searx.search.* packages 
it prepares the new architecture change,
everything about multithreading in moved in the searx.search.* packages

previously the call to the "init" function of the engines was done in searx.engines:
* the network was not set (request not sent using the defined proxy)
* it requires to monkey patch the code to avoid HTTP requests during the tests
 2021-05-05 13:12:42 +02:00
Markus Heiser
d36adfa59f
Merge pull request #55 from searxng/upd-relialibility-column 
[themes] /preferences, reliablity: warning icons & error logs link
 2021-05-04 12:20:45 +00:00
Markus Heiser
5253df946b
[mod] option to enable or disable "proxy" button next to each result (#54) 
* [mod] option to enable or disable "proxy" button next to each result

Closes: https://github.com/searxng/searxng/issues/51
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Co-authored-by: Alexandre Flament <alex@al-f.net>
 2021-05-04 11:57:14 +02:00
Alexandre Flament
e83862c22f [themes] /preferences, reliablity: warning icons & error logs link 
When there is at least one errors or one failed checker test:
* the warning icon is displayed in the reliability column
* the link "View error logs and submit a bug report" is displayed on engine name tooltip.

Before:
* the warning icon was displayed only when one or more checker test(s) failed.
* the link "View error logs and submit a bug report" was not shown when a checker test failed but there were no error.
 2021-05-04 10:50:03 +02:00
Markus Heiser
45293270e2
Merge pull request #44 from searxng/dependabot/pip/master/pylint-2.8.2 
Bump pylint from 2.7.4 to 2.8.2
 2021-05-03 13:58:14 +00:00
dependabot[bot]
2a53f79a2d Bump pylint from 2.7.4 to 2.8.2 
Bumps [pylint](https://github.com/PyCQA/pylint) from 2.7.4 to 2.8.2.
- [Release notes](https://github.com/PyCQA/pylint/releases)
- [Changelog](https://github.com/PyCQA/pylint/blob/master/ChangeLog)
- [Commits](https://github.com/PyCQA/pylint/compare/pylint-2.7.4...v2.8.2)

Signed-off-by: dependabot[bot] <support@github.com>
 2021-05-03 15:45:30 +02:00
Marc Abonce Seguin
448bfe6005 fix Qwant's fetch_languages function 2021-05-02 17:46:40 -07:00
Alexandre Flament
2756e170ce
Merge pull request #49 from searxng/update_data_update_ahmia_blacklist.py 
Update searx.data - update_ahmia_blacklist.py
 2021-05-01 10:28:32 +02:00
Alexandre Flament
a3a7be90ff
Merge pull request #50 from searxng/update_data_update_currencies.py 
Update searx.data - update_currencies.py
 2021-05-01 10:28:22 +02:00
Alexandre Flament
4558c0c027
Merge pull request #48 from searxng/update_data_update_wikidata_units.py 
Update searx.data - update_wikidata_units.py
 2021-05-01 10:28:11 +02:00
dalf
dd8a5a00dc Update searx.data - update_currencies.py 2021-05-01 08:23:05 +00:00
dalf
de54209f9e Update searx.data - update_ahmia_blacklist.py 2021-05-01 08:23:00 +00:00
dalf
982a5a7d2e Update searx.data - update_wikidata_units.py 2021-05-01 08:22:48 +00:00
dalf
c004712eef Update searx.data - update_firefox_version.py 2021-05-01 08:22:43 +00:00
Alexandre Flament
e5bcc3cbbe
Merge pull request #40 from searxng/link-to-stats 
[mod] engine errors: link to the stats to create an github issue
 2021-04-29 14:37:12 +02:00
Markus Heiser
8026ed0d37 [mod] UI: engine pref-page, about toolbox: add link to stats 
In the preference page, in the 'about' toolbox of an engine, add a link to the
stats page of the engine, if the engine had one or more errors.

Condition is::

    reliabilities[<engine.name>].errors

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-04-29 11:11:05 +02:00
Alexandre Flament
34bced29ae [mod] engine errors: link to the stats to create an github issue 2021-04-29 11:10:38 +02:00
Michael Ilsaas
0c43cf89ca [fix] URL to solidtorrent result page 
Reported-by: https://github.com/searx/searx/pull/2786
 2021-04-29 10:40:47 +02:00
Markus Heiser
8291804082 [static] make themes.all - from commit 9e8171e38 
Based on commit 9e8171e38 this patch is generated by::

     make themes.all

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-04-28 16:22:47 +02:00
Markus Heiser
70cc196e2d [fix] new_issue.html: drop inline style attributes (CSP conformance) 
Inline styles are blocked by default with Content Security Policy (CSP).  Move
the inline styles from 'new_issue.html' to::

    searx/static/themes/__common__/less/new_issue.less

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-04-28 16:22:37 +02:00
Markus Heiser
7f2dc530b2 [fix] simple theme - remove no longer used stats.less file 
File searx/static/themes/simple/less/stats.less is not used (imported) in any
other less file.  I can't say when it's usage was dropped or if it has ever been
used.  ATM this file is without any usage.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-04-28 15:42:08 +02:00
Markus Heiser
54d57c8296 [fix] grunt/less/development: oscar light & dark logicodev mixes CSS 
closes: https://github.com/searxng/searxng/issues/39

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-04-28 15:15:29 +02:00
Alexandre Flament
283ae7bfad [fix] searx.network: fix rare cases where LOOP is None 
* searx.network.client.LOOP is initialized in a thread
* searx.network.__init__ imports LOOP which may happen
  before the thread has initialized LOOP

This commit adds a new function "searx.network.client.get_loop()"
to fix this issue
 2021-04-27 17:47:36 +02:00
Alexandre Flament
87e914e398
Merge pull request #28 from searxng/mod-processors-error-message 
[mod] processors: show identical error messages on /search and /stats
 2021-04-27 16:50:41 +02:00
Markus Heiser
924f9afea3 [lint] pylint searx/search/processors files / BTW add some doc-strings 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-04-27 15:13:39 +02:00
Alexandre Flament
b1557b5443 [mod] processors: show identical error messages on /search and /stats 2021-04-27 14:20:07 +02:00
Alexandre Flament
b8f5adf500 [mod] searx.metrics.error_recorder: store relative file name instead of the full absolute file name. 2021-04-27 10:42:00 +02:00