kamp/searxng
1
0
Fork 0
mirror of https://github.com/searxng/searxng.git synced 2024-11-05 04:40:11 +01:00
Code
5,030 Commits 7 Branches 17 Tags 128 MiB
bc540dce5f
Commit Graph

3237 Commits

Author SHA1 Message Date
Markus Heiser
4c9b8b29ee [mod] engine duckduckgo - use DuckDuckGo-Lite 
Implement a scrapper for DuckDuckGo-Lite [1].  The existing DuckDuckGo [2]
engine does not support paging.  DuckDuckgo-Lite is much faster, less verbose
and does have a paging option (reversed engineered from the input form of [1]).

[1] https://lite.duckduckgo.com/lite
[2] https://duckduckgo.com/

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-10-01 20:01:41 +02:00
Alexandre Flament
a53a4d5012 [build] /static 2021-10-01 18:09:37 +02:00
Alexandre Flament
74b0830362 SearXNG: simple theme 2021-10-01 18:06:55 +02:00
Alexandre Flament
499c7b3cce SearXNG: .github, searxng-docker 2021-10-01 09:48:53 +02:00
Alexandre Flament
458575c221
Merge pull request #361 from dalf/searxng-theme-oscar 
SearXNG theme oscar
 2021-10-01 09:15:46 +02:00
Markus Heiser
c74f45b1df
Merge pull request #365 from searxng/update_data_update_wikidata_units.py 
Update searx.data - update_wikidata_units.py
 2021-10-01 07:11:13 +00:00
Markus Heiser
62ea3c7074
Merge pull request #366 from searxng/update_data_update_languages.py 
Update searx.data - update_languages.py
 2021-10-01 06:41:09 +00:00
Markus Heiser
d814f1182f
Merge pull request #364 from searxng/update_data_update_firefox_version.py 
Update searx.data - update_firefox_version.py
 2021-10-01 06:34:20 +00:00
Markus Heiser
8479a83411
Merge pull request #363 from searxng/update_data_update_ahmia_blacklist.py 
Update searx.data - update_ahmia_blacklist.py
 2021-10-01 06:33:51 +00:00
dalf
db51a64390 Update searx.data - update_languages.py 2021-10-01 06:23:31 +00:00
dalf
a582e55af0 Update searx.data - update_wikidata_units.py 2021-10-01 06:23:25 +00:00
dalf
56c68beb42 Update searx.data - update_firefox_version.py 2021-10-01 06:23:14 +00:00
dalf
453114fbe4 Update searx.data - update_ahmia_blacklist.py 2021-10-01 06:23:12 +00:00
dalf
0c24a03f05 Update searx.data - update_currencies.py 2021-10-01 06:23:04 +00:00
Alexandre Flament
1dfac73a64 [build] /static 2021-09-30 19:01:50 +02:00
Alexandre Flament
df06dddc04 SearXNG: oscar theme 2021-09-30 18:50:21 +02:00
Alexandre Flament
a582cf3d82
Merge pull request #353 from return42/stackexchange 
[mod] engines - add Stack Exchange API v2.3
 2021-09-29 10:38:50 +02:00
Markus Heiser
ecb3912bd0 [fix] engine stackexchange - decode HTML entities in title & content 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-29 08:08:18 +02:00
Alexandre Flament
a9c3c88cc0 [mod] searx.network.stream returns a tuple (response, stream) 2021-09-28 19:33:29 +02:00
Alexandre Flament
29893cf816 [fix] searx.network.stream: fix memory leak 2021-09-28 19:28:12 +02:00
Markus Heiser
5efe77bdf5 [mod] engines - add superuser.com (Stack Exchange API) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-28 19:20:13 +02:00
Markus Heiser
29eb06ab16 [mod] engines - add askubuntu.com (Stack Exchange API) 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-28 19:17:26 +02:00
Markus Heiser
b62851559b [mod] replace old stackoverflow engine by Stack Exchange API v2.3 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-28 19:12:37 +02:00
Markus Heiser
55fee1e45d [mod] engines - add Stack Exchange API v2.3 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-28 19:01:04 +02:00
Alexandre Flament
2eab89b4ca [fix] checker: fix memory usage 
* download images using the "image_proxy" network (HTTP/1 instead of HTTP/2)
* don't cache data: URL (reduce memory usage)
* after each test: purge image URL cache then call garbage collector
* download only the first 64kb of images
 2021-09-28 15:26:02 +02:00
MrPaulBlack
43fc136207 [build] /static 2021-09-28 00:26:43 +02:00
MrPaulBlack
57f58f85bd [fix] position of engine description tooltip 2021-09-28 00:26:17 +02:00
Markus Heiser
73cb80e71d [build] /static 2021-09-27 17:40:11 +02:00
Markus Heiser
8ac776765c [theme] simple: do not prefer arial over user's sans-serif font 
The default *sans-serif* font from the browsers most often renders much better
compared to Arial font.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-27 17:37:15 +02:00
MrPaulBlack
e8ed56251a [build] /static 2021-09-27 13:47:00 +02:00
MrPaulBlack
f81afed60b [fix] backtotop and pagination on image result page 2021-09-27 13:46:40 +02:00
MrPaulBlack
8268873701 [fix] several issues in the simple theme 
- using more rem in style and definitions
- mobile width in preferences.less fix max-width: 75em to 80em (normalized with
  style.less and other)
- do not display #backToTop position on tablet (when max-width: 80em)
- fix answer box on mobile (when max-width: 50em)
 2021-09-27 13:17:17 +02:00
MrPaulBlack
b6ae1f1c7a [theme] margin around checkboxes is bigger now, index centers the search more and the pref, stats and about page have a max-width now 2021-09-27 12:26:54 +02:00
MrPaulBlack
859179f0c9 [theme] optimize switching between different device types 2021-09-27 12:26:54 +02:00
MrPaulBlack
385dce213b [theme] cleanup grid layout and remove various margins and paddings from elements 2021-09-27 12:26:54 +02:00
Alexandre Flament
cc3c54f4cd simple theme: update 2021-09-27 12:26:54 +02:00
MrPaulBlack
88c17d106d [fix] make selected tabs not change wifth anymore compared to not being selected 2021-09-27 12:26:54 +02:00
Alexandre Flament
b046322c7b
Merge pull request #333 from dalf/enh-engine-descriptions 
RFC: /preferences: display engine descriptions
 2021-09-25 11:29:25 +02:00
Alexandre Flament
0c3314b338
Merge pull request #344 from dalf/osm-lang-all-fix 
[fix] openstreetmap engine: map "all" language to English
 2021-09-25 11:29:13 +02:00
Alexandre Flament
ab569c1e12 [fix] openstreetmap engine: optmizer SPARQL query 
add
hint:Query hint:optimizer "None".
to the SPARQL query to keep the response time small.

It tells the optimizer to follow the path from ?item to the different property values
instead of the other way around.
See https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/query_optimization#Property_paths
 2021-09-25 11:16:22 +02:00
Alexandre Flament
39876d9f14 [build] /static 2021-09-24 20:38:56 +02:00
Alexandre Flament
2f1384f198 [enh] themes: display the engine descriptions 2021-09-24 20:38:05 +02:00
Alexandre Flament
bfd24d1226 [mod] add /engine_descriptions.json endpoint 
returns engine descriptions (JSON):
* key: engine name
* value: description in the user locale, use English description as a fallback
 2021-09-24 20:20:30 +02:00
Alexandre Flament
8961131497 [fix] fix the about section of some engines 2021-09-24 20:20:30 +02:00
Alexandre Flament
6f11b61cd5 [fix] openstreetmap engine: map "all" language to English 2021-09-24 20:12:18 +02:00
MrPaulBlack
3017227c14 [preferences] remove checkbox class from query section 2021-09-24 19:14:35 +02:00
MrPaulBlack
30a9146bdc [build] /static 2021-09-24 18:02:07 +02:00
MrPaulBlack
7cb5099c88 [preferences] make elements in query selecter align left 2021-09-24 18:01:20 +02:00
MrPaulBlack
d24bcbb6f0 [preferences] reorder tabs in simple to match oscar theme 2021-09-24 16:21:43 +02:00
MrPaulBlack
45eb2b5637 [preferences] cleanup whitespaces 2021-09-24 16:15:26 +02:00
MrPaulBlack
2d26ace265 [preferences] add special queries tab to preferences 2021-09-24 16:04:10 +02:00
MrPaulBlack
fa95cd91a8 [build] /static 2021-09-22 21:44:35 +02:00
MrPaulBlack
79351c2e4d [fix] searx.js null pointer exception when category div is missing 2021-09-22 21:41:19 +02:00
Alexandre Flament
12c90a1b47
Merge pull request #336 from MrPaulBlack/simple-search-form 
[theme] remove categories, language and time range from index.html
 2021-09-22 18:20:33 +02:00
MrPaulBlack
575483ad2a [theme] add newline at the end of the file 2021-09-21 00:12:39 +02:00
MrPaulBlack
880fe67172 [theme] remove categories, language and time range from index 2021-09-21 00:10:12 +02:00
Alexandre Flament
e8e81c7432 [build] /static 2021-09-19 22:01:19 +02:00
Alexandre Flament
2cc6301c98 [fix] Oscar theme: fix input field on Safari 
close #323
 2021-09-19 22:00:59 +02:00
searxng-bot
1bc50c1eb4 [translations] update 
3834942d - 2021-09-15 - ButterflyOfFire <butterflyoffire@protonmail.com>
 2021-09-17 13:50:32 +00:00
Alexandre Flament
dc74df3a55
Merge pull request #261 from dalf/upgrade_httpx 
[upd] upgrade httpx 0.19.0
 2021-09-17 11:48:37 +02:00
Markus Heiser
443bf35e09 [pylint] fix global-variable-not-assigned issues 
If there is no write access, there is no need for global.  Remove global
statement if there is no assignment.

global-variable-not-assigned:
  Using global for names but no assignment is done Used when a variable is
  defined through the "global" statement but no assignment to this variable is
  done.

In Pylint 2.11 the global-variable-not-assigned checker now catches global
variables that are never reassigned in a local scope and catches (reassigned)
functions [1][2]

[1] https://pylint.pycqa.org/en/latest/whatsnew/2.11.html
[2] https://github.com/PyCQA/pylint/issues/1375

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-17 10:14:27 +02:00
Alexandre Flament
41f6359d06 [fix] error recorder: avoid RuntimeError on some rare occasion 
httpx.RequestError (subclass of httpx.HTTPError) has a property request.
This property raises a RuntimeError if the attributes _request is None.
To avoid a cascade of errors, this commit reads directly the _request attribute.
 2021-09-17 10:06:24 +02:00
Alexandre Flament
b10403d3a1 [mod] searx.network: remove redundant code 
searx.client.new_client: the proxies parameter is a dictonnary,
and the protocol (key of the dictionnary) is already normalized
(see usage of searx.network.network.PROXY_PATTERN_MAPPING)
 2021-09-17 10:06:24 +02:00
Alexandre Flament
8e73438cbe [upd] upgrade httpx 0.19.0 
adjust searx.network module to the new internal API
see https://github.com/encode/httpx/pull/1522
 2021-09-17 10:06:22 +02:00
Alexandre Flament
602cbc2c99
Merge pull request #297 from dalf/engine-logger-enh 
debug mode: more readable logging
 2021-09-14 07:06:28 +02:00
Markus Heiser
de5a8ee7d7 [fix] settings - displayed name is SearXNG 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-13 14:36:24 +02:00
Markus Heiser
97acedd9fe
Merge pull request #305 from dalf/fix-engine-logger 
[fix] logger per engine: make .logger is always initialized
 2021-09-13 07:03:37 +00:00
Alexandre Flament
f8793fbda0 [fix] logger per engine: make .logger is always initialized 
the openstreetmap engine imports code from the wikidata engine.
before this commit, specific code make sure to copy the logger variable to the wikidata engine.

with this commit searx.engines.load_engine makes sure the .logger is initialized.
The implementation scans sys.modules for module name starting with searx.engines.
 2021-09-13 08:47:59 +02:00
MrPaulBlack
644ccf16be [theme] make engine stats clickable 2021-09-12 10:52:15 +02:00
MrPaulBlack
48d4296caa [theme] link to engine stats in footer 2021-09-12 10:37:21 +02:00
Markus Heiser
ecd6ca86d0
Merge pull request #308 from dalf/plugin_hostname_replace 
[enh] add hostname_replace plugin
 2021-09-12 07:53:09 +00:00
Alexandre Flament
0f43b39eac [enh] add hostname_replace plugin 
* backport of https://github.com/searx/searx/pull/2724
* allow to remove result if the replacement is the boolean value false
 2021-09-11 13:23:06 +02:00
Alexandre Flament
2b53d718e4 [fix] PR #257: use the image_proxy network instead of the default network 2021-09-11 11:15:51 +02:00
Alexandre Flament
6abc37efc7 [mod] production mode: make sure to set the log level to WARNING 
set:
* the root logger level to WARNING
* the werkzeug logger level to WARNING (useful when there is no uwsgi)
 2021-09-11 10:13:16 +02:00
Alexandre Flament
0e42db9da1 [mod] xpath engine: remove logging of the requested URL 2021-09-11 10:13:16 +02:00
Alexandre Flament
91a6d80e82 [mod] debug mode: log HTTP requests with network name 
For example wikipedia requests use the logger name "searx.network.wikipedia"

Log is disable when searx_debug is False
 2021-09-11 10:13:14 +02:00
Alexandre Flament
577998106a [mod] debug mode: use coloredlogs package 
in debug mode, add some padding to make the output more readable
in procution mode, add the timestamp
 2021-09-10 21:49:34 +02:00
Alexandre Flament
b513917ef9 [mod] searx.metrics & searx.search: use the engine loggers 
metrics & processors use the engine logger
 2021-09-10 21:49:34 +02:00
Alexandre Flament
d9658a7f91 [fix] searx/results.py: strip result['content'] only if it exists 
fix PR #302
 2021-09-10 17:58:19 +02:00
Markus Heiser
b671e0364f
Merge pull request #302 from dalf/mod_plugin_on_result 
[mod] plugin: call on_result for each result of each engines.
 2021-09-10 09:20:26 +00:00
Alexandre Flament
0b27c8698f [doc] update docs/dev/plugins.rst 2021-09-10 10:58:22 +02:00
Alexandre Flament
70c9bb6f2c [fix] remove references to transifex 2021-09-09 21:29:55 +02:00
Alexandre Flament
b941763e20 [mod] ahmia_filter: use on_result instead of post_search 
see commit 6c9ae7911e9639bc46cd53af215734b4bdb61ba9
 2021-09-09 11:31:46 +02:00
Alexandre Flament
660c180170 [mod] plugin: call on_result after each engine from the ResultContainer 
Currently, searx.search.Search calls on_result once the engine results have been merged
(ResultContainer.order_results).

on_result plugins can rewrite the results: once the URL(s) are modified, even they can be merged,
it won't be the case since ResultContainer.order_results has already be called.

This commit call on_result inside for each result of each engines.
In addition the on_result function can return False to remove the result.

Note: the on_result function now run on the engine thread instead of the Flask thread.
 2021-09-09 11:31:44 +02:00
Alexandre Flament
fc20c561bf [mod] oa_doi_rewrite plugin: get_doi_resolver: remove args parameter 
doi_resolvers.get_value('preferences') already contains the value from
request.args.get('doi_resolver')
 2021-09-07 19:14:36 +02:00
Alexandre Flament
a3789b3bb4
Merge pull request #295 from return42/fix-csp-2 
[fix] make HTML templates more CSP compliant
 2021-09-07 18:51:00 +02:00
Markus Heiser
b0623ac9f6 [build] /static 2021-09-07 17:48:56 +02:00
Markus Heiser
0240ca9567 [fix] oscar template - merge the two class attributes into one 
Suggested-by: @dalf https://github.com/searxng/searxng/pull/295#discussion_r703337237
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-07 16:46:49 +02:00
Markus Heiser
bd830f7449 [fix] simple theme - preferences: fix tooltip overflow 
Suggested-by: @dalf https://github.com/searxng/searxng/pull/295#discussion_r703370358
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-07 16:46:49 +02:00
Markus Heiser
2a3b9a2e26 [pylint] searx: drop no longer needed 'missing-function-docstring' 
Suggested-by: @dalf https://github.com/searxng/searxng/issues/102#issuecomment-914168470
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-07 13:34:35 +02:00
Markus Heiser
f0059b80ed [pylint] engines: drop no longer needed 'missing-function-docstring' 
Suggested-by: @dalf https://github.com/searxng/searxng/issues/102#issuecomment-914168470
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-07 13:26:59 +02:00
Markus Heiser
9068a72c6a [fix] oscar theme - preferences: rename col-checkbox/col-stat 
[1] https://github.com/searxng/searxng/pull/295#discussion_r703318053
[2] https://github.com/searxng/searxng/pull/295#discussion_r703337237

Suggested-by: @dalf [1] [2]
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-07 12:26:07 +02:00
Markus Heiser
82847df300 [fix] add 'categories' to PYLINT_ADDITIONAL_BUILTINS_FOR_ENGINES 
androp no longer needed (see line 591 in 7b235a1)::

    # pylint: disable=undefined-variable

Suggested-by: @dalf https://github.com/searxng/searxng/issues/102#issuecomment-914068609
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-07 10:29:38 +02:00
Markus Heiser
cd033b5416 [fix] drop useless pylint: disable=undefined-variable 
Since 7b235a1 (see line 591) it is no longer needed to disable
'undefined-variable' for names defined in::

   PYLINT_ADDITIONAL_BUILTINS_FOR_ENGINES

Suggested-by: @dalf https://github.com/searxng/searxng/issues/102#issuecomment-914068609
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-07 10:26:15 +02:00
Markus Heiser
222031e975 [fix] make oscar theme more CSP compliant - col-checkbox 
Add col-checkbox in::

    searx/static/themes/oscar/src/less/preferences.less

Replaced style in file::

    searx/templates/oscar/preferences.html

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-07 08:24:48 +02:00
Markus Heiser
9d106ae972 [fix] make oscar theme more CSP compliant - default-image-style 
Add default-image-style in::

    searx/static/themes/oscar/src/less/result_templates.less

Replaced style= in files::

    ./oscar/result_templates/default.html:19:        <img src="{{ image_proxify(result.img_src or result.thumbnail) }}" title="{{ result.title|striptags }}" style="width: auto; max-height: 60px; min-height: 60px;" class="col-xs-2 col-sm-4 col-md-4 result-content">
    ./oscar/result_templates/files.html:24:<img src="{{ image_proxify(result.img_src) }}" alt="{{ result.title|striptags }}" title="{{ result.title|striptags }}" style="width: auto; max-height: 60px; min-height: 60px;" class="col-xs-2 col-sm-4 col-md-4 result-content">

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-07 08:20:30 +02:00
Markus Heiser
e564cef0fe [fix] make /preferences more CSP compliant - simple & oscar theme 
In simple & oscar theme remove ``style=`` properties which violates CSP but seem
not to have a magnification in GUI (UX).

Fixed files::

    ./oscar/preferences.html:25:<td class="{{ label }}" style="padding: 2px">{{- "" -}}
    ./simple/preferences.html:39:<td class="{{ label }}" style="padding: 2px; width: 13rem;">{{- "" -}}
    ./simple/preferences.html:77:    <div class="engine-tooltip" style="right: 12rem;" role="tooltip" id="{{engine_name}}_reliability">
    ./simple/preferences.html:258:     <p class="text-muted" style="margin:20px 0;">

Reported-by: https://github.com/searxng/searxng/issues/57
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-07 08:20:30 +02:00
Alexandre Flament
ea60c03827 [fix] fix openstreetmap engine 
close #298

This is a workaround: inside engine code, any call to function in another engine can crash
since the logger won't be initialized except if it is done explicitly.
 2021-09-06 22:44:22 +02:00
Markus Heiser
aecfb2300d [mod] one logger per engine - drop obsolete logger.getChild 
Remove the no longer needed `logger = logger.getChild(...)` from engines.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-06 18:05:46 +02:00
Markus Heiser
7b235a1c36 [mod] one logger per engine 
Suggested-by: @dalf in https://github.com/searxng/searxng/issues/98#issuecomment-849013518
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2021-09-06 17:47:28 +02:00