kamp/searxng
1
0
Fork 0
mirror of https://github.com/searxng/searxng.git synced 2024-07-05 00:40:15 +02:00
Code
14323d683f
searxng/searx/engines
History
Markus Heiser 14323d683f [fix] ddg-lite & ddg-extra: don't send empty vqd value 
DDG's bot detection is sensitive to the vqd value.  For some search terms (such
as extremely long search terms that are often sent by bots), no vqd value can be
determined.

If SearXNG cannot determine a vqd value, then no request should go out to
DDG (WEB): a request with a wrong vqd value leads to DDG temporarily putting
SearXNG's IP on a block list.

Requests from IPs in this block list run into timeouts.

Not sure, but it seems the block list is a sliding window: to get my IP rid from
the bot list I had to cool down my IP for 1h (send no requests from that IP to
DDG).

Since such issues can't reproduce in a local instance I tested this patch 24h on
my public SearXNG instance: There are still errors (rare), but the reliability
is still 100%.

Related:

- https://github.com/searxng/searxng/pull/2922
- https://github.com/searxng/searxng/pull/2923

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-12 08:52:28 +02:00
..
__init__.py [doc] rearranges Settings & Engines docs for better readability 2023-07-01 22:45:19 +02:00
9gag.py get the not cropped version of the thumbnail when the image height is not too important 2022-08-24 18:33:11 +07:00
1337x.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
ahmia.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
annas_archive.py [fix] engine annas archive - fetch traits (modified xpath selectors) 2023-09-27 09:12:38 +02:00
apkmirror.py rollback test 2023-03-15 19:55:20 +01:00
apple_app_store.py remove thumbnail from results 2022-08-27 06:23:30 +07:00
apple_maps.py add poi's website and phone number, doesn't crash when there is no displayMapRegion, query the token on the first request 2022-08-27 06:17:58 +07:00
archlinux.py [fix] spelling 2023-09-18 16:20:27 +02:00
artic.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
arxiv.py Science category: update the engines 2022-09-23 20:45:58 +02:00
bandcamp.py [mod] bandcamp & genius: in result set img_src instead thumbnail 2022-02-21 22:12:07 +01:00
base.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
bilibili.py [fix] bilibili new api used 2023-09-28 08:24:51 +02:00
bing_images.py [fix] Revision of the Bing engines 2023-10-01 08:01:38 +02:00
bing_news.py [fix] Revision of the Bing engines 2023-10-01 08:01:38 +02:00
bing_videos.py [fix] Revision of the Bing engines 2023-10-01 08:01:38 +02:00
bing.py [fix] Revision of the Bing engines 2023-10-01 08:01:38 +02:00
brave.py [fix] engine brave - fetch traits (modified settings menu) 2023-09-27 09:12:38 +02:00
bt4g.py [fix] spelling 2023-09-18 16:20:27 +02:00
btdigg.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
chefkoch.py [feat] engine: implementation of chefkoch.de 2023-09-21 17:23:59 +02:00
command.py [doc] rearranges Settings & Engines docs for better readability 2023-07-01 22:45:19 +02:00
core.py [fix] doc of the paper.html template (isbn, issn) 2022-09-25 15:46:29 +02:00
crossref.py [fix] engine - Crossref 2023-09-14 17:39:23 +02:00
currency_convert.py [fix] engine - currency fix and simplify 2023-09-28 08:29:38 +02:00
dailymotion.py [fix] engine & network issues / documentation and type annotations 2023-06-25 13:58:26 +02:00
deepl.py [mod] add deepl translation engine 2022-08-10 09:14:36 +02:00
deezer.py [mod] templates: rename field for <iframe> URL to iframe_src 2022-02-18 19:00:49 +01:00
demo_offline.py [mod] replace engines_languages.json by engines_traits.json 2023-03-24 10:37:42 +01:00
demo_online.py [mod] add 'Accept-Language' HTTP header to online processores 2022-08-01 17:01:59 +02:00
deviantart.py [fix] engine deviantart: review of the result-scrapper 2023-09-11 13:22:36 +02:00
dictzone.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
digbt.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
docker_hub.py [mod] make python code pylint 2.16.1 compliant 2023-02-10 13:59:21 +01:00
doku.py [fix] issues reported by pylint 2021-12-27 10:16:20 +01:00
duckduckgo_definitions.py [mod] engine duckduckgo definitions: support for answer source 2023-08-18 19:07:14 +02:00
duckduckgo_extra.py [fix] ddg-lite & ddg-extra: don't send empty vqd value 2023-10-12 08:52:28 +02:00
duckduckgo_weather.py [fix] engine duckduckgo weather api changes 2023-09-05 16:55:00 +02:00
duckduckgo.py [fix] ddg-lite & ddg-extra: don't send empty vqd value 2023-10-12 08:52:28 +02:00
duden.py [fix] engine duden - don't raise exception on empty result list 2022-08-20 08:41:03 +02:00
dummy-offline.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
dummy.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
ebay.py [mod] Pass desired ebay domain in settings 2022-04-16 19:10:35 +02:00
elasticsearch.py [doc] rearranges Settings & Engines docs for better readability 2023-07-01 22:45:19 +02:00
emojipedia.py [fix] emojipedia: fix engine 2023-10-01 08:19:45 +02:00
fdroid.py [enh] add more categories 2022-01-05 11:00:11 +01:00
flickr_noapi.py [fix] engine flickr: adapt to the new data model from flicker's response 2023-03-30 21:04:53 +02:00
flickr.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
framalibre.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
freesound.py [mod] result_templates/default.html replace embedded HTML by data_src audio_src 2022-02-13 14:20:47 +01:00
frinkiac.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
genius.py [mod] bandcamp & genius: in result set img_src instead thumbnail 2022-02-21 22:12:07 +01:00
gentoo.py [mod] remove obsolete EngineTraits.supported_languages 2023-03-24 10:37:42 +01:00
github.py [fix] typos / reported by @kianmeng in searx PR-3366 2022-09-27 18:32:14 +02:00
google_images.py [fix] engine - google images error when no results 2023-09-21 16:38:37 +02:00
google_news.py [fix] spelling 2023-09-18 16:20:27 +02:00
google_play.py [mod] engine google_play: raise error on unsupported category 2023-08-10 12:35:24 +02:00
google_scholar.py [mod] Google: reversed engineered & upgrade to data_type: traits_v1 2023-03-24 10:37:42 +01:00
google_videos.py [fix] engine google_video: google has changed the layout of the rsponse 2023-08-22 08:34:04 +02:00
google.py [fix] spelling 2023-09-18 16:20:27 +02:00
hackernews.py [feat] implement hackernews engine - news.ycombinator.com 2023-10-09 14:00:04 +02:00
imdb.py [mod] add movies category for tmdb, imdb and moviepilot 2023-09-29 22:37:51 +02:00
imgur.py [feat] engine: implementation of imgur 2023-09-22 20:50:53 +02:00
ina.py [fix] ina engine 2022-01-28 22:33:41 +01:00
internet_archive_scholar.py [fix] internet archive scholar: crash when there's no title 2023-09-22 18:49:39 +02:00
invidious.py [fix] engine - invidious thumbnails 2023-09-13 11:37:42 +02:00
jisho.py [format.python] based on bugfix in 9ed626130 2022-05-07 18:23:10 +02:00
json_engine.py [enh] Initial no paging support for Yep.com 2022-06-11 14:17:44 +02:00
kickass.py [fix] kickass: crash when no results 2023-10-07 11:48:23 +02:00
lemmy.py [mod] utils.py: add markdown_to_text helper function 2023-09-08 11:47:13 +02:00
lingva.py [mod] Adds Lingva translate engine 2022-07-04 19:06:45 +02:00
loc.py [mod] library of congress: fix engine 2023-09-11 19:42:31 +02:00
mastodon.py [feat] engine: implementation of mastodon 2023-10-06 10:58:23 +02:00
material_icons.py [feat] engine: implementation of google icons/material design icons 2023-09-21 15:16:49 +02:00
mediathekviewweb.py [fix] engine mediathekviewweb: replace http links by https 2022-03-07 19:49:16 +01:00
mediawiki.py [mod] engine: wikimedia - improve results, add addition settings & doc 2023-08-04 19:06:50 +02:00
meilisearch.py [doc] rearranges Settings & Engines docs for better readability 2023-07-01 22:45:19 +02:00
metacpan.py Add MetaCPAN engine 2022-11-07 08:07:06 -06:00
mixcloud.py [mod] add artwork to mixcloud & soundcloud engines 2022-02-19 21:59:12 +01:00
mongodb.py [doc] rearranges Settings & Engines docs for better readability 2023-07-01 22:45:19 +02:00
moviepilot.py [fix] engine - moviepilot instead of thumbnail use img_src 2023-09-30 11:29:19 +02:00
mrs.py [fix] matrixrooms add proper MRS integration 2023-10-09 13:25:13 +02:00
mwmbl.py [doc] add documentation of Mwmbl engine & autocompleter 2023-08-27 17:25:26 +02:00
mysql_server.py [doc] rearranges Settings & Engines docs for better readability 2023-07-01 22:45:19 +02:00
nyaa.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
odysee.py [fix] spelling 2023-09-18 16:20:27 +02:00
opensemantic.py [enh] engines: add about variable 2021-01-14 20:57:17 +01:00
openstreetmap.py [fix] spelling 2023-09-18 16:20:27 +02:00
openverse.py [fix] ccengine engine - avoid unwanted redirects 2022-01-07 14:14:31 +01:00
pdbe.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
peertube.py [fix] engine & network issues / documentation and type annotations 2023-06-25 13:58:26 +02:00
photon.py [fix] typos / reported by @kianmeng in searx PR-3366 2022-09-27 18:32:14 +02:00
pinterest.py [feat] engine: implementation of pinterest 2023-09-30 15:01:45 +02:00
piped.py [mod] piped: always show video length if available 2023-10-07 11:45:46 +02:00
piratebay.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
postgresql.py [doc] rearranges Settings & Engines docs for better readability 2023-07-01 22:45:19 +02:00
pubmed.py [fix] pubmed content being None 2023-09-14 18:40:15 +02:00
qwant.py [fix] engine qwant (web-lite) - ignore advertising adds 2023-09-19 17:06:56 +02:00
radio_browser.py [mod] engine - simplify region & lang handling, make filters configurable 2023-10-05 10:55:08 +02:00
recoll.py [doc] rearranges Settings & Engines docs for better readability 2023-07-01 22:45:19 +02:00
reddit.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
redis_server.py [doc] rearranges Settings & Engines docs for better readability 2023-07-01 22:45:19 +02:00
rumble.py [fix] rumble redirect 2023-09-14 19:21:21 +02:00
scanr_structures.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
searchcode_code.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
searx_engine.py reference docs.searxng.org 2022-01-02 21:18:29 +01:00
seekr.py [mod] improve seekr engines and add documentation 2023-08-15 16:17:42 +02:00
semantic_scholar.py [mod] science category: various update of about PR 1705 2022-09-23 20:52:55 +02:00
sepiasearch.py [mod] SepiaSearch: re-engineered & upgrade to data_type: traits_v1 2023-03-24 10:37:42 +01:00
seznam.py [fix] engine seznam: fix issues reported by black & pylint 2023-03-31 17:25:39 +02:00
sjp.py [fix] sjp engine - convert enginename to a latin1 compliance name 2022-07-24 21:10:55 +02:00
solidtorrents.py [fix] solidtorrents redirects 2023-09-14 18:03:21 +02:00
solr.py [doc] rearranges Settings & Engines docs for better readability 2023-07-01 22:45:19 +02:00
soundcloud.py [mod] add artwork to mixcloud & soundcloud engines 2022-02-19 21:59:12 +01:00
spotify.py [mod] templates: rename field for <iframe> URL to iframe_src 2022-02-18 19:00:49 +01:00
springer.py [fix] springer: unsupported operand type(s) for +: 'NoneType' and 'str' 2022-09-25 15:25:55 +02:00
sqlite.py [fix] spelling 2023-09-18 16:20:27 +02:00
stackexchange.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
startpage.py [fix] engine & network issues / documentation and type annotations 2023-06-25 13:58:26 +02:00
svgrepo.py [feat] new engine: svgrepo 2023-09-12 20:38:36 +02:00
tagesschau.py [mod] tagesschau: add option to only use tagesschau urls 2023-09-30 11:00:11 +02:00
tineye.py [fix] engine tineye: handle 422 response of not supported img format 2022-07-23 16:00:58 +02:00
tokyotoshokan.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
torznab.py [fix] spelling 2023-09-18 16:20:27 +02:00
translated.py [enh] move dictionaries, Erowid & IMDb out of general category 2022-01-05 11:03:44 +01:00
unsplash.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
vimeo.py [mod] templates: rename field for <iframe> URL to iframe_src 2022-02-18 19:00:49 +01:00
wallhaven.py [feat] engine: implementation of wallhaven 2023-09-21 14:25:43 +02:00
wikicommons.py [fix] engine: wikicommons - don't quoute ':|' in URL parameters 2023-09-03 18:45:39 +02:00
wikidata.py wikipedia wikidata infobox + disable wikisource (#2806) 2023-09-19 10:31:02 +02:00
wikipedia.py wikipedia wikidata infobox + disable wikisource (#2806) 2023-09-19 10:31:02 +02:00
wolframalpha_api.py [fix] typos / reported by @kianmeng in searx PR-3366 2022-09-27 18:32:14 +02:00
wolframalpha_noapi.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
wordnik.py [format.python] initial formatting of the python code 2021-12-27 09:26:22 +01:00
wttr.py [mod] clarify the difference of the default category and subgrouping 2023-04-07 11:03:25 +02:00
www1x.py [fix] 1x engine 2022-01-30 19:48:40 +01:00
xpath.py [fix] spelling 2023-09-18 16:20:27 +02:00
yacy.py [mod] yacy: use official instance by default and fix crashes 2023-10-09 20:50:24 +02:00
yahoo_news.py [fix] issues reported by pylint 2021-12-27 10:16:20 +01:00
yahoo.py [typo] solved a typo in yahoo error message. 2023-10-01 08:29:06 +02:00
youtube_api.py [mod] templates: rename field for <iframe> URL to iframe_src 2022-02-18 19:00:49 +01:00
youtube_noapi.py [fix] google & youtube - set EU consent cookie 2022-07-25 13:27:06 +02:00
yummly.py [feat] engine: implementation of Yummly 2023-09-08 11:47:13 +02:00
zlibrary.py [feat] engine: re-enables z-library (zlibrary-global.se) 2023-07-07 21:36:51 +02:00