Bnyro
48cb58bd2e
[feat] duckduckgo: support for videos and news
2023-10-09 06:53:43 +02:00
Bnyro
c3ab49cd90
[fix] kickass: crash when no results
2023-10-07 11:48:23 +02:00
Bnyro
f22daf8b47
[mod] piped: always show video length if available
2023-10-07 11:45:46 +02:00
Bnyro
ce270961e8
[feat] engine: implementation of mastodon
2023-10-06 10:58:23 +02:00
searxng-bot
48f8ac48d1
[translations] update from Weblate
...
68d74328142f091b7f2479c0d7ba4e6cd5929d4e5f5c3b79d44775746291184f24d7e8b16140911f9
2023-10-06 09:19:00 +02:00
Markus Heiser
fd1422a670
[mod] engine - simplify region & lang handling, make filters configurable
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-05 10:55:08 +02:00
Bnyro
3e2ae756f0
[feat] engine: implementation of radio-browser.info
2023-10-05 10:55:08 +02:00
Markus Heiser
d56b31d674
[fix] limiter / botdetection: remove http_connection method
...
Related:
- https://github.com/searxng/searxng/issues/2892#issuecomment-1742153932
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-01 20:24:22 +02:00
Jinyuan Huang
e509cb7c45
[typo] solved a typo in yahoo error message.
2023-10-01 08:29:06 +02:00
Jinyuan Huang
d4d9f2073e
[fix] Bug: Yahoo results for simplified Chinese search sometimes have the first character cut off #2866
...
Co-authored-by: Blair Noctis <n@sail.ng>
2023-10-01 08:29:06 +02:00
Bnyro
fe9386b58d
[fix] emojipedia: fix engine
2023-10-01 08:19:45 +02:00
Markus Heiser
32a4ea350e
[fix] Revision of the Bing engines
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-10-01 08:01:38 +02:00
jazzzooo
079636c079
[fix] engine - bing fix search, pagination, remove safesearch
2023-10-01 08:01:38 +02:00
Bnyro
5ce1792432
[feat] engine: implementation of pinterest
2023-09-30 15:01:45 +02:00
Bnyro
6096457e4d
[fix] matrixrooms.info: pagination not working properly
2023-09-30 14:51:07 +02:00
Markus Heiser
e1a8b8189f
[fix] engine - moviepilot instead of thumbnail use img_src
...
Instead of thumbnail use img_src in the result item, otherwise the "movies"
categories looks clunky.
Related:
- b4e0d2eedc (r128785388)
2023-09-30 11:29:19 +02:00
Bnyro
159629c588
[mod] tagesschau: add option to only use tagesschau urls
2023-09-30 11:00:11 +02:00
Bnyro
2ca60a19fc
[feat] engine: implementation of matrixrooms.info
2023-09-30 09:09:23 +02:00
Bnyro
d1a3fc5be6
[feat] engine: implementation of tootfinder
2023-09-29 23:00:30 +02:00
Bnyro
fc4a20f734
[mod] add movies category for tmdb, imdb and moviepilot
2023-09-29 22:37:51 +02:00
Émilien (perso)
0330648b98
[mod] warn to info for failed ping limiter
2023-09-29 22:20:56 +02:00
Bnyro
26fed56d51
[mod] settings.yml: remove plugin settings for plugins that don't exist anymore
2023-09-29 11:26:49 +02:00
searxng-bot
9a065832e0
[translations] update from Weblate
...
80bbf74ef5d554ef71771fd1b90d412012c81ac171d0aa004c40cc47e729f89f85b28cc475ce5d58140ce0600921f343f714c80f80fdef3c859e4
2023-09-29 09:12:12 +02:00
dalf
ae2762eb78
Update searx.data - update_firefox_version.py
2023-09-29 06:58:11 +02:00
dalf
18da3d65c5
Update searx.data - update_ahmia_blacklist.py
2023-09-29 06:57:40 +02:00
dalf
4a66fa3e7d
Update searx.data - update_wikidata_units.py
2023-09-29 06:56:50 +02:00
dalf
9be6e20256
Update searx.data - update_engine_traits.py
2023-09-29 06:56:10 +02:00
dalf
d821ec44dd
Update searx.data - update_currencies.py
2023-09-29 06:55:04 +02:00
dalf
cdee5aa3b9
Update searx.data - update_engine_descriptions.py
2023-09-29 06:53:05 +02:00
Bnyro
75587d03a6
[fix] categories.html: can't select social media category using search on category select
2023-09-28 10:45:49 +02:00
jazzzooo
e37d775fa2
[fix] engine - currency fix and simplify
2023-09-28 08:29:38 +02:00
Jinyuan Huang
ae28d429c9
[fix] bilibili new api used
2023-09-28 08:24:51 +02:00
jazzzooo
1a66d74673
[fix] engine - kickass update url, fix parsing, use multiple mirrors
2023-09-27 10:19:41 +02:00
Markus Heiser
0c39064d60
Update searx.data - update_engine_traits.py
...
Update searx.data - update_engine_traits.py
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-27 09:12:38 +02:00
Markus Heiser
b428ccc5a0
[fix] engine brave - fetch traits (modified settings menu)
...
Brave has changed it settings menu fundamental. Region codes are no longer in
the HTML page, we have to read the regional codes from a JS:
https://cdn.search.brave.com/serp/v2/_app/immutable/chunks/parameters.734c106a.js
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-27 09:12:38 +02:00
Markus Heiser
3a456b1282
[fix] engine annas archive - fetch traits (modified xpath selectors)
...
Anna’s Archive has cleaned up their languages, available file extensions and
changed the HTML form.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-27 09:12:38 +02:00
Jinyuan Huang
1f835a9f1f
solve #2839
2023-09-25 23:01:14 +02:00
jazzzooo
3c6549a17f
[fix] update user agent
2023-09-25 22:46:22 +02:00
Emilien Devos
ffec8d1b4c
add warning about enabling public_instance setting
2023-09-25 22:31:14 +02:00
Emilien Devos
47721a3485
add new parameter called server.public_instance
...
for enabling by default advanced limiter functions
in the future allow us to add features just for the public instances
2023-09-25 22:31:14 +02:00
Jinyuan Huang
1df4588279
[build] /static
2023-09-25 17:45:21 +02:00
Jinyuan Huang
98366bcf90
[fix] settings icon invisble on mobile devices
2023-09-25 17:45:21 +02:00
Alexandre Flament
19383921d7
/config: expose limiter configuration
2023-09-24 11:01:11 +02:00
Émilien (perso)
1851f27154
[mod] remove twitter ( #2843 )
2023-09-24 08:32:53 +00:00
Markus Heiser
597c68b4aa
[doc] move dosc of botdetection from developer to admin section
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-23 14:00:03 +02:00
Bnyro
75c9de02d1
[feat] engine: implementation of imgur
2023-09-22 20:50:53 +02:00
Bnyro
fb72f71f0a
[fix] internet archive scholar: crash when there's no title
2023-09-22 18:49:39 +02:00
searxng-bot
0660d20e1d
[translations] update from Weblate
...
f27a3bbd4a97336d0a91e8946b319f53731edaf10bd75ab004b8b8cdedd99c9
2023-09-22 18:05:02 +02:00
Bnyro
95885e7f9f
[build] /static
2023-09-22 17:54:55 +02:00
Bnyro
8b0b6731fc
[fix] search.less: start page search view on mobile devices
2023-09-22 17:54:55 +02:00
Markus Heiser
71358e9c67
Revert "[fix] engine - duckduckgo vqd edge-case"
...
This reverts commit 102502a4f0
2023-09-22 09:31:25 +02:00
Bnyro
51236ae47a
[feat] engine: implementation of chefkoch.de
2023-09-21 17:23:59 +02:00
jazzzooo
8bcca0e620
[fix] engine - brave don't show ads
2023-09-21 16:55:39 +02:00
jazzzooo
b729542a66
[fix] engine - google images error when no results
2023-09-21 16:38:37 +02:00
Bnyro
f0df53673e
[mod] search-syntax.md: add tutorial about the feeling lucky feature
2023-09-21 15:17:23 +02:00
Bnyro
cc2e0537a3
[feat] engine: implementation of google icons/material design icons
2023-09-21 15:16:49 +02:00
Bnyro
c999cfb422
[feat] engine: implementation of wallhaven
2023-09-21 14:25:43 +02:00
jazzzooo
102502a4f0
[fix] engine - duckduckgo vqd edge-case
2023-09-20 20:05:06 +02:00
Markus Heiser
d013f51a25
[fix] Search on category select without JavaScript PR #2740
...
Small addendum to #2740 ; search_on_category_select is now no longer a plugin.
Related: https://github.com/searxng/searxng/pull/2740#issuecomment-1725437584
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 21:03:02 +02:00
Markus Heiser
0ce10046c4
[build] /static
2023-09-19 20:15:51 +02:00
ormai
a76c8e4101
Improve the layout of the search page on mobile
2023-09-19 20:15:51 +02:00
Émilien (perso)
33496e7f51
Switch back to qwant web normal API
2023-09-19 17:07:22 +02:00
Markus Heiser
043dcbf7c5
[fix] engine qwant (web-lite) - ignore advertising adds
...
Closes: https://github.com/searxng/searxng/issues/2812
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 17:06:56 +02:00
Markus Heiser
3ac7c40b6a
[build] /static
2023-09-19 11:15:25 +02:00
Emilien Devos
33e722f83b
better error message when no results found
2023-09-19 11:10:48 +02:00
Émilien (perso)
ad725ce7d7
wikipedia wikidata infobox + disable wikisource ( #2806 )
...
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 10:31:02 +02:00
Bnyro
dcee823345
[feat] implement feeling lucky feature
2023-09-19 09:40:57 +02:00
Markus Heiser
71508abcbf
[build] /static
2023-09-19 08:06:06 +02:00
Jinyuan Huang
2ec77ef813
[mod] reformatted keyBindingLayouts in JS client (default, vim)
2023-09-19 08:06:06 +02:00
Markus Heiser
317db5b04f
[mod] preferences: implement drop-down menu for hotkeys (default, vim)
...
Replace the on/off checkbox of the vim-hotkeys in the preferences by a drop-down
menu.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-19 08:06:06 +02:00
Markus Heiser
09935e2897
[build] /static
2023-09-18 21:29:11 +02:00
Bnyro
a55e0ac553
[feat] search on category select without JS
...
Co-authored-by: Alexandre Flament <alex@al-f.net>
2023-09-18 21:29:11 +02:00
Bnyro
efd3a2d6d1
[feat] engine: implementation of internet archive scholar
2023-09-18 18:12:00 +02:00
jazzzooo
223b3487c3
[fix] spelling
2023-09-18 16:20:27 +02:00
Markus Heiser
a9b6963971
[fix] engine - qwant delivers only 5 pages maximum
...
all qwant engines (incl qwant-lite) delivers only 5 pages maximum
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-18 01:43:43 +02:00
jazzzooo
da1446c5ed
[fix] engine - qwant wrong error type
2023-09-18 01:43:43 +02:00
Markus Heiser
7398d525c8
[fix] qwant: subsequent fix of commit d9dbcedeb
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-17 17:52:56 +02:00
Markus Heiser
d9dbcedeb6
[feat] implementation of qwant lite for web search
...
Related: https://github.com/searxng/searxng/issues/2719
Replace: https://github.com/searxng/searxng/pull/2748
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-17 16:53:25 +02:00
Bnyro
b4e0d2eedc
[feat] engine: implemenation of moviepilot (de)
2023-09-17 14:30:56 +02:00
jazzzooo
7dfcc3386e
[fix] tagesschau videos
2023-09-16 18:40:26 +02:00
Markus Heiser
89fbac5763
[build] /static
2023-09-16 13:45:43 +02:00
Jinyuan Huang
92d0c378e0
[fix] keyboard navigation / simple theme (UI)
...
- avoid loop select
- fix select next item in mixed result lists
Replaces: https://github.com/searxng/searxng/pull/2789
Closes: https://github.com/searxng/searxng/issues/2751
Closes: https://github.com/searxng/searxng/issues/2788
2023-09-16 13:45:43 +02:00
jazzzooo
ec540a967a
[fix] brave.videos
2023-09-15 22:00:09 +02:00
jazzzooo
27477f51fd
[fix] brave.news
2023-09-15 22:00:09 +02:00
Bnyro
140125e641
[feat] implementation of the movie database
2023-09-15 19:41:37 +02:00
searxng-bot
4eebb25882
[translations] update from Weblate
...
4f055ccb89edaacb09ccb3e8934206640415d8db608c26db95be680792cd970949ad5db80921f948e3677f0ccb2dea9cd3a5038588e86b63e980b858a8660b5852633d292d25097b8ae9d7a20989ccd21a1c309f3fc3ee5e679e7420db0cf7327ffb1f542814e225f26236007bea3da4568a854d931781efe60016a6044200f1431198e61bb7acd04327a418412274159ed21b385914b774daff7b45ec84146f8707383602051b050d9ee48d6e7a20cee77e20720420f44b6
2023-09-15 09:12:13 +02:00
Justas Zabulionis
41ef73ca3a
[fix] rumble redirect
2023-09-14 19:21:21 +02:00
Justas Zabulionis
b701ee257b
[fix] habrahabr
2023-09-14 18:43:14 +02:00
Justas Zabulionis
be888810ba
[fix] pubmed content being None
2023-09-14 18:40:15 +02:00
Justas Zabulionis
cf734e6008
[fix] engine - pub.dev
2023-09-14 18:07:55 +02:00
Justas Zabulionis
92d39de410
[fix] solidtorrents redirects
2023-09-14 18:03:21 +02:00
Justas Zabulionis
cf8a6cf6db
[fix] solidtorrents pagination
2023-09-14 18:03:21 +02:00
Justas Zabulionis
8172f89075
[fix] solidtorrents
2023-09-14 18:03:21 +02:00
jazzzooo
74600c028d
[fix] engine - Crossref
...
Crossref was broken on result types journal-issue and component .. The old code
had lots of assumptions, and broke during parsing. Now the assumptions are more
explicit and checked them with the API.
2023-09-14 17:39:23 +02:00
Markus Heiser
ed6a5a01bb
[build] /static
2023-09-14 16:26:10 +02:00
Jinyuan Huang
0984b00b7e
[mod] UI smooth scroll added / CSS scroll-behavior
...
This PR improves the UX by making auto-scroll more smoother. The css is changed
so all the auto-scroll will be smoother but User-scroll will not be influenced.
The scroll-behavior CSS property sets the behavior for a scrolling box when
scrolling is triggered by the navigation or CSSOM scrolling APIs.[1]
[1] https://developer.mozilla.org/en-US/docs/Web/CSS/scroll-behavior
2023-09-14 16:26:10 +02:00
Bnyro
bf7af99ef2
[mod] make data.traits: add Odysee
2023-09-13 21:41:33 +02:00
Bnyro
3568a3cafb
[feat] odysee: implement fetch_traits for language support
2023-09-13 21:41:33 +02:00
Bnyro
09c61dabc9
[mod] odysee: time range support
2023-09-13 21:41:33 +02:00
jazzzooo
b98907e91f
[fix] engine - piped.music incorrect timestamps
2023-09-13 21:39:37 +02:00
jazzzooo
6039dbf211
[fix] engine - invidious thumbnails
2023-09-13 11:37:42 +02:00
jazzzooo
ed3819bdd5
add inv.tux.pizza
2023-09-13 11:12:01 +02:00
jazzzooo
122b4342eb
remove invidious.snopyta.org
2023-09-13 11:12:01 +02:00
jazzzooo
e2a5e183cc
[fix] settings.yml - change default invidious instances
2023-09-13 11:12:01 +02:00
jazzzooo
b2fd6304bf
[fix] engine - openstreetmap currency rendering
2023-09-13 10:56:52 +02:00
jazzzooo
976b3e21c4
[fix] engine - pypi redirect
2023-09-12 21:14:48 +02:00
jazzzooo
54a3e03b45
[fix] engine - openstreetmap currency matching
2023-09-12 20:57:05 +02:00
Bnyro
64d9587ac8
[feat] new engine: svgrepo
2023-09-12 20:38:36 +02:00
Markus Heiser
aa1453db0c
[build] /static
2023-09-12 20:30:50 +02:00
Markus Heiser
a987672340
[mod] improve handling of the hotkeys
...
- KeyboardEvent: keyCode property is depricated, replaced by key property [2]
- the check for ifDetailOpened is not necessary, because the hotkeys are not
only applicable to image-results, by example:
`!goi !go !scc hello`
- Key bindings like h for help are to be used in general (not only in vim-mode)
[1] https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/keyCode
[2] https://developer.mozilla.org/en-US/docs/Web/API/KeyboardEvent/key
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-12 20:30:50 +02:00
Jinyuan Huang
e6b160da62
[feature] key bindings: left & right arrow are always on
2023-09-12 20:30:50 +02:00
jazzzooo
b189578b6b
[fix] engine - brave
2023-09-12 11:31:43 +02:00
Bnyro
f182abd6f8
[mod] library of congress: fix engine
2023-09-11 19:42:31 +02:00
Bnyro
e73a6f5d14
[fix] engine deviantart: review of the result-scrapper
...
The deviantart site changed and hence deviantart is currently unusable.
2023-09-11 13:22:36 +02:00
Markus Heiser
1f7366060e
[build] /static
2023-09-11 12:27:56 +02:00
Markus Heiser
935aed7ca4
[feature] dark theme for code highlighter in the result list
...
Closes: https://github.com/searxng/searxng/issues/1354
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-11 12:27:56 +02:00
Alexandre Flament
d07c006aed
Replace chompjs with pure Python code
...
The new implementation is good enough for the current usage (brave)
2023-09-09 13:02:36 +02:00
Markus Heiser
8e45ac4271
[build] /static
2023-09-09 00:39:51 +02:00
Markus Heiser
15a3dbf7a0
[fix] preferences - layout of the "special queries" tab
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-09 00:39:51 +02:00
Bnyro
9e83c0dedc
[feat] engine: implementation of Yummly
...
Co-authored-by: Markus Heiser <markus.heiser@damarit.de>
2023-09-08 11:47:13 +02:00
Bnyro
a3d7e9c285
[mod] utils.py: add markdown_to_text helper function
2023-09-08 11:47:13 +02:00
searxng-bot
c62605b3e3
[translations] update from Weblate
...
304f6af5c76a4b6496
2023-09-08 09:20:19 +02:00
Markus Heiser
019bd97f4b
[fix] settings.yml - fix docs-links in the comments
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-06 11:35:39 +02:00
Hackurei
1f21ac7d62
[feat] engine: implementation of bilibili https://www.bilibili.com
2023-09-05 22:53:03 +02:00
Markus Heiser
696c35d2c3
[fix] engine - duckduckgo_images / determination of vqd value incorrect
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-05 22:24:51 +02:00
bonswouar
4fb26cd96d
[fix] engine duckduckgo weather api changes
2023-09-05 16:55:00 +02:00
Markus Heiser
01be9e0e20
[fix] engine: wikicommons - don't quoute ':|' in URL parameters
...
From [1]: It seems to be because of [2] For some reason it gets url encoded
twice, resulting in
- ``filetype%253Abitmap%257Cdrawing+birds`` instead of
- ``filetype:bitmap%7Cdrawing+birds``
[1] https://github.com/searxng/searxng/issues/2707
[2] https://github.com/searxng/searxng/blob/master/searx/engines/wikicommons.py#L43
Closes : #2707
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-03 18:45:39 +02:00
Markus Heiser
4f8895c6de
[fix] follow-up of 4da7003ae / add missing review from @Bnyro
...
[1] https://github.com/searxng/searxng/pull/2656#pullrequestreview-1607956209
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-09-02 09:26:45 +02:00
Hackurei
4da7003ae0
[feat] engine: implementation of odysee
2023-09-02 09:14:12 +02:00
Bnyro
9c4e9d3814
[feat] implementation of Wikimedia commons for images
2023-09-01 18:39:24 +02:00
searxng-bot
9fce459c7f
[translations] update from Weblate
...
979909accf40af3403
2023-09-01 09:29:07 +02:00
Jafar Farganlooj
7af66736c0
Create search-syntax.md fa
2023-08-30 16:39:10 +02:00
Jafar Farganlooj
26cd770cf7
[mod] Persian translation of the about page (fa_IR)
2023-08-30 16:13:44 +02:00
Alexandre Flament
faa4280e1a
[mod] bing: resolve redirect without additional requests
...
Remove the usage of searx.network.multi_requests
The results from Bing contains the target URL encoded in base64
See the u parameter, remove the first two character "a1", and done.
Also add a comment the check of the result_len / pageno
( from https://github.com/searx/searx/pull/1387 )
2023-08-29 07:39:06 +02:00
dalf
f57842b05d
Update searx.data - update_firefox_version.py
2023-08-29 07:21:46 +02:00
dalf
a5cd777350
Update searx.data - update_ahmia_blacklist.py
2023-08-29 07:21:21 +02:00
dalf
f361a10af6
Update searx.data - update_currencies.py
2023-08-29 07:20:42 +02:00
dalf
978062bdbe
Update searx.data - update_engine_descriptions.py
2023-08-29 07:20:06 +02:00
dalf
f59fd8dc9b
Update searx.data - update_wikidata_units.py
2023-08-29 07:17:44 +02:00
Markus Heiser
3294e7cfe4
[fix] reduce error logs related to X-Forwarded-For & X-Real-IP headers
...
To test start a `make run` instance and query multiple times for `ip` (the
answerer), you should see only once the error message(s):
ERROR searx.botdetection : X-Forwarded-For header is not set!
ERROR searx.botdetection : X-Real-IP header is not set!
[1] https://github.com/searxng/searxng-docker/discussions/174
[2] https://github.com/searxng/searxng-docker/pull/162#issuecomment-1692834457
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-28 08:05:17 +02:00
Bnyro
1428385d1c
[mod] preferences.py: add and use BooleanSetting for checkbox preferences
2023-08-27 21:23:43 +02:00
Bnyro
e5535ec078
[feat] settings: replace boolean select preferences with checkboxes
2023-08-27 21:23:43 +02:00
Bnyro
5a5cfc1930
[mod] random answerer: add random hex color generator
2023-08-27 18:59:54 +02:00
Markus Heiser
b0d2cd5ca9
[doc] add documentation of Mwmbl engine & autocompleter
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-27 17:25:26 +02:00
Bnyro
df71c24b20
[mod] autocomplete.py: add support for mwmbl completions
2023-08-27 17:25:26 +02:00
Bnyro
19300a5659
[mod] engine mwmbl: add link to official api docs
2023-08-27 17:25:26 +02:00
Alexandre Flament
97b1df1629
[mod] searx.network: memory optimization
...
Avoid to create a SSLContext in AsyncHTTPTransportNoHttp
See:
* 0f61aa58d6/httpx/_transports/default.py (L271)https://github.com/encode/httpx/issues/2298
2023-08-27 11:49:40 +02:00
Alexandre Flament
e16c007c22
[fix] openstreetmap engine
...
It seems there is an API change:
extratags can be either a dictionnary or None.
This commit avoid crash when extratags is None
Test query "!osm gare du nord"
2023-08-27 11:49:16 +02:00
Markus Heiser
0647f83a3e
[fix] google engine: don't overspecify the search query to Google
...
The method EngineTraits.get_region(..) returns engine's region string
that **best fits** to SearXNG's locale. This means it returns a
region (country) if only a language is set in the locale. By example the method
returns for a locale tag `es` a region `ES`.
Google's search parameter `cr` restricts search results to documents originating
in a particular country / in case of a locale tag (language) as described above,
this argument should be unset in the query send to Google.
Closes: https://github.com/searxng/searxng/issues/2672
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-26 07:47:07 +02:00
searxng-bot
7d806d1020
[translations] update from Weblate
...
a6c2d198838bc33a710ff21aa3c
2023-08-25 09:36:49 +02:00
Paul Braeuning
9069da051a
Merge pull request #2644 from dalf/update_httpx
...
Bump httpx 0.21.2 from to 0.24.1
2023-08-24 23:12:06 +02:00
Markus Heiser
4b42644579
[fix] engine google_video: google has changed the layout of the rsponse
...
Closes: https://github.com/searxng/searxng/issues/2664
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-22 08:34:04 +02:00
Alexandre Flament
b4e4cfc026
Bump httpx 0.21.2 from to 0.24.1
2023-08-21 22:05:12 +02:00
Markus Heiser
d52919cc3e
[mod] decrease the weighting of wikivoyage, wikibooks, wikisource and wikiversity
...
The search engines deliver hits for many search terms [1], but these are usually
not the focus of the user. In order to arrange these hits further down in the
list, their weighting is reduced.
[1] https://github.com/searxng/searxng/pull/2589#issuecomment-1670915089
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-21 21:52:40 +02:00
PlayDay
018b0a9322
[feat] engine: XPath configuration for pkg.go.dev
2023-08-19 19:19:31 +02:00
Bnyro
c59ae91b76
[feat] engine: implementation of mwmbl
2023-08-19 18:23:42 +02:00
searxng-bot
bb9e9ab773
[translations] update from Weblate
...
d8e4100b6b8e03f3bc
2023-08-18 19:22:13 +02:00
Markus Heiser
c741fc6f00
[mod] currency_convert: support for showing the answer source url
...
Show URL of the ddg-search page, not the URL of a (generic) Javascript. The
latter one is not usefull for the user.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-18 19:07:14 +02:00
Markus Heiser
e2744520f8
[mod] google: support for showing the answer source url
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-18 19:07:14 +02:00
Bnyro
5ec7df3480
[mod] engine duckduckgo definitions: support for answer source
2023-08-18 19:07:14 +02:00
Bnyro
64bc98b5fb
[mod] brave: support for showing the answer source url
2023-08-18 19:07:14 +02:00
Markus Heiser
6fc2ee514d
[build] /static
2023-08-18 19:07:14 +02:00
Bnyro
13d801b75d
[feat] results: show source of suggested answer
2023-08-18 19:07:14 +02:00
Markus Heiser
9100a48541
[mod] improve seekr engines and add documentation
...
Tis patch adds some more fields to the result items and changed paging to the
``nextResultSet`` given in seekr's JSON response.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-15 16:17:42 +02:00
Bnyro
2bab658d39
[feat] engine: implementation of seekr for news, images and videos
2023-08-15 16:17:42 +02:00
Markus Heiser
7052a1a7cf
[drop] engine archive.is / blocked by CAPTCHA [1]
...
Sadly archive.is is blocked by a CAPTCHA that can't be avoid (at least in a
XPath engine).
[1] https://github.com/searxng/searxng/issues/2643
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-14 08:50:13 +02:00
mrpaulblack
621b09b9b9
[fix] preferences: change engine table category colspan to 8
...
* this is a small fix to increase the colspan of the category in engine preferences from 7 to 8, since there was a column added
=> fixing a small fallout from 4731290317
2023-08-14 07:11:06 +02:00
ToxyFlog1627
f175574f37
[fix] typos in documentation & messages
2023-08-13 08:50:29 +02:00
Markus Heiser
c348427f07
[update] make pygments.less
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-11 10:54:50 +02:00
searxng-bot
99df7b84d1
[translations] update from Weblate
...
12c527bcdea46b66fcae3d8dcdfab6a59e351db1affcc94b682aa007c49bb6c
2023-08-11 09:17:56 +02:00
Bnyro
c76c999a1e
[feat] engine: implementation of anaconda
2023-08-10 23:51:07 +02:00
Bnyro
e25d1c7288
[feat] engine: implementation of German news, Tagesschau
...
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-10 20:27:54 +02:00
GeckoLinux
2256ba2ffb
[mod] settings.yml: move wolframalpha to categ general & disable by default
...
As suggested in comment [1] as a good compromise between [2] and [3].
This will help users that want Wolfram Alpha instant answers by default with
every query and don't mind the slowness. It will be disabled by default so it's
fast for all users, and those that need it can enable it as a user-configurable
option without requiring setting up their own instance to change the Wolfram
Alpha categorization.
[1] https://github.com/searxng/searxng/pull/813#issuecomment-1574851572
[2] https://github.com/searxng/searxng/pull/813#issuecomment-1020581765
[3] https://github.com/searxng/searxng/issues/966#issue-1171107451
2023-08-10 14:31:09 +02:00
Bnyro
834e1c3f12
[mod] engine lemmy: increase thumbnail quality to align with theme
2023-08-10 12:58:40 +02:00
Markus Heiser
c381fc001f
[mod] settings: remove lemmy from categ 'general' & enable by default
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-10 12:58:40 +02:00
Markus Heiser
fda111c0c9
[mod] engine lemmy: add more info fields to the result items
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-10 12:58:40 +02:00
Bnyro
224f2250ae
[feat] engine: support for lemmy communities, posts, comments and users
2023-08-10 12:58:40 +02:00
Bnyro
9f82c39610
[mod] engine google_play: raise error on unsupported category
2023-08-10 12:35:24 +02:00
Bnyro
0a99dc85b9
[mod] engine brave: raise error on unsupported category
2023-08-10 12:35:24 +02:00
Bnyro
4731290317
[mod] settings: show weight of each instance
2023-08-10 10:50:14 +02:00
Markus Heiser
733b795d53
[fix] make flask_babel.gettext() work in engine modules (L10n & threads)
...
incident:
flask_babel.gettext() does not work in the engine modules.
cause:
the request() and response() functions of the engine modules run in the
processor, whose search() method runs in a thread and in the threads the
context of the Flask app does not exist. The context of the Flask app is
needed by the gettext() function for the L10n.
Solution:
copy context of the Flask app into the threads. [1]
special case:
We cannot equip the search() method of the processors with the decorator [1],
because the decorator requires a context (Flask app) that does not yet exist
at the time of the initialization of the processors (the initialization of the
processors is part of the initialization of the Flask app).
[1] https://flask.palletsprojects.com/en/2.3.x/api/#flask.copy_current_request_context
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-09 13:27:43 +02:00
allendema_searxng_pi
c00c0c5434
[mod] remove discontinued petalsearch engines
2023-08-09 07:17:40 +02:00
Markus Heiser
b8352eca0c
[mod] brave engines: add fetch_traits() / improve language support
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-08 16:21:45 +02:00
Markus Heiser
460bbe5b81
[mod] implement brave (WEB) engine to replace XPath configuration
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-08 16:21:45 +02:00
Bnyro
d151497db3
[feat] engine: brave - support for news
2023-08-08 16:21:45 +02:00
Bnyro
cae06f2781
[feat] engine: brave - support for videos
2023-08-08 16:21:45 +02:00
Bnyro
73364e158e
[feat] engine: brave - support for images
2023-08-08 16:21:45 +02:00
Markus Heiser
bcaaae699f
[doc] fix some minor typos in the doc-strings of class EngineTraits
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-08 08:15:53 +02:00
Markus Heiser
a2108432f7
[fix] settings.yml: typo in engine name wikipecies -> wikispecies
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-07 10:35:44 +02:00
Markus Heiser
1d0abb7157
[doc] engine bt4g: add documentation to docs/dev/engines/online/
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-06 09:30:48 +02:00
Emilien Devos
0fc8f99ecc
[feat] new engine: bt4g added & enabled and disable by default btdigg
...
Disable btdigg because on most SearXNG instances, SearXNG is blocked by btdigg
due to cloudflare too many requests.
This impementation did not parse the HTML page because there is an API in
XML (RSS). The RSS feed provides fewer data like amount of seeders/leechers and
the files in the torrent file. It's a tradeoff for a "stable" engine as the XML
from RSS content will change way less than the HTML page.
Closes: https://github.com/searxng/searxng/issues/2553
2023-08-06 09:30:48 +02:00
Markus Heiser
dad16c0949
[mod] settings.yml: enable engines from Wikimedia Foundation
...
The Wikimedia wikis [1] engines provide good answers and have short response
times --> no reason to disable these enhgines by default. BTW: this patch adds
a (sub-) category ``wikimedia`` for the engines [1].
[1] https://meta.wikimedia.org/wiki/Wikimedia_wikis
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-04 19:06:50 +02:00
Markus Heiser
db522cf76d
[mod] engine: wikimedia - improve results, add addition settings & doc
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-04 19:06:50 +02:00
Bnyro
7d8c20c80d
[feat] new engine: wikispecies
2023-08-04 19:06:50 +02:00
searxng-bot
9a5e91ce84
[translations] update from Weblate
...
310daeb7201be8a264
2023-08-04 09:30:11 +02:00
Markus Heiser
1b030d4b41
[doc] engine: Yacy
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-03 19:58:51 +02:00
zutto
ca518c6803
add option to change yacy search mode
2023-08-03 19:58:51 +02:00
Markus Heiser
203f1f0928
[fix] engine piped: 'invalid content'
...
SearXNG does not allow a None value in the content field of a result item.
If the key (shortDescription, uploaderName) in the JSON response from piped
exists but is set to None, SearXNG ignores this result item::
DEBUG searx : result: invalid content: { .., 'content': None, ..}
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-03 16:23:36 +02:00
Markus Heiser
207fcc0c8c
[mod] engine piped: add paging support
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-03 16:23:36 +02:00
Markus Heiser
ef5831cd84
[mod] engine piped: split into two dedicated engiens for video & music
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-03 16:23:36 +02:00
Markus Heiser
7aa95d2d52
[doc] engine piped: add documentation to docs/dev/engines/online/
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-08-03 16:23:36 +02:00
Bnyro
636bfdac68
[feat] engine: implementation of Piped
2023-08-03 16:23:36 +02:00
dalf
8b4ba204b7
Update searx.data - update_engine_descriptions.py
2023-07-29 11:02:54 +02:00
dalf
d421385dc6
Update searx.data - update_wikidata_units.py
2023-07-29 11:02:23 +02:00
dalf
290f16207b
Update searx.data - update_currencies.py
2023-07-29 11:02:14 +02:00
dalf
fcd2013511
Update searx.data - update_firefox_version.py
2023-07-29 11:01:28 +02:00
dalf
cfa8169509
Update searx.data - update_ahmia_blacklist.py
2023-07-29 09:25:24 +02:00
searxng-bot
f45d1920d9
[translations] update from Weblate
...
ba4888c966ec8a8a280a7b701b3c0b34cbdb37cbd41c2
2023-07-29 08:15:21 +02:00
mrpaulblack
65d8b1a310
[fix] remove disabled: false from engine definitions in settings.yml
...
* setting disabled: false is not needed, since it is by default enabled
2023-07-22 18:19:01 +02:00
searxng-bot
51c531d450
[translations] update from Weblate
...
b7f1e9ae3c7c821e0e305f8480745a22afef0e2e4a7687ac
2023-07-22 17:47:05 +02:00
mrpaulblack
b477349824
[build] /static
2023-07-19 15:07:45 +02:00
Kiru
de5c1cedca
fix "#backToTop" button always being clickable
...
`pointer-events` never gets set to "none" when the button is hidden,
allowing you to click the button. And your mouse further changes it's
cursor to the pointer style.
2023-07-19 15:06:09 +02:00
searxng-bot
b7b184244d
[translations] update from Weblate
...
01350cf15f037a4d820a78ad73037743e656795c0ee18285c087c7fb6eb318c53b4a3d1fb3187499c12266462356a402c9a74b523d9f293868af8585261a2a72fcea15cf7685385eec0a37270130ddf7b93f96094a5cdcb39cba3939d973d937ce0762455c36ccab226ff7d48148a9ed840bc18951ffc22e394ec63e428c16a8218cf51e70260934c6244c2bb92dc5c143917957df1bf630d1c00dff0a6da54f12377e28b5b8ea78ec31e65f6c33b1fe393d390ca4f6b3530f8d6b6b67f2fc965f2d3f025ae2b8dc0bd4fb1ece768726a22ae2f2b5b8774f
2023-07-14 10:21:27 +02:00
Paolo Basso
cada89ee36
[feat] engine: re-enables z-library (zlibrary-global.se)
...
- re-enables z-library as the new domain zlibrary-global.se is now available
from the open web. The announcement of the domain:
https://www.reddit.com/r/zlibrary/comments/13whe08/mod_note_zlibraryglobalse_domain_is_officially/
It is an official domain, it requires to log in to the "personal" subdomain
only to download files, but the search works.
- changes the result template of zlibrary to paper.html, filling the appropriate fields
- implements language filtering for zlibrary
- implement zlibrary custom filters (engine traits)
- refactor and document the zlibrary engine
2023-07-07 21:36:51 +02:00
Hackurei
cb92767f19
[feat] enigine: add CrowdView forum search engine
2023-07-07 21:36:11 +02:00
searxng-bot
4a2f310da3
[translations] update from Weblate
...
152f20089dbf6b224ad4c00f
2023-07-07 21:13:47 +02:00
Markus Heiser
5720844fcd
[doc] rearranges Settings & Engines docs for better readability
...
We have built up detailed documentation of the *settings* and the *engines* over
the past few years. However, this documentation was still spread over various
chapters and was difficult to navigate in its entirety.
This patch rearranges the Settings & Engines documentation for better
readability.
To review new ordered docs::
make docs.clean docs.live
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-07-01 22:45:19 +02:00
searxng-bot
81c9a18456
[translations] update from Weblate
...
2238e87bc70d228a389c0c62656d9fcba9c9b116528b845fb8c50f2339f47c0fae0aa811c8216259
2023-06-30 11:49:07 +02:00
dalf
fbb72fc1f4
Update searx.data - update_engine_descriptions.py
2023-06-29 13:59:25 +02:00
Markus Heiser
87e7926ae9
[fix] engine: Anna's Archive - grep results from '.js-scroll-hidden' elements
...
The renderuing of the WEB page is very strange; except the firts position all
other positions of Anna's result page are enclosed in SGML comments. These
cooments are *uncommented* by some JS code, see query of the class
'.js-scroll-hidden' in Anna's HTML template [1].
[1] https://annas-software.org/AnnaArchivist/annas-archive/-/blob/main/allthethings/templates/macros/md5_list.html
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-29 09:32:57 +02:00
Markus Heiser
e2df6b77a3
[mod] engine: Anna's Archive - additionl settings (content, sort, ext)
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-29 09:32:57 +02:00
Markus Heiser
eafc2906f1
[mod] engine: Anna's Archive - fetch search arguments from search form
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-29 09:32:57 +02:00
Paolo Basso
7adb9090e5
[mod] engine: Anna's Archive - add language support
2023-06-29 09:32:57 +02:00
Paolo Basso
e5637fe7b9
[feat] engine: implementation of Anna's Archive
...
Anna's Archive [1] is a free non-profit online shadow library metasearch engine
providing access to a variety of book resources (also via IPFS), created by a
team of anonymous archivists [2].
[1] https://annas-archive.org/
[2] https://annas-software.org/AnnaArchivist/annas-archive
2023-06-29 09:32:57 +02:00
Markus Heiser
fd26f37073
[upd] make data.all
...
- ahmia_blacklist.txt
- currencies.json
- engine_descriptions.json
- engine_traits.json
- osm_keys_tags.json
- useragents.json
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-28 21:21:53 +02:00
Markus Heiser
efea962504
[fix] simple template: preferences - add missing icon_smal import
...
Related: https://github.com/searxng/searxng/commit/2149e88bdd64#r119535272
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-28 18:36:52 +02:00
Paolo Basso
401561cb58
[mod] engine torznab - refactor & option to hide links
...
- torznab engine using types and clearer code
- torznab option to hide torrent and magnet links.
- document the torznab engine
- add myself to authors
Closes: https://github.com/searxng/searxng/issues/1124
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-28 10:03:44 +02:00
Markus Heiser
da7c30291d
[fix] Google API changed
...
It seems that Google is rolling out a modified WEB API [1][2].
In the past there was only the UI language in the `hl` argument but nowadays it
seems a combination of the UI language and the "search region" is mixed in this
argument and the `gl` argument has been removed. I'm very surprised that google
is starting to mix the parameters of the UI with the parameters of the search
index.
This patch modifies the get_google_info(..) function. Beside Google-WEB this
function is also used by other Google services, here are some examples to test
region & language of ..
- Google-WEB: `!go dragon boat :en-CA`
- Google-News: `!gon dragon boat :en-CA`
- Google-Videos: `!gov bmw :en-CA`
- Goolge-Images `!goi bmw :en-CA`
- [1] https://github.com/searxng/searxng/issues/2515#issuecomment-1606294635
- [2] https://github.com/searxng/searxng/issues/2515#issuecomment-1607150817
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-26 18:28:09 +02:00
Markus Heiser
e8706fb738
[fix] engine & network issues / documentation and type annotations
...
This patch fixes some quirks and issues related to the engines and the network.
Each engine has its own network and this network was broken for the following
engines[1]:
- archlinux
- bing
- dailymotion
- duckduckgo
- google
- peertube
- startpage
- wikipedia
Since the files have been touched anyway, the type annotaions of the engine
modules has also been completed so that error messages from the type checker are
no longer reported.
Related and (partial) fixed issue:
- [1] https://github.com/searxng/searxng/issues/762#issuecomment-1605323861
- [2] https://github.com/searxng/searxng/issues/2513
- [3] https://github.com/searxng/searxng/issues/2515
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-25 13:58:26 +02:00
searxng-bot
2e4a435134
[translations] update from Weblate
...
9512b92aca08c51e56ad4f213ee419d6
2023-06-23 09:34:46 +02:00
Markus Heiser
86db08793b
[fix] implement a JSONEncoder for the json format
...
This patch implements a simple JSONEncoder just to fix #2502 / on the long term
SearXNG needs a data schema for the result items and a json generator for the
result list.
Closes: https://github.com/searxng/searxng/issues/2505
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-19 19:49:44 +02:00
Markus Heiser
fa1ef9a07b
[mod] move some code from webapp module to webutils module (no functional change)
...
Over the years the webapp module became more and more a mess. To improve the
modulaization a little this patch moves some implementations from the webapp
module to webutils module.
HINT: this patch brings non functional change
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-19 19:49:44 +02:00
searxng-bot
71b6ff07ca
[translations] update from Weblate
...
98f61c70a1679b93ebd1d574b28a1da356409bf0abc4916cb1900abeb48e84c4bf395e32c9c0a3c93f50d31e9da1c142
2023-06-16 09:20:43 +02:00
searxng-bot
1be27d5d83
[translations] update from Weblate
...
b40da1a3666ee7d41e0e8ead404b9937a627f9a1a234d2f8cc41f9b524651eacc37b06279a435ea140e0adad6833b14200f397ad7d3d4a97f7d713a4b1ec316004591a3acb3ac67cfe81dbc77882670f38882f3bc6df50476ca23c3b72f1ee09
2023-06-09 07:07:51 +00:00
Markus Heiser
22b13f4fa5
[mod] tools.Config.get(): add missing type annotations
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-05 14:07:19 +02:00
Markus Heiser
f3763d73ad
[mod] limiter: blocklist and passlist (ip_lists)
...
A blocklist and a passlist can be configured in /etc/searxng/limiter.toml::
[botdetection.ip_lists]
pass_ip = [
'51.15.252.168', # IPv4 of check.searx.space
]
block_ip = [
'93.184.216.34', # IPv4 of example.org
]
Closes: https://github.com/searxng/searxng/issues/2127
Closes: https://github.com/searxng/searxng/pull/2129
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-05 14:07:19 +02:00
Markus Heiser
f77807257b
[fix] engines: don't spam marginalia.nu with default settings
...
The engine configuration of marginalia [2][3][4][5] spams marginalia.nu with
requests from SearXNG instances [1]. It is not in the interest of SearXNG to
disturb other FOSS projects, so the engine will be removed::
- name: marginalia
engine: json_engine
shortcut: mar
categories: general
paging: false
# Key and license: https://www.marginalia.nu/marginalia-search/api/
# index: 0 popular, 1 blogs, 2 big_sites, 3 default, 4 experimental
search_url: https://api.marginalia.nu/ <insert your key here>/search/{query}?index=4&count=20
results_query: results
url_query: url
title_query: title
content_query: description
timeout: 1.5
disabled: true
about:
website: https://www.marginalia.nu/
official_api_documentation: https://api.marginalia.nu/
use_official_api: true
require_api_key: true
results: JSON
[1] https://github.com/searxng/searxng/issues/1673
[2] https://github.com/searxng/searxng/pull/1627
[3] https://github.com/searxng/searxng/issues/1620
[4] https://news.ycombinator.com/item?id=35874640
[5] d82a858491/code/services-satellite/api-service/src/main/java/nu/marginalia/api/svc/ResponseCache.java (L12-L20)
2023-06-05 08:23:17 +02:00
Markus Heiser
80aaef6c95
Merge pull request #2357 / limiter -> botdetection
...
The monolithic implementation of the limiter was divided into methods and
implemented in the Python package searx.botdetection. Detailed documentation on
the methods has been added.
The methods are divided into two groups:
1. Probe HTTP headers
- Method http_accept
- Method http_accept_encoding
- Method http_accept_language
- Method http_connection
- Method http_user_agent
2. Rate limit:
- Method ip_limit
- Method link_token (new)
The (reduced) implementation of the limiter is now in the module
searx.botdetection.limiter. The first group was transferred unchanged to this
module. The ip_limit contains the sliding windows implemented by the limiter so
far.
This merge also fixes some long outstandig issue:
- limiter does not evaluate the Accept-Language correct [1]
- limiter needs a IPv6 prefix to block networks instead of IPs [2]
Without additional configuration the limiter works as before (apart from the
bugfixes). For the commissioning of additional methods (link_toke), a
configuration must be made in an additional configuration file. Without this
configuration, the limiter runs as before (zero configuration).
The ip_limit Method implements the sliding windows of the vanilla limiter,
additionally the link_token method can be used in this method. The link_token
method can be used to investigate whether a request is suspicious. To activate
the link_token method in the ip_limit method add the following to your
/etc/searxng/limiter.toml::
[botdetection.ip_limit]
link_token = true
[1] https://github.com/searxng/searxng/issues/2455
[2] https://github.com/searxng/searxng/issues/2477
2023-06-03 06:00:15 +02:00
Markus Heiser
1a1ab34d9d
[fix] URL percent-encoding in translations fail in babel
...
Closes: https://github.com/searxng/searxng/issues/2482
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-02 20:30:41 +02:00
Markus Heiser
b867c39ce0
[build] /static
2023-06-02 19:05:43 +02:00
Markus Heiser
2149e88bdd
[mod] template preferences: split into elements (no functional change)
...
HINT: this patch has no functional change / it is the preparation for following
changes and bugfixes
Over the years, the preferences template became an unmanageable beast. To make
the source code more readable the monolith is splitted into elements. The
splitting into elements also has the advantage that a new template can make use
of them.
The reversed checkbox is a quirk that is only used in the prefereces and must be
eliminated in the long term. For this the macro 'checkbox_onoff_reversed' was
added to the preferences.html template. The 'checkbox' macro is also a quirk of
the preferences.html we don't want to use in other templates (it is an
input-checkbox in a HTML form that was misused for status display).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-02 19:05:43 +02:00
searxng-bot
789b43ab60
[translations] update from Weblate
...
5344314fee8fd9551ce31caffe75c53dca60af52f34b88f322d76a2643d8c98243a92e852bfc12dde2b5fb5f9f088420bdf81b4cf6a24c5d01bcea568c0209f8c629c610a4e4945d96bad166b0032d90366adaef2e4271bfc5856fd6790b5a6f6c9f92a9f5a6a35d4c8eeb327b8c06184e851dd40fa6006e877f4396c3bb1da7e66e6fae1cac4771949e994f8b18158265f8fb93e5088e1cf151100c51d169fae68ac961c336c5a188bda0d06a57c29a0c585b4de8ca9891817b2da46b2508aa3a5b1842fd826ab8a3938c4330cad6b2e997055fde6bd3d8ba5e0129e48fd248b0e7d3f1
2023-06-02 09:34:36 +02:00
Markus Heiser
80af38d37b
[mod] increase SUSPICIOUS_IP_WINDOW from one day to 30 days
...
In my tests I see bots rotating IPs (with endless IP lists). If such a bot has
100 IPs and has three attempts (SUSPICIOUS_IP_MAX = 3) then it can successfully
send up to 300 requests in one day while rotating the IP. To block the bots for
a longer period of time the SUSPICIOUS_IP_WINDOW, as the time period in which an
IP is observed, must be increased.
For normal WEB-browsers this is no problem, because the SUSPICIOUS_IP_WINDOW is
deleted as soon as the CSS with the token is loaded.
SUSPICIOUS_IP_WINDOW = 3600 * 24 * 30
Time (sec) before sliding window for one suspicious IP expires.
SUSPICIOUS_IP_MAX = 3
Maximum requests from one suspicious IP in the :py:obj:`SUSPICIOUS_IP_WINDOW`."""
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-01 16:00:49 +02:00
Markus Heiser
281e36f4b7
[fix] limiter: replace real_ip by IPv4/v6 network
...
Closes: https://github.com/searxng/searxng/issues/2477
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-01 15:51:14 +02:00
Markus Heiser
38431d2e14
[fix] correct determination of the IP for the request
...
For correct determination of the IP to the request the function
botdetection.get_real_ip() is implemented. This fonction is used in the
ip_limit and link_token method of the botdetection and it is used in the
self_info plugin.
A documentation about the X-Forwarded-For header has been added.
[1] https://github.com/searxng/searxng/pull/2357#issuecomment-1566211059
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-01 14:38:53 +02:00
Markus Heiser
b8c7c2c9aa
[mod] botdetection - improve ip_limit and link_token methods
...
- counting requests in LONG_WINDOW and BURST_WINDOW is not needed when the
request is validated by the link_token method [1]
- renew a ping-key on validation [2], this is needed for infinite scrolling,
where no new token (CSS) is loaded. / this does not fix the BURST_MAX issue in
the vanilla limiter
- normalize the counter names of the ip_limit method to 'ip_limit.*'
- just integrate the ip_limit method straight forward in the limiter plugin /
non intermediate code --> ip_limit now returns None or a werkzeug.Response
object that can be passed by the plugin to the flask application / non
intermediate code that returns a tuple
[1] https://github.com/searxng/searxng/pull/2357#issuecomment-1566113277
[2] https://github.com/searxng/searxng/pull/2357#discussion_r1208542206
[3] https://github.com/searxng/searxng/pull/2357#issuecomment-1566125979
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-01 14:38:53 +02:00
Markus Heiser
52f1452c09
[mod] limiter: ip_limt - monitore suspicious IPs
...
To intercept bots that get their IPs from a range of IPs, there is a
``SUSPICIOUS_IP_WINDOW``. In this window the suspicious IPs are stored for a
longer time. IPs stored in this sliding window have a maximum of
``SUSPICIOUS_IP_MAX`` accesses before they are blocked. As soon as the IP makes
a request that is not suspicious, the sliding window for this IP is droped.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-01 14:38:53 +02:00
Markus Heiser
9d7456fd6c
[fix] limiter.toml: botdetection.ip_limit turn off link_token by default
...
To activate the ``link_token`` method in the ``ip_limit`` method add the
following to your ``/etc/searxng/limiter.toml``::
[botdetection.ip_limit]
link_token = true
Related: https://github.com/searxng/searxng/pull/2357#issuecomment-1554116941
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-01 14:38:53 +02:00
Markus Heiser
66fdec0eb9
[mod] limiter: add config file /etc/searxng/limiter.toml
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-06-01 14:38:53 +02:00
Markus Heiser
1ec325adcc
[mod] limiter -> botdetection: modularization and documentation
...
In order to be able to meet the outstanding requirements, the implementation is
modularized and supplemented with documentation.
This patch does not contain functional change, except it fixes issue #2455
----
Aktivate limiter in the settings.yml and simulate a bot request by::
curl -H 'Accept-Language: de-DE,en-US;q=0.7,en;q=0.3' \
-H 'Accept: text/html'
-H 'User-Agent: xyz' \
-H 'Accept-Encoding: gzip' \
'http://127.0.0.1:8888/search?q=foo '
In the LOG:
DEBUG searx.botdetection.link_token : missing ping for this request: .....
Since ``BURST_MAX_SUSPICIOUS = 2`` you can repeat the query above two time
before you get a "Too Many Requests" response.
Closes: https://github.com/searxng/searxng/issues/2455
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29 14:54:56 +02:00
Markus Heiser
5226044c13
[mod] limiter: add random token to the limiter URL
...
By adding a random component in the limiter URL a bot can no longer send a ping
by request a static URL.
Related: https://github.com/searxng/searxng/pull/2357#issuecomment-1518525094
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29 14:54:56 +02:00
Markus Heiser
dba569462d
[mod] limiter: reduce request rates for requests without a ping
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-29 14:54:56 +02:00
dalf
c1b5ff7e1c
Update searx.data - update_engine_descriptions.py
2023-05-29 07:28:50 +02:00
dalf
2ba50d392e
Update searx.data - update_currencies.py
2023-05-29 07:28:18 +02:00
dalf
cb843ef13c
Update searx.data - update_engine_traits.py
2023-05-29 07:27:40 +02:00
dalf
512e001277
Update searx.data - update_firefox_version.py
2023-05-29 07:27:07 +02:00
dalf
f03ac9b152
Update searx.data - update_wikidata_units.py
2023-05-29 07:26:47 +02:00
dalf
e12e350f7f
Update searx.data - update_ahmia_blacklist.py
2023-05-29 07:26:20 +02:00
Markus Heiser
3ca97cf5e3
[fix] simple theme: move engine alerts in case of no results into sidebar
...
If there were no results but errors in the engines then the error dialogs of the
engines was displayed in the result list.
With the new design errors of the engines should only be displayed in the
sidebar and at the same time duplications of the (template) code will be
avoided.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-28 12:19:32 +02:00
mrpaulblack
60b94dfdca
[build] /static
2023-05-28 12:19:32 +02:00
mrpaulblack
f087959b02
[mod] simple theme: build design for details (collapsables)
...
* set border top and bottom on sidebar collasables
* inrease peading on summary so its easier to click on mobile
* remove margins and add flex wrapper to normalize elements in sidebar
2023-05-28 12:19:32 +02:00
Markus Heiser
b7e315563d
[mod] simple theme: collaps/expand elements in the sidebar
...
Make elements in the sidebar collapse able. Except infoboxes all elements in
the sidebar are collapsed by default.
By folding out the sidebar elements, the UI looks less cluttered. Especially on
small devices like smartphones, where the sidebar is above the results list, the
UX should be improved [1].
[1] https://github.com/searxng/searxng/issues/2140
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-28 12:19:32 +02:00
searxng-bot
e8054026fb
[translations] update from Weblate
...
69171f122caaed0a84d1702b65cc6eb8e0ab338323e87f1514f0fc6b5b7c7b7dc725b38d
2023-05-26 07:08:21 +00:00
Markus Heiser
ebe22a4319
[fix] typo: dues --> does
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-22 13:18:22 +02:00
Markus Heiser
bc647fabaf
[fix] ClientPref - don't raise exception if Accept-Language is invalid
...
If the Accept-Language header [1] is set but empty or holds a value that is
unknown to babel, an excpetion is raised::
$ curl --header 'Accept-Language: xyz' 'http://127.0.0.1:8888/search?q=foo '
...
Traceback (most recent call last):
File "searx/preferences.py", line 335, in from_http_request
return cls(locale=pairs[0][0])
IndexError: list index out of range
[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Language
Reported by: @Eolien55 in https://github.com/searxng/searxng/issues/2434#issuecomment-1556199789
Closes: https://github.com/searxng/searxng/issues/2434
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-22 12:38:59 +02:00
Markus Heiser
bab730c8a8
Merge pull request #2446 from searxng/translations_update
...
Update translations
2023-05-19 19:59:58 +02:00
pankaj
4900c091a6
use logger.warning
...
logger.warn() is depricated.
logger.warning is already being used in some files.
2023-05-19 19:35:29 +05:30
searxng-bot
23b53a03c2
[translations] update from Weblate
...
2eeec66c87058e51
2023-05-19 07:07:56 +00:00
Markus Heiser
007a615ffa
[mod] donation_url: disable by default
...
SearXNG's donation campaign has been ended.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-15 09:19:17 +02:00
Markus Heiser
6625b6193e
Merge pull request #2420 from searxng/translations_update
...
Update translations
2023-05-13 06:06:07 +02:00
Markus Heiser
caebd297e9
[fix] engine ddg: minor change in the API of ddg
...
Closes: https://github.com/searxng/searxng/issues/2419
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-12 18:58:49 +02:00
searxng-bot
f5c407eaa0
[translations] update from Weblate
...
df7e1be37ae9877ec2fe5131
2023-05-12 07:07:40 +00:00
Markus Heiser
a60851bd59
[fix] version format string generated by 'git show'
...
Newer versions of git [1] do no longer support a format string that includes a minus
to remove leading zeros [2]. The format string '%Y.%m.%d' is more version rod.
[1] https://github.com/searxng/searxng/issues/2413#issuecomment-1542320387
[2] https://github.com/searxng/searxng/pull/2122/files
Closes: https://github.com/searxng/searxng/issues/2413
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-05-10 18:33:45 +02:00
Markus Heiser
db95c4713b
Merge pull request #2406 from searxng/dependabot/npm_and_yarn/searx/static/themes/simple/master/grunt-contrib-cssmin-5.0.0
...
Bump grunt-contrib-cssmin from 4.0.0 to 5.0.0 in /searx/static/themes/simple
2023-05-06 05:54:15 +02:00
dependabot[bot]
673837f83e
Bump grunt-contrib-cssmin in /searx/static/themes/simple
...
Bumps [grunt-contrib-cssmin](https://github.com/gruntjs/grunt-contrib-cssmin ) from 4.0.0 to 5.0.0.
- [Release notes](https://github.com/gruntjs/grunt-contrib-cssmin/releases )
- [Changelog](https://github.com/gruntjs/grunt-contrib-cssmin/blob/main/CHANGELOG )
- [Commits](https://github.com/gruntjs/grunt-contrib-cssmin/compare/v4.0.0...v5.0.0 )
---
updated-dependencies:
- dependency-name: grunt-contrib-cssmin
dependency-type: direct:development
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-05-05 07:57:12 +00:00
searxng-bot
54c4d08167
[translations] update from Weblate
...
7033661355d82b963911fe3581b6ebd16655ac633b9cccb851601c0b
2023-05-05 07:07:41 +00:00
Markus Heiser
823c490c84
[mod] limiter: block requests from PetalBot
...
Block requests from PetalBlock. Normally robots.txt is enough to stop
PetalBlock from making requests [1]. However, if SearXNG is offered below a
path (example.org/search), then the robots.txt is not available in the root
paths of the domain / subdomain.
[1] https://webmaster.petalsearch.com/site/petalbot
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-30 09:49:26 +02:00
Markus Heiser
7b43f98ebb
Merge pull request #2382 from searxng/translations_update
...
Update translations
2023-04-30 08:11:00 +02:00
Markus Heiser
e0c8e1923d
Merge pull request #2390 from searxng/update_data_update_wikidata_units.py
...
Update searx.data - update_wikidata_units.py
2023-04-30 07:16:31 +02:00
Markus Heiser
d63dbb10fc
Merge pull request #2391 from searxng/update_data_update_firefox_version.py
...
Update searx.data - update_firefox_version.py
2023-04-30 07:15:55 +02:00
Markus Heiser
e9fdfab76e
Merge pull request #2392 from searxng/update_data_update_currencies.py
...
Update searx.data - update_currencies.py
2023-04-30 07:15:28 +02:00
Markus Heiser
836827517d
Merge pull request #2393 from searxng/update_data_update_ahmia_blacklist.py
...
Update searx.data - update_ahmia_blacklist.py
2023-04-30 07:14:53 +02:00
Markus Heiser
cfc01ea068
Merge pull request #2394 from searxng/update_data_update_engine_traits.py
...
Update searx.data - update_engine_traits.py
2023-04-30 07:14:23 +02:00
dalf
c2fbace534
Update searx.data - update_engine_descriptions.py
2023-04-29 00:24:27 +00:00
dalf
4f31ab7d4b
Update searx.data - update_engine_traits.py
2023-04-29 00:14:09 +00:00
dalf
df4cc070ec
Update searx.data - update_ahmia_blacklist.py
2023-04-29 00:13:21 +00:00
dalf
5b93f97fb2
Update searx.data - update_currencies.py
2023-04-29 00:13:19 +00:00
dalf
7c90a6a222
Update searx.data - update_firefox_version.py
2023-04-29 00:13:09 +00:00
dalf
4336f70b59
Update searx.data - update_wikidata_units.py
2023-04-29 00:13:08 +00:00
searxng-bot
274fcd46b6
[translations] update from Weblate
...
b6658877d7a3917b879248ad6ccafe4ed202aed8
2023-04-28 07:08:09 +00:00
Markus Heiser
45529f51a1
Merge pull request #2347 from return42/mod-lang-detection
...
If language recognition fails use the Accept-Language
2023-04-25 15:46:26 +02:00
Jakub Łukasiewicz
7da47edd93
[mod] external bang: go to main instead of search page when query is empty
...
Closes : #2368
2023-04-25 15:02:34 +02:00
Markus Heiser
4080eca3dd
[build] /static
2023-04-21 13:38:29 +02:00
searxng-bot
b41e47eea6
[translations] update from Weblate
...
72d42638560e18857370b02620946697
2023-04-21 07:07:58 +00:00
Markus Heiser
9b575a997b
[fix] doc of locales.get_engine_locale() / zh-classical is missleading
...
Wikipedia's zh-classical is not zh_Hant (see doc-string of engines.wikipedia).
Fixed the example in the doc-string of locales.get_engine_locale() to 'zh_TW'.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-17 08:48:57 +02:00
Markus Heiser
f1b6351ae1
[fix] engine: google play movies
...
Closes: https://github.com/searxng/searxng/pull/1746
Closes: https://github.com/searxng/searxng/issues/1599
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-16 19:15:44 +02:00
Marc Abonce Seguin
be1f6ee1c4
Update searx.data - update_engine_traits.py
2023-04-16 08:40:44 +02:00
Markus Heiser
8adbc4fcec
[mod] settings.yml: enable language detection by default_lang (auto)
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15 22:24:59 +02:00
Markus Heiser
d5ecda9930
[mod] move language recognition to get_search_query_from_webapp
...
To set the language from language recognition and hold the value selected by the
client, the previous implementation creates a copy of the SearchQuery object and
manipulates the SearchQuery object by calling function replace_auto_language().
This patch tries to implement a similar functionality in a more central place,
in function get_search_query_from_webapp() when the SearchQuery object is build
up.
Additional this patch uses the language preferred by the client, if language
recognition does not have a match / the existing implementation does not care
about client preferences and uses 'all' in case of no match.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15 22:23:33 +02:00
Markus Heiser
c03b0ea650
[mod] add a Preferences.client property to store client prefs
...
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2023-04-15 21:41:57 +02:00
