kamp/searxng
1
0
Fork 0
mirror of https://github.com/searxng/searxng.git synced 2024-11-09 14:40:11 +01:00
Code
6,444 Commits 6 Branches 17 Tags 130 MiB
7fab950284
Commit Graph

1344 Commits

Author SHA1 Message Date
Markus Heiser
0be0e63117 [fix] demo_online.py - fixed typo 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-07-25 20:04:00 +02:00
Emilien Devos
5fb2071cb2 [fix] google & youtube - set EU consent cookie 
This change the previous bypass method for Google consent using
``ucbcb=1`` (6face215b8) to accept the consent using ``CONSENT=YES+``.

The youtube_noapi and google have a similar API, at least for the consent[1].

Get CONSENT cookie from google reguest::

    curl -i "https://www.google.com/search?q=time&tbm=isch" \
         -A "Mozilla/5.0 (X11; Linux i686; rv:102.0) Gecko/20100101 Firefox/102.0" \
         | grep -i consent
    ...
    location: https://consent.google.com/m?continue=https://www.google.com/search?q%3Dtime%26tbm%3Disch&gl=DE&m=0&pc=irp&uxe=eomtm&hl=en-US&src=1
    set-cookie: CONSENT=PENDING+936; expires=Wed, 24-Jul-2024 11:26:20 GMT; path=/; domain=.google.com; Secure
    ...

PENDING & YES [2]:

  Google change the way for consent about YouTube cookies agreement in EU
  countries. Instead of showing a popup in the website, YouTube redirects the
  user to a new webpage at consent.youtube.com domain ...  Fix for this is to
  put a cookie CONSENT with YES+ value for every YouTube request

[1] https://github.com/iv-org/invidious/pull/2207
[2] https://github.com/TeamNewPipe/NewPipeExtractor/issues/592

Closes: https://github.com/searxng/searxng/issues/1432
 2022-07-25 13:27:06 +02:00
Markus Heiser
4231a5770b [fix] sjp engine - convert enginename to a latin1 compliance name 
The engine name is not only a *name* its also a identifier that is used in
logs, HTTP headers and more.  Unicode characters in the name of an engine could
cause various issues.

Closes: https://github.com/searxng/searxng/issues/1544
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-07-24 21:10:55 +02:00
james-still
2516e21c58 [fix] emojipedia - update XPath to be relative 2022-07-24 19:14:26 +02:00
Markus Heiser
1540891561 [fix] engine tineye: handle 422 response of not supported img format 
Closes: https://github.com/searxng/searxng/issues/1449
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-07-23 16:00:58 +02:00
Markus Heiser
4e05197444
Merge pull request #1475 from return42/Emojipedia 
[mod] Add engine for Emojipedia
 2022-07-15 09:30:40 +02:00
Jay
10edcbe3c2 [mod] Add engine for Emojipedia 
Emojipedia is an emoji reference website which documents the meaning and
common usage of emoji characters in the Unicode Standard.  It is owned by Zedge
since 2021. Emojipedia is a voting member of The Unicode Consortium.[1]

Cherry picked from @james-still [2[3] and slightly modified to fit SearXNG's
quality gates.

[1] https://en.wikipedia.org/wiki/Emojipedia
[2] 2fc01eb20f
[3] https://github.com/searx/searx/pull/3278
 2022-07-15 09:26:44 +02:00
Alexandre Flament
44f2eb50a5
Merge pull request #1219 from dalf/follow_bing_redirect 
bing.py: remove redirection links
 2022-07-10 18:06:22 +02:00
Emilien Devos
6face215b8 bypass google consent with ucbcb=1 2022-07-09 21:33:24 +00:00
Alexandre Flament
a1e8af0796 bing.py: resolve bing.com/ck/a redirections 
add a new function searx.network.multi_requests to send multiple HTTP requests at once
 2022-07-08 22:02:21 +02:00
Markus Heiser
970a69012b [fix] engine z-zlibrary https URL 
before this patch:

    DEBUG   searx.engines.z-library : using base_url: https:https://de1lib.org

with this patch URL is fixed to:

    DEBUG   searx.engines.z-library : using base_url: https://de1lib.org

Closes: https://github.com/searxng/searxng/issues/1435
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-07-05 22:27:55 +02:00
ta
14756a2674 [mod] Adds Lingva translate engine 
Add the lingva engine (which grabs data from google translate).  Results from
Lingva are added to the infobox results.
 2022-07-04 19:06:45 +02:00
Markus Heiser
5831c15b49 [fix] engines/openstreetmap.py typo: user_langage --> user_language 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-07-02 16:51:25 +02:00
Alexandre Flament
6716c6b0c3 openstreetmap engine: return the localized named. 
For example: display "Tokyo" instead of "東京都" when the language is English.
 2022-07-02 16:51:25 +02:00
ta
8883aed132 [fix] google play apps engine: implement engines/google_play_apps.py 2022-06-18 16:02:39 +02:00
Alexandre Flament
5bcbec9b06 Fix: use sys.modules.copy() to avoid RuntimeError 
use sys.modules.copy() to avoid "RuntimeError: dictionary changed size during iteration"
see https://github.com/python/cpython/issues/89516
and https://docs.python.org/3.10/library/sys.html#sys.modules

close https://github.com/searxng/searxng/issues/1342
 2022-06-18 07:39:46 +02:00
Alexandre Flament
2455f1d06a
Merge pull request #1308 from allendema/add-yep-com-json 
[enh] Add yep.com via json_engine
 2022-06-12 11:09:04 +02:00
Allen
fd9a13a3e5 [enh] Initial no paging support for Yep.com 
Upstream example query:
https://yep.com/web?q=test

https://yep.com/about
 2022-06-11 14:17:44 +02:00
Alexandre Flament
cd2dd5dd55 Wikidata engine: ignore dummy entities 
Close #641
 2022-06-11 11:09:21 +02:00
Alexandre Flament
d068b67a71 Wikidata engine: minor change of the SPARQL request 
The engine can be slow especially when the query won't return any answer.
See https://www.mediawiki.org/wiki/Wikidata_Query_Service/User_Manual/MWAPI#Find_articles_in_Wikipedia_speaking_about_cheese_and_see_which_Wikibase_items_they_correspond_to

Related to #1290
 2022-06-11 10:50:11 +02:00
Markus Heiser
2de007138c [fix] prepare for pylint 2.14.0 
Remove issue reported by Pylint 2.14.0:

- no-self-use: has been moved to optional extension [1]
- The refactoring checker now also raises 'consider-using-generator' messages
  for max(), min() and sum(). [2]

.pylintrc:
  - <option name>-hint has been removed since long, Pylint 2.14.0 raises an
    error on invalid options
  - bad-continuation and bad-whitespace have been removed [3]

[1] https://pylint.pycqa.org/en/latest/whatsnew/2/2.14/summary.html#removed-checkers
[2] https://pylint.pycqa.org/en/latest/whatsnew/2/2.14/full.html#what-s-new-in-pylint-2-14-0
[2] https://pylint.pycqa.org/en/latest/whatsnew/2/2.6/summary.html#summary-release-highlights

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-06-03 15:41:52 +02:00
Allen
43dc9eb7d6 [enh] Initial Petalsearch Images support 
Upstream example query:

  https://petalsearch.com/search?query=test&channel=image&ps=50&pn=1&region=de-de&ss_mode=off&ss_type=normal

Depending on locale it will internally use some/all results from other
engines. See:

  https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/#general-indexing-search-engines
 2022-06-02 14:32:37 +02:00
Émilien Devos
06cb15cbf7
Reflect the real world parameter from settings.yml 2022-05-10 20:44:35 +00:00
Markus Heiser
4326009d00 [format.python] based on bugfix in 9ed626130 2022-05-07 18:23:10 +02:00
capric98
8c7e6cc983 [fix] FutureWarning from lxml 
Just in case if content is None, the original code will skip extract_text(), and
just append the None value to 'content'. So just add allow_none=True, and this
will return None without raising a ValueError in extract_text().
 2022-04-22 16:09:36 +02:00
Alexandre Flament
bbf13a4657
Merge pull request #1101 from allendema/pass-cookies-from-settings 
[enh] Allow passing headers/cookies from settings.yml
 2022-04-17 11:37:07 +02:00
Allen
dae8a08089
[fix[ Update only cookies/headers 2022-04-17 11:29:23 +02:00
Allen
67fb6fba84
[lint] Remove whitespace 
From GH GUI
 2022-04-17 10:42:25 +02:00
Allen
15862ebc35
[mod] Pass desired ebay domain in settings 
https://www.ebay.de
https://www.ebay.com
htttps://www.ebay.es

etc
 2022-04-16 19:10:35 +02:00
Allen
155333f625
[enh] Allow passing headers/cookies from settings.yml 
Example:

   - engine: xpath
   - search_url: example.org
   - headers: {'example_header': 'example_header'}
   - cookies: {'safesearch': 'off'}
 2022-04-16 17:42:04 +02:00
Alexandre Flament
c474616642
Merge pull request #1071 from return42/fix-lang-dailymotion 
[fix] dailymotion engine: filter by language & country
 2022-04-16 11:54:49 +02:00
Alexandre Flament
1a82e79b50 dailymotion: send valid value for the language parameter 2022-04-16 09:27:34 +02:00
Markus Heiser
3bb62823ec [fix] dailymotion engine: filter by language & country 
- fix the issue of fetching more the 7000 *languages*
- improve the request function and filter by language & country
- implement time_range_support & safesearch
- add more fields to the response from dailymotion (allow_embed, length)
- better clean up of HTML tags in the 'content' field.

This is more or less a complete rework based on the '/videos' API from [1].
This patch cleans up the language list in SearXNG that has been polluted by the
ISO-639-3 2 and 3 letter codes from dailymotion languages which have never been
used.

[1] https://developers.dailymotion.com/tools/

Closes: https://github.com/searxng/searxng/issues/1065
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-04-16 09:27:34 +02:00
Jabster28
9eb1b04f48
change "Wolfram|Alpha" to "Wolfram Alpha" in search results 2022-04-12 10:37:33 +01:00
Alexandre Flament
592cea0e5e
Merge pull request #1030 from austinhuang0131/master 
(feat) add jisho.org
 2022-04-09 18:57:20 +02:00
Alexandre Flament
74c7aee9ec jisho : code refactoring 2022-04-09 18:01:57 +02:00
Austin Huang
19fa0095a0
(fix) satisfy the linter, and btw reduce timeout 2022-04-01 09:23:24 -04:00
Austin Huang
a399248f56
update jisho.py according to suggestions 2022-04-01 09:18:19 -04:00
Alexandre FLAMENT
f00cdb5e51 bing engine: _fetch_supported_languages: don't use the language code as a country 
ref #1029
 2022-03-31 20:03:34 +00:00
Austin Huang
934ae4e086
(feat) add jisho.org 
Closes #1016
 2022-03-31 14:45:39 -04:00
Alexandre Flament
378b29be2f fix startpage: update XPath in _fetch_supported_languages 2022-03-19 14:16:37 +01:00
Markus Heiser
53b5a804e2 [fix] engine mediathekviewweb: replace http links by https 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-03-07 19:49:16 +01:00
Markus Heiser
20f4538e13 [fix] engine: Semantic Scholar (Science) // rework & fix 
Closes: https://github.com/searxng/searxng/issues/939
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-03-05 11:53:41 +01:00
Markus Heiser
8d937179ab
Merge pull request #913 from return42/add-artwork 
[mod] add artwork to mixcloud & soundcloud engines
 2022-02-21 22:24:40 +01:00
Markus Heiser
b08b81b434 [mod] bandcamp & genius: in result set img_src instead thumbnail 
Suggested-by: @dalf https://github.com/searxng/searxng/pull/900#issuecomment-1046009057
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-02-21 22:12:07 +01:00
Markus Heiser
bded1ee280 [fix] genius: add player an avoid exceptional programming 
Add player:

- The players are just playing 30sec from the title.  Some of the player will be
  blocked because of a cross-origin request and some players will link to apple
  when you press the play button.

Avoid exceptions and (and BTW improve results)

-  ERROR   searx.engines.genius          : list index out of range

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-02-21 22:12:07 +01:00
Markus Heiser
36aee70c24
Merge pull request #910 from tiekoetter/fix-909 
[fix] google images engine: Fix 'scrap_img_by_id' function
 2022-02-20 18:29:50 +01:00
Markus Heiser
2921d3cd17 [mod] add artwork to mixcloud & soundcloud engines 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-02-19 21:59:12 +01:00
Markus Heiser
4a28b593c2 [fix] google images engine: Fix 'scrap_img_by_id' function 
The 'scrap_img_by_id' function didn't return any longer anything useful.  This
fix allows the google images engine to present the full source image instead of
only the thumbnail.

The function scrap_img_by_id() is rpelaced by a fully rewrite to parse image
URLs by a regular expression. The new function parse_urls_img_from_js(dom)
returns a mapping of data-id to image URL.

Closes: https://github.com/searxng/searxng/issues/909
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2022-02-19 14:33:56 +01:00
Alexandre Flament
ace5401632
Merge pull request #900 from return42/fix-883 
[fix] bandcamp: fix itemtype (album|track) and exceptions
 2022-02-19 13:42:53 +01:00