kamp/searxng
1
0
Fork 0
mirror of https://github.com/searxng/searxng.git synced 2024-11-17 01:50:10 +01:00
Code
3,638 Commits 6 Branches 17 Tags 132 MiB
230a5ecd04
Commit Graph

975 Commits

Author SHA1 Message Date
Markus Heiser
16f8ec894a [fix] revise google images engine 
this commit is picked from #1985
 2020-07-07 21:59:15 +02:00
Markus Heiser
410c2f903d [fix] revise google engine 
this commit is picked from #1985
 2020-07-07 21:50:59 +02:00
Markus Heiser
8d318ee142
Merge branch 'master' into gigablast 2020-06-29 16:09:59 +00:00
Sophie Tauchert
71db7b1238
Fix YaCy text results returned as images 2020-06-29 14:48:56 +02:00
Noémi Ványi
93cbd85b8a
Merge branch 'master' into duckduckgo_correction 2020-06-28 20:28:12 +02:00
Markus Heiser
5fac6cffa2
Merge branch 'master' into gigablast 2020-06-26 08:09:33 +00:00
Markus Heiser
5293e58032 [fix] yahoo engine - changed content_xpath 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-06-25 21:45:42 +02:00
Markus Heiser
223430ff30
Merge branch 'master' into gigablast 2020-06-16 07:36:44 +00:00
Adam Tauber
32f7877235 [fix] resolve flickr_noapi encoding issues 2020-06-15 19:15:24 +02:00
Gordon Quad
385e9b5c9e add correction support for duckduckgo 2020-06-13 22:43:10 +01:00
Markus Heiser
ee0da61cbb
Merge branch 'master' into gigablast 2020-06-10 04:54:57 +00:00
Adam Tauber
aa7c043ff4 [fix] resolve pep8 errors 2020-06-10 00:34:57 +02:00
Adam Tauber
29960aa1d9 [enh] add official site link to the top of the infobox - closes #1644 2020-06-09 23:49:13 +02:00
Adam Tauber
6c06286251 [enh] add length and author details to youtube videos 
closes #775
 2020-06-09 20:31:51 +02:00
Adam Tauber
2c6531b233 [enh] add routing directions to osm search - closes #254 2020-06-09 17:18:44 +02:00
Markus Heiser
74135007eb
Merge branch 'master' into gigablast 2020-06-01 06:23:45 +00:00
Noémi Ványi
e3282748d0 add display_error_messages option to engine settings 
A new option is added to engines to hide error messages from users. It
is called `display_error_messages` and by default it is set to `True`.
If it is set to `False` error messages do not show up on the UI.

Keep in mind that engines are still suspended if needed regardless of
this setting.

Closes #1828
 2020-05-31 19:17:48 +02:00
Markus Heiser
ee5d2b319b [fix] gigablast requires a random extra parameter 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-05-29 12:48:58 +02:00
Markus Heiser
a18760b322 [fix] revise of the gigablast engine (WIP) 
The gigablast API has changed and seems to have some quirks, this is the first
revise.  More work (hacks) are needed.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-05-28 19:24:03 +02:00
Markus Heiser
57c7b90edd [fix] gigablast does no longer support *supported_languages_url* 
Since there are zero results, we can remove it:

    $ make engines.languages
    fetch languages ..
    ...
    fetched 0 languages from engine gigablast

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-05-28 15:21:01 +02:00
Markus Heiser
de179ecc5b [fix] remove debug print from commit e5305f8 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-05-28 15:17:59 +02:00
Markus Heiser
9302d1fc17
Merge branch 'master' into master 2020-04-20 12:44:50 +00:00
Noémi Ványi
fcb44c6542
Merge branch 'master' into fix_startpage_ValueError_on_spanish_datetime 2020-04-15 23:33:16 +02:00
HLFH
3a26093c46
Remove discontinued faroo engine 2020-04-15 16:46:15 +01:00
Spühler Stefan
4f90fb6a92 [Fix] Startpage ValueError on Spanish date format 
datetime.parser.parse() does not know the Spanish date format which
leads to a ValueError. Fixes #1870

Traceback (most recent call last):
  File "/usr/local/searx/searx/search.py", line 160, in search_one_http_request_safe
    search_results = search_one_http_request(engine, query, request_params)
  File "/usr/local/searx/searx/search.py", line 97, in search_one_http_request
    return engine.response(response)
  File "/usr/local/searx/searx/engines/startpage.py", line 102, in response
    published_date = parser.parse(date_string, dayfirst=True)
  File "/usr/local/searx/searx-ve/lib/python3.6/site-packages/dateutil/parser/_parser.py", line 1358, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/usr/local/searx/searx-ve/lib/python3.6/site-packages/dateutil/parser/_parser.py", line 649, in parse
    raise ValueError("Unknown string format:", timestr)
ValueError: ('Unknown string format:', '24 Ene 2013')
 2020-03-09 09:31:20 +01:00
Markus Heiser
ad7a6e6e10 bugfix(!biv) : bing-video do not like "older" User-Agents 
When selecting other languages than 'en', bing-video did not handle the language
correct and gave very bad results.  Since User-Agent is normaly rotated in
searx, the behavior of a !biv search was unpredictable and paging was broken.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-01 17:28:59 +01:00
Markus Heiser
1c853f9573 bing_news: parital rollback of c89c05bc 
The bing_news bug (discussed in #1838) was caused by wrong language tags, which
was fixed e0c99d9d / no need to change the bing_news search string.

closes: https://github.com/asciimoo/searx/issues/1838

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-01 11:07:59 +01:00
Markus Heiser
e0c99d9dcb bugfix: fetch_supported_languages bing, -news, -videos, -images 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-03-01 08:01:36 +01:00
Markus Heiser
c89c05bceb bugfix: google-news and bing-news has changed the language parameter 
closes: https://github.com/asciimoo/searx/issues/1838

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2020-02-25 18:44:28 +01:00
piplongrun
f0684a5bb5
Add eTools engine 2020-02-12 23:58:50 +01:00
Noémi Ványi
99435381a8 [enh] introduce private engines 
This PR adds a new setting to engines named `tokens`.
It expects a list of tokens which lets searx validate
if the request should be accepted or not.
 2020-02-08 11:47:39 +01:00
frankdelange
db9d7d47bd Fix double-encode error (fixes #1799) 2020-01-07 21:41:43 +01:00
Adam Tauber
17b6faa4c3 [fix] pep8 2020-01-02 22:38:12 +01:00
Adam Tauber
ad5bb994b1 [fix] add py3 compatibility 2020-01-02 22:30:18 +01:00
Adam Tauber
1e6253ce16 [fix] handle empty response 2020-01-02 22:29:55 +01:00
Adam Tauber
86a378bd01 [fix] handle missing thumbnail 2020-01-02 22:29:28 +01:00
Adam Tauber
2dc2e1e8f9 [fix] skip invalid encoded attributes 2020-01-02 22:29:10 +01:00
Adam Tauber
2292e6e130 [fix] handle missing result size 2020-01-02 22:28:47 +01:00
Markus Heiser
36e72a4619
Merge branch 'master' into fix-engine-spotify 2019-12-29 09:47:06 +01:00
Marc Abonce Seguin
5706c12fba remove empty parenthesis in wikipedia's summary 
They're usually IPA pronunciations which are removed
by the API.
 2019-12-21 22:47:08 -06:00
Marc Abonce Seguin
c18048e045 exclude disambiguation pages from wikipedia infobox 2019-12-21 22:47:08 -06:00
Adam Tauber
34ad3d6b34 [enh] display error message if gigablast extra param expired 2019-12-21 21:25:50 +01:00
Adam Tauber
fc457569f7 [fix] pep8 2019-12-21 21:13:43 +01:00
Adam Tauber
00512e36c1 [fix] handle empty response from wikipedia engine - closes #1114 2019-12-21 21:01:08 +01:00
Adam Tauber
f8713512be [fix] convert byte query to string in osm engine - fixes #1220 2019-12-21 20:56:38 +01:00
Adam Tauber
e5305f886c [fix] fetch extra search param of gigablast - fixes #1293 2019-12-21 20:51:30 +01:00
Adam Tauber
8850036ded [fix] add explicit useragent header to requests - closes #1459 2019-12-21 20:25:39 +01:00
Marc Abonce Seguin
ccaf6ca02c [fix] update xpaths for new google results page 2019-12-07 16:37:24 -07:00
Adam Tauber
731e34299d
Merge pull request #1744 from dalf/optimizations 
[mod] speed optimization
 2019-12-02 13:39:58 +00:00
Adam Tauber
574cb25a16
Merge pull request #1758 from return42/ddd-fix 
[fix] duckduckgo_definitions
 2019-12-02 13:38:54 +00:00
Markus Heiser
30ad0c666d duckduckgo_definitions: remove the debug message 
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2019-12-02 14:26:11 +01:00
Adam Tauber
20da8f2cbf
Merge pull request #1754 from MarcAbonce/seedpeer 
Add Seedpeer again
 2019-12-02 13:09:09 +00:00
Markus Heiser
b6d9f5aa71 [fix] duckduckgo_definition issues reported by 'manage.sh test' 
Fix this error while travis build::

  /home/travis/build/asciimoo/searx/searx/engines/duckduckgo_definitions.py:21:44: E225 missing whitespace around operator

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2019-11-29 19:59:35 +01:00
Markus Heiser
4998e9ec85 [fix] duckduckgo_definitions - where 'AnswerType' is 'calc' 
Do not try to get text when 'AnswerType' is 'calc'.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2019-11-29 19:20:14 +01:00
Markus Heiser
2aa95c16e3 [fix] soundcloud: URLs of JS sources has been moved 
The client_id is found under (new) URL:

  https://a-v2.sndcdn.com/assets/49-a0c01933-3.js

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
 2019-11-29 15:05:36 +01:00
Adam Tauber
789d71350d
Merge pull request #1745 from lorddavidiii/python3.8-fix 
Fix python 3.8 compatibility
 2019-11-28 19:31:59 +00:00
Adam Tauber
05033ea8d8
Merge pull request #1689 from MarcAbonce/images_fixes 
[fix] Google Images
 2019-11-26 13:20:19 +00:00
Marc Abonce Seguin
9299355570 add seedpeer again 2019-11-24 22:01:44 -07:00
Emilien Devos
8f51430f5c [fix] Force Google old UI with a new user agent 2019-11-22 23:01:41 +01:00
lorddavidiii
5e5ff0cbf8 webapp.py: use html.escape if cgi.escape is not available 
- cgi.escape was removed in python 3.8
- also use html.escape in framalibre.py
 2019-11-16 21:58:25 +01:00
Dalf
85b3723345 [mod] speed optimization 
compile XPath only once
avoid redundant call to urlparse
get_locale(webapp.py): avoid useless call to request.accept_languages.best_match
 2019-11-15 09:33:15 +01:00
Noémi Ványi
5796dc60c9 fix pep 8 check 2019-10-16 15:52:48 +02:00
Noémi Ványi
a6f20caf32 add initial support for offline engines && command engine 2019-10-16 15:52:48 +02:00
Adam Tauber
7d8fd4b95e [fix] pep8 2019-10-16 15:32:03 +02:00
Adam Tauber
bbe4442a86 [fix] update gigablast engine 2019-10-16 15:27:37 +02:00
Adam Tauber
1057e42cfd [fix] update digg engine 2019-10-16 15:11:42 +02:00
Adam Tauber
7177c9e12f [fix] update devianart engine 2019-10-16 14:42:31 +02:00
Adam Tauber
6ca1622378 [fix] update 1x engine 2019-10-16 13:27:05 +02:00
Adam Tauber
c98a2df36d [fix] enable paging support for arxiv engine 2019-10-16 13:12:17 +02:00
Adam Tauber
ed1c1bdb04 [fix] pep8 2019-10-14 15:09:39 +02:00
Adam Tauber
77a70fe541 [fix] update startpage engine - closes #1601 2019-10-14 14:18:41 +02:00
Adam Tauber
94ea9d6622 [fix] duckduckgo paging - closes #1677 2019-10-14 13:52:15 +02:00
Marc Abonce Seguin
bb4d223770 [fix] google images 2019-08-26 21:54:01 -07:00
Léo Bourrel
88261e111c Fix bing engine results count (#1387) 
This PR fixes the result count from bing which was throwing an (hidden) error and add a validation to avoid reading more results than avalaible.

For example :
If there is 100 results from some search and we try to get results from 120 to 130, Bing will send back the results from 0 to 10 and no error. If we compare results count with the first parameter of the request we can avoid this "invalid" results.
 2019-08-05 16:15:40 +02:00
Dalf
1cee2c1796 [fix] bing engine 
before this commit, sometimes there are no results
use a generic user-agent instead of one with the OS "Windows NT 6.3; WOW64"
 2019-08-05 15:46:40 +02:00
Dalf
fcc9587ee9 [fix] fdroid engine 2019-08-05 15:44:02 +02:00
Dalf
fbf6b689dd [fix] dictzone engine 2019-08-05 15:43:21 +02:00
Dalf
9ff5001816 [fix] arxiv engine 2019-08-05 15:43:01 +02:00
Alexandre Flament
2179079a91
[fix] fix flickr_noapi decoding (#1655) 
Characters that were not ASCII were incorrectly decoded.
Add an helper function: searx.utils.ecma_unescape (Python implementation of unescape Javascript function).
 2019-08-02 13:37:13 +02:00
cy8aer
4dc792e1e2 [enh] add invidious engine. (#1657) 
closes #1372
 2019-08-02 13:25:25 +02:00
0xhtml
b2e1ee8d35 Fix some more errors with none/wrong credentials 2019-07-31 21:09:02 +02:00
0xhtml
275b37cc7c Fix error if the user hasn't set api credentials 2019-07-31 21:01:24 +02:00
0xhtml
c329ea135e Fix spotify engine 2019-07-31 20:44:41 +02:00
Dalf
0c032c8429 [fix] youtube_noapi engine: fix the title 2019-07-31 08:39:40 +02:00
Dalf
8b7ac56669 [fix] google_videos engine: some results don't a thumbnail 2019-07-31 08:38:45 +02:00
Dalf
d44677e226 [fix] dailymotion engine: remove HTML tags from the description 2019-07-31 08:37:51 +02:00
Dalf
6e0285b2db [fix] wikidata engine: faster processing, remove one HTTP redirection. 
* Search URL is https://www.wikidata.org/w/index.php?{query}&ns0=1 (with ns0=1 at the end to avoid an HTTP redirection)
* url_detail: remove the disabletidy=1 deprecated parameter
* Add eval_xpath function: compile once for all xpath.
* Add get_id_cache: retrieve all HTML with an id, avoid the slow to procress dynamic xpath '//div[@id="{propertyid}"]'.replace('{propertyid}')
* Create an etree.HTMLParser() instead of using the global one (see #1575)
 2019-07-29 07:39:39 +02:00
Frank de Lange
cbc5e13275 [enh] flickr_noapi: use complete JSON data block, add 'content', 'img_format', 'source', etc. (#1571) 
Fetch complete JSON data block, use legend to extract images. 
Unquote urlencoded strings.
Add image description as 'content'. 
Add 'img_format' and 'source' data (needs PR #1567 to enable this data to be displayed). 
Show images which lack ownerid instead of discarding them.
 2019-07-28 10:42:00 +02:00
Frank de Lange
204a2cbbf0 [fix] bing_videos (#1579) 
use JSON where possible, compose 'content' using all available data, use correct 'url' (direct to source instead of redirect through bing)
 2019-07-27 17:49:30 +02:00
Dalf
23611897ec [fix] make sure then engine name is lower case 
Minor fix: "%s engine initialized" display the right engine name
 2019-07-27 08:52:30 +02:00
Frank de Lange
11fc9913e9 [enh] bing_images: use data from embedded JSON to improve results (e.g. real page title) (#1568) 
use data from embedded JSON to improve results (e.g. real page title), add image format and source info (see PR #1567), improve paging logic (it now works)
 2019-07-27 08:22:02 +02:00
Alexandre Flament
f34b5cedb1
[fix] fixes google play engines (#1651) 
update commit 87baa74a86
 2019-07-25 09:31:47 +02:00
volth
eb182df132 [mod] restore btdigg engine as btdig.com (#1515) 2019-07-25 08:40:48 +02:00
rachmadani haryono
3b1122c5fa [fix] fix duden engine (#1594) 2019-07-25 08:17:45 +02:00
Venca24
87baa74a86 [fix] fixes google play engines and adds thumbnails to their results (#1612) 
fix google play apps, google play apps, google play music engines

xpath engine: thumbnail_xpath can define an optional thumbnail
 2019-07-25 07:46:41 +02:00
Dalf
da0ce5880f [fix] fix soundcloud engine, speed up searx start time 2019-07-19 08:59:11 +02:00
Dalf
45702b77ca embedded iframe (youtube, dailymotion, vimeo): use https 2019-07-13 15:49:59 +02:00
Emilien Devos
cbd1ebdce8 [fix] Force Google old UI (#1597) 2019-05-29 10:05:57 +09:00
Frank de Lange
4b7332286a Use string formatter to create source and img_format labels (#1566) 
google_images :  use JSON embedded in HTML (engine expected pure JSON)
 2019-05-28 12:33:31 +09:00
Dalf
ffe0972f91 Remove some engines : subtitleseeker, seedpeer, swisscows 
http://www.subtitleseeker.com and http://www.seedpeer.eu don't exist anymore.
https://swisscows.ch/ has change : the engine needs to be updated
 2019-05-28 04:06:35 +02:00