1
0
mirror of https://github.com/searxng/searxng.git synced 2024-11-16 17:40:11 +01:00
Commit Graph

72 Commits

Author SHA1 Message Date
Markus Heiser
977e9a4330
Merge pull request #686 from return42/lib_redis
Add redis DB and connector
2022-01-11 19:55:14 +01:00
Markus Heiser
c9cfa6181b [fix] debug console (pdb) in a session started by make run
Commit c7f27404 moves the `python -m searx.webapp` process to the background.  A
background job can't open a simple python-debugger (pdb) console.

This patch moves all processes to the background except the searx.webapp.

Insert a break point somewhere in the webapp application::

    import pdb
    pdb.set_trace()

And start a debug session by::

    make run

and test you break point.

To test that the entire 'make run' stops in the case of an error in the
themes.live [1] background process try:

    make LIVE_THEME=typo-theme-name run

[1] https://github.com/searxng/searxng/pull/664#discussion_r776419585

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-05 10:09:17 +01:00
Markus Heiser
00d3a7477c
Merge pull request #664 from return42/themes.live
[mod] manage: add themes.live command (rebuild on modification)
2022-01-04 19:44:51 +01:00
Markus Heiser
fe8b88c8a4 [mod] script to build & install a redis instance
A script to build & install a simple & isolated redis service, dedicated to
SearXNG and connected via Unix socket.

    $ ./manage redis.help
    redis.:
      devpkg    : install essential packages to compile redis
      build     : build redis binaries at /800GBPCIex4/share/SearXNG/dist/redis/6.2.6/amd64
      install   : create user (searxng-redis) and install systemd service (searxng-redis)
      remove    : delete user (searxng-redis) and remove service (searxng-redis)
      shell     : start bash interpreter from user searxng-redis
      src       : clone redis source code to <path> and checkput 6.2.6
      useradd   : create user (searxng-redis) at /usr/local/searxng-redis
      userdel   : delete user (searxng-redis)
      addgrp    : add <user> to group (searxng-redis)
      rmgrp     : remove <user> from group (searxng-redis)

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2022-01-03 14:34:26 +01:00
Markus Heiser
c7f2740414 [mod] live build of a theme: LIVE_THEME=simple make run
Environment variable to get live builds while modifying CSS & JS of a theme::

    LIVE_THEME=simple make run

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-29 21:53:12 +01:00
Markus Heiser
1706b130a4 [fix] remove leftover from 59f4c792
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-29 21:53:12 +01:00
Markus Heiser
edfd069d16 [mod] manage: add themes.live command (rebuild on modification)
To have a live build while editing the HTML client of a theme.  The first
argument of the command is the theme name:

    ./manage themes.live simple

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-29 21:53:12 +01:00
Markus Heiser
c849731eb1 [mod] test.black: show issues (diff) reported by python-black
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-27 08:59:47 +01:00
Alexandre Flament
3c77412d33 [mod] replace pycodestyle by black
"make test.black" checks for the code style
"make format.python" format the python code
2021-12-27 08:58:59 +01:00
Alexandre Flament
81abc5f4c2 [fix] ./manage: upgrade geckodriver when GECKODRIVER_VERSION change 2021-12-24 09:14:15 +01:00
Markus Heiser
69af219d39 [test.robot] update gecko driver / required by selenium 4.1.0
Update gecko driver to v0.30.0 [1]

[1] https://github.com/mozilla/geckodriver/releases/tag/v0.30.0

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-23 13:09:48 +01:00
Markus Heiser
8069bc83e6 [fix] weblate: separate commit description from commit body
It is a common convention to separate commit description from commit body by a
empty line [1].

[1] https://www.conventionalcommits.org/en/v1.0.0/#summary

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-17 10:48:12 +01:00
Markus Heiser
f287787c44 [simple theme] activate - SearXNG JavaScript Style Guide
Add the 'SearXNG JavaScript Style Guide' as one quality gate in the build chain
of the simple theme::

    make themes.simple
    make themes.simple.test

In the github CI the `themes.all` target enforce a `themes.simple`.

BTW: Remove 'jshint' left overs from 0ee316f3d

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-12-14 07:50:53 +01:00
Alexandre Flament
59f4c792b4 [mod] simple theme: use sharp instead of convert to create .png from .svg
define a custom grunt task, since grunt-sharp is too old (it can't be installed).
in gruntfile.js, the image tasks are moved at the end the build chain.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-28 20:05:37 +01:00
Markus Heiser
c68c53acd8 [brand] initial - folder src/brand for searxng-wordmark
Folder src/brand is intended to place logo, wordmark and more in.  This commit
moves the origin (source) of searxng-wordmark.svg from the simple theme into
this folder

About the new created src/ folder:

On the long term we can move all the files from static/themes/<name>/src/* into
a src/themes/<name>/* folder.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-20 10:14:55 +01:00
Markus Heiser
1e997386e9
Merge pull request #508 from return42/arm64
[mod] Tools to install and maintain golang binaries & packages.
2021-11-18 19:49:01 +00:00
Markus Heiser
dc1442a2d1 [mod] Tools to install and maintain NVM versions manager for Node.js
[1] https://github.com/nvm-sh/nvm

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-16 23:31:11 +01:00
Markus Heiser
abd5eee392 [mod] Tools to install and maintain golang [1] binaries & packages.
[1] https://golang.org/doc/devel/release#policy

A simple *helloworld* test with user 'my_user' :

  sudo -H adduser my_user
  ./manage go.golang go1.17.3 my_user
  ./manage go.install github.com/go-training/helloworld@latest my_user
  ./manage go.bash my_user
  $ helloword
  Hello World!!

Don't forget to remove 'my_user':  sudo -H deluser --remove-home my_user

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-16 18:00:47 +01:00
Markus Heiser
5db24d4818 [fix] make test.shell - ./manage line 80
make test.shell reports an issue that has been added in [PR-500]::

    In ./manage line 80:
      yamllint  : lint YAML files: $YAMLLINT_FILES
                                   ^-------------^
                                   SC2128: Expanding an array without an
                                   index only gives the first element.

[PR-500] https://github.com/searxng/searxng/pull/500

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-15 17:02:44 +01:00
Markus Heiser
e7a9cd7a34 [fix] manage help - add description of test.yamllint
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-13 15:35:08 +01:00
Markus Heiser
3a981257de [simple theme] create favicon.png from src/svg/searxng-wordmark.svg
new bash function convert_if_newer() / usage::

    convert_if_newer <origfile> <outfile> [<options>, ...]
    convert_if_newer "path/to/origin.svg" "path/to/converted.png" -transparent white -resize 64x64

Run's ImageMagik' convert comand to generate <outfile> from <origfile>, if
<origfile> is newer than <outfile>.  The command line is to convert is::

    convert <origfile> [<options>, ...] <outfile>

PNG 'searx/static/themes/simple/img/favicon.png' has been created by::

  $ make themes.simple
  CONVERT   searx/static/themes/simple/src/svg/searxng-wordmark.svg -transparent white -resize 64x64 searx/static/themes/simple/img/favicon.png
  ...

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-11-13 09:05:20 +01:00
Markus Heiser
68effd5eda [brand] SearXNG - bash env PYLINT_SEARXNG_DISABLE_OPTION
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-30 16:11:01 +02:00
Alexandre Flament
9e03823b98 SearXNG: docker 2021-10-12 20:23:57 +02:00
Markus Heiser
60edf2623d [brand] SearXNG - reference /etc/searxng/settings.yml
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-11 12:28:55 +00:00
Alexandre Flament
3fad483b7a SearXNG: partial update: reference /etc/searxng/settings.yml 2021-10-11 12:28:55 +00:00
Markus Heiser
afc5c9870e [fix] commit message of 'make weblate.push.translations'
Fix `\n` issue in the commit message [1] by using multiple `-m` options [2]::

    7d9ffd680 translations     [translations] update messages.pot and messages.po files\nFrom cebc0e39 - 2021-10-04 - Markus Heiser <markus.heiser@darmarIT.de>

[1] https://github.com/searxng/searxng/pull/379#issuecomment-933242702
[2] https://git-scm.com/docs/git-commit#Documentation/git-commit.txt--mltmsggt

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-10-04 11:09:30 +02:00
Alexandre Flament
f4dd5b226d
[fix] make weblate.push.translations
Even when there are changes, the function exit without pushing them.
2021-10-03 21:11:32 +02:00
Alexandre Flament
47eb836c65
Merge pull request #375 from dalf/searxng_extra
SearXNG: searx_extra
2021-10-03 19:09:07 +02:00
Alexandre Flament
1bb82a6b54 SearXNG: searxng_extra 2021-10-02 17:30:39 +02:00
Alexandre Flament
253b850376 SearXNG: SEARXNG_SETTINGS_PATH 2021-10-02 17:18:05 +02:00
Alexandre Flament
430babca25 SearXNG: environment variables 2021-10-02 16:54:11 +02:00
Markus Heiser
7a3a1cd115 [mod] manage - use pyenv.activate where it makes sense
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-17 15:31:09 +02:00
Alexandre Flament
e5e322e8ed [mod] utils/lib.sh: add pyenv.activate
* initialize virtualenv (pyenv.install)
* activate virtualenv

close #310
2021-09-17 14:15:35 +02:00
Markus Heiser
3f2c10ef9c [fix] manage help(): commands babel.* has been renamed to weblate.*
In commit 97355672c the functions named babel.* has been renamed to weblate.*
but it was forgotten to change it also in the help().

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-12 15:46:00 +02:00
Markus Heiser
82847df300 [fix] add 'categories' to PYLINT_ADDITIONAL_BUILTINS_FOR_ENGINES
androp no longer needed (see line 591 in 7b235a1)::

    # pylint: disable=undefined-variable

Suggested-by: @dalf https://github.com/searxng/searxng/issues/102#issuecomment-914068609
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-07 10:29:38 +02:00
Alexandre Flament
b57d776edb
Merge pull request #293 from dalf/fix-290
[fix] translation: lock weblate only when there is an actual change of messages.pot
2021-09-06 20:00:19 +02:00
Markus Heiser
7b235a1c36 [mod] one logger per engine
Suggested-by: @dalf in https://github.com/searxng/searxng/issues/98#issuecomment-849013518
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-09-06 17:47:28 +02:00
Alexandre Flament
2a5f690a71 [fix] translation: lock weblate only when there is an actual change of messages.pot
Close #290
2021-09-06 17:16:51 +02:00
Alexandre Flament
b9c73fb697 [mod] move searx/testing.py to the tests directory
move robot tests to tests.robot
manage calls "python -m tests.robot"
2021-09-02 16:01:34 +02:00
Markus Heiser
97355672cd [translations] ./manage rename shell functions to integrate weblate
Functions implemented to run *Weblate* workflows should use prefix 'weblate.':

- babel.setup.translations.worktree --> weblate.translations.worktree

  Create git worktree ${TRANSLATIONS_WORKTREE} and checkout branch
  'translations' from Weblate's counterpart (weblate) of the SearXNG
   (origin)::

       remote weblate https://weblate.bubu1.eu/git/searxng/searxng/

- babel.weblate.to.translations     --> weblate.to.translations

  Update 'translations' branch of SearXNG (origin) with last additions from
  Weblate.

- babel.translations.to.master      --> weblate.translations.commit

  Update 'translations' branch of SearXNG (origin) with last additions from
  Weblate.  Copy the changes to the master branch, compile translations and
  create a commit in the local branch (master)

- babel.master.to.translations      --> weblate.push.translations

  Push *translation changes* from SearXNG (origin) to Weblate's
  counterpart (weblate).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-16 12:06:12 +02:00
Markus Heiser
0dc8d244c5 [translation] ./manage: add comments to new functions, rename job
./mange

  Comment functions:
  - babel.setup.translations.worktree
  - babel.weblate.to.translations
  - babel.translations.to.master
  - babel.master.to.translations
  BTW:
  - changed some cd into pushd
  - removed obsolete `set -x`
  - remove not needed echo from `last_commit_message=$(echo ..`

.github/workflows/translations-update.yml
docs/dev/translation.rst

  Rename job to: "create PR for additons from weblate"

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-16 12:00:16 +02:00
Alexandre Flament
901ab87717 [translations] web integration
* make babel.translations.to.master: pull weblate updates
* make babel.master.to.translations: push .pot and .po files to weblate
2021-08-07 15:06:06 +02:00
Markus Heiser
900baf2eae [mod] manage - implement babel commands, drop update_translations.sh
In ./manage implement babel.*:

-  extract   : extract messages from source files and generate POT file
-  update    : update existing message catalogs from POT file
-  compile   : compile translation catalogs into binary MO files

Replace searx_extra/update/update_translations.sh by command:

- ci.babel.update

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-08-02 10:06:27 +02:00
Alexandre Flament
a7a181064f manage script: remove unused reference to utils/brand.env and .config.sh 2021-08-01 07:05:00 +02:00
Alexandre Flament
4b43775c91 version based on the git repository
This commit remove the need to update the brand for GIT_URL and GIT_BRANCH:
there are read from the git repository.

It is possible to call python -m searx.version freeze to freeze the current version.
Useful when the code is installed outside git (distro package, docker, etc...)
2021-07-30 14:40:09 +02:00
Markus Heiser
61d3914b63 [fix] test.robot - do not use environment from ./utils/brand.env
Some defaults in the settings.yml are taken from the environment.
By example;

The manage scripts sources the ./utils/brand.env and sets SEARX_PORT
environment.  This enviroment *wins over* any settings in a YAML file.
Whe we run a::

    make test.robot

The searx/settings_robot.yml is used, in this file the server settings are::

    server:
      port: 11111
      bind_address: 127.0.0.1

To get in use of the 'port: 11111' we have to unset the SEARX_PORT environment
which was sourced before.

The function buildenv.unset_env() can be called in all use cases where the
enviroment from ./utils/brand.env is not wanted.  ATM it unset the enviroment
variables::

    unset GIT_URL
    unset GIT_BRANCH
    unset SEARX_URL
    unset SEARX_PORT
    unset SEARX_BIND_ADDRESS

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-24 16:28:15 +02:00
Markus Heiser
2964a24b3d [fix] utils/build_env.py and add documentation
modified   docs/admin/engines/settings.rst
  - Fix documentation and add section 'brand'.
  - Add remarks about **buildenv** variables.
  - Add remarks about settings from environment variables $SEARX_DEBUG,
    $SEARX_PORT, $SEARX_BIND_ADDRESS and $SEARX_SECRET

modified   docs/admin/installation-searx.rst & docs/build-templates/searx.rst
   Fix template location /templates/etc/searx/settings.yml

modified   docs/dev/makefile.rst
  Add description of the 'make buildenv' target and describe
  - we have all SearXNG setups are centralized in the settings.yml file
  - why some tasks need a utils/brand.env (aka instance's buildenv)

modified   manage
  Settings file from repository's working tree are used by default and
  ask user if a /etc/searx/settings.yml file exists.

modified   searx/settings.yml
  Add comments about when it is needed to run 'make buildenv'

modified   searx/settings_defaults.py
  Default for server:port is taken from enviroment variable SEARX_PORT.

modified   utils/build_env.py
  - Some defaults in the settings.yml are taken from the environment,
    e.g. SEARX_BIND_ADDRESS (searx.settings_defaults.SHEMA).  When the
    'brand.env' file is created these enviroment variables should be
    unset first.
  - The CONTACT_URL enviroment is not needed in the utils/brand.env

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 14:38:54 +02:00
Markus Heiser
51c06722c3 [mod] ./manage docker.buildx make variables local
The environment variables ..

- SEARX_GIT_VERSION
- VERSION_GITCOMMIT
- SEARX_PYTHON_VERSION
- GITHUB_USER
- SEARX_IMAGE_NAME
- BUILD

are local to the docker.buildx() function, other variables like GIT_URL are
comming from the utils/brand.env file (rebuild by: make buildenv).

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-21 13:38:28 +02:00
Markus Heiser
e179dba7e4 [fix] ./manage script, prompt the unknown command
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-07-17 17:15:28 +02:00
Markus Heiser
25b6309cf2 [mod] move functions from utils/manage_static.sh to ./manage script
The functions:

- static.build.commit
- static.build.commit.drop
- static.build.restore

are imported into the ./manage script.  To avoid name collisions some variables
and fucntions has been renamed by adding the prefix *static_*.

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
2021-06-26 13:45:25 +02:00