diff --git a/.devcontainer/README.md b/.devcontainer/README.md index c81b72f6e..e2de79f24 100644 --- a/.devcontainer/README.md +++ b/.devcontainer/README.md @@ -41,11 +41,12 @@ Checklist: :warning: **Important for all developers** :warning: -- [ ] Whe you are using the `NODE_ENV=development` the server will not store the configurations you set for security reasons. Please set the proper config on file `.env.development`. The side-effect if you don't, everytime you restart the server, you will be sent to the "Onboarding" page again. +- [ ] When you are using the `NODE_ENV=development` the server will not store the configurations you set for security reasons. Please set the proper config on file `.env.development`. The side-effect if you don't, everytime you restart the server, you will be sent to the "Onboarding" page again. -:warning: **Important for Github Codespaces** :warning: +**Note when using Github Codespaces** + +- [ ] When running the "Server" for the first time, it will automatically configure its port to be publicly accessible by default, as this is required for the front end to reach the server backend. To know more, read the content of the `.env` file on the frontend folder about this, and if any issues occur, make sure to manually set the port "Visibility" of the "Server" is set to "Public" if needed. Again, this is only needed for developing on Github Codespaces. -- [ ] When running the "Server" for the first time, its port will be automatically forward, but privately. Read the content of the `.env` file on the frontend folder about this, and make sure the port "Visibility" is set to "Public", so the frontend can reach the backend. Again, this is only needed for developing on Github Codespaces. We appreciate to know if you have a better solution. **For the Collector:** diff --git a/.github/workflows/build-and-push-image.yaml b/.github/workflows/build-and-push-image.yaml index 72d120039..591cf63d0 100644 --- a/.github/workflows/build-and-push-image.yaml +++ b/.github/workflows/build-and-push-image.yaml @@ -21,7 +21,8 @@ on: - '**/.env.example' - '.github/ISSUE_TEMPLATE/**/*' - '.devcontainer/**/*' - - 'embed/**/*' # Embed should be published to frontend (yarn build:publish) if any changes are introduced + - 'embed/**/*' # Embed is submodule + - 'browser-extension/**/*' # Chrome extension is submodule - 'server/utils/agents/aibitat/example/**/*' # Do not push new image for local dev testing of new aibitat images. jobs: diff --git a/.github/workflows/dev-build.yaml b/.github/workflows/dev-build.yaml index aef9a6c35..86e10bbcd 100644 --- a/.github/workflows/dev-build.yaml +++ b/.github/workflows/dev-build.yaml @@ -6,7 +6,7 @@ concurrency: on: push: - branches: ['encrypt-jwt-value'] # put your current branch to create a build. Core team only. + branches: ['agent-skill-plugins'] # put your current branch to create a build. Core team only. paths-ignore: - '**.md' - 'cloud-deployments/*' diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 000000000..526045ba7 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,7 @@ +[submodule "browser-extension"] + path = browser-extension + url = git@github.com:Mintplex-Labs/anythingllm-extension.git +[submodule "embed"] + path = embed + url = git@github.com:Mintplex-Labs/anythingllm-embed.git + branch = main diff --git a/.vscode/settings.json b/.vscode/settings.json index 3fcc79cd5..4769a939c 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -41,8 +41,10 @@ "Qdrant", "royalblue", "searxng", + "SearchApi", "Serper", "Serply", + "streamable", "textgenwebui", "togetherai", "Unembed", diff --git a/README.md b/README.md index 178fef08e..e07633266 100644 --- a/README.md +++ b/README.md @@ -69,7 +69,7 @@ AnythingLLM divides your documents into objects called `workspaces`. A Workspace ### Supported LLMs, Embedder Models, Speech models, and Vector Databases -**Language Learning Models:** +**Large Language Models (LLMs):** - [Any open-source llama.cpp compatible model](/server/storage/models/README.md#text-generation-llm-selection) - [OpenAI](https://openai.com) @@ -137,7 +137,8 @@ This monorepo consists of three main sections: - `server`: A NodeJS express server to handle all the interactions and do all the vectorDB management and LLM interactions. - `collector`: NodeJS express server that process and parses documents from the UI. - `docker`: Docker instructions and build process + information for building from source. -- `embed`: Code specifically for generation of the [embed widget](./embed/README.md). +- `embed`: Submodule for generation & creation of the [web embed widget](https://github.com/Mintplex-Labs/anythingllm-embed). +- `browser-extension`: Submodule for the [chrome browser extension](https://github.com/Mintplex-Labs/anythingllm-extension). ## 🛳 Self Hosting @@ -146,9 +147,9 @@ Mintplex Labs & the community maintain a number of deployment methods, scripts, |----------------------------------------|----:|-----|---------------|------------| | [![Deploy on Docker][docker-btn]][docker-deploy] | [![Deploy on AWS][aws-btn]][aws-deploy] | [![Deploy on GCP][gcp-btn]][gcp-deploy] | [![Deploy on DigitalOcean][do-btn]][do-deploy] | [![Deploy on Render.com][render-btn]][render-deploy] | -| Railway | RepoCloud | -| --- | --- | -| [![Deploy on Railway][railway-btn]][railway-deploy] | [![Deploy on RepoCloud][repocloud-btn]][repocloud-deploy] | +| Railway | RepoCloud | Elestio | +| --- | --- | --- | +| [![Deploy on Railway][railway-btn]][railway-deploy] | [![Deploy on RepoCloud][repocloud-btn]][repocloud-deploy] | [![Deploy on Elestio][elestio-btn]][elestio-deploy] | [or set up a production AnythingLLM instance without Docker →](./BARE_METAL.md) @@ -246,3 +247,5 @@ This project is [MIT](./LICENSE) licensed. [railway-deploy]: https://railway.app/template/HNSCS1?referralCode=WFgJkn [repocloud-btn]: https://d16t0pc4846x52.cloudfront.net/deploylobe.svg [repocloud-deploy]: https://repocloud.io/details/?app_id=276 +[elestio-btn]: https://elest.io/images/logos/deploy-to-elestio-btn.png +[elestio-deploy]: https://elest.io/open-source/anythingllm diff --git a/browser-extension b/browser-extension new file mode 160000 index 000000000..385d36c08 --- /dev/null +++ b/browser-extension @@ -0,0 +1 @@ +Subproject commit 385d36c0807221e6674d1325ef06505746c49ceb diff --git a/collector/extensions/index.js b/collector/extensions/index.js index 30beaa3e7..47989d5d5 100644 --- a/collector/extensions/index.js +++ b/collector/extensions/index.js @@ -38,7 +38,6 @@ function extensions(app) { reqBody(request), response, ); - console.log({ success, reason, data }) response.status(200).json({ success, reason, diff --git a/collector/utils/extensions/RepoLoader/GithubRepo/RepoLoader/index.js b/collector/utils/extensions/RepoLoader/GithubRepo/RepoLoader/index.js index 08121f44f..935cab197 100644 --- a/collector/utils/extensions/RepoLoader/GithubRepo/RepoLoader/index.js +++ b/collector/utils/extensions/RepoLoader/GithubRepo/RepoLoader/index.js @@ -119,6 +119,7 @@ class GitHubRepoLoader { maxConcurrency: 5, unknown: "ignore", ignorePaths: this.ignorePaths, + verbose: true, }); const docs = []; diff --git a/docker/.env.example b/docker/.env.example index 56be87cb4..1521a307a 100644 --- a/docker/.env.example +++ b/docker/.env.example @@ -252,6 +252,10 @@ GID='1000' # AGENT_GSE_KEY= # AGENT_GSE_CTX= +#------ SearchApi.io ----------- https://www.searchapi.io/ +# AGENT_SEARCHAPI_API_KEY= +# AGENT_SEARCHAPI_ENGINE=google + #------ Serper.dev ----------- https://serper.dev/ # AGENT_SERPER_DEV_KEY= diff --git a/docker/HOW_TO_USE_DOCKER.md b/docker/HOW_TO_USE_DOCKER.md index 1e95bd8a0..2eeaee060 100644 --- a/docker/HOW_TO_USE_DOCKER.md +++ b/docker/HOW_TO_USE_DOCKER.md @@ -117,8 +117,8 @@ services: - WHISPER_PROVIDER=local - TTS_PROVIDER=native - PASSWORDMINCHAR=8 - - AGENT_SERPER_DEV_KEY="SERPER DEV API KEY" - - AGENT_SERPLY_API_KEY="Serply.io API KEY" + # Add any other keys here for services or settings + # you can find in the docker/.env.example file volumes: - anythingllm_storage:/app/server/storage restart: always diff --git a/embed b/embed new file mode 160000 index 000000000..22a0848d5 --- /dev/null +++ b/embed @@ -0,0 +1 @@ +Subproject commit 22a0848d58e3a758d85d93d9204a72a65854ea94 diff --git a/embed/.gitignore b/embed/.gitignore deleted file mode 100644 index 4d3751d9a..000000000 --- a/embed/.gitignore +++ /dev/null @@ -1,25 +0,0 @@ -# Logs -logs -*.log -npm-debug.log* -yarn-debug.log* -yarn-error.log* -pnpm-debug.log* -lerna-debug.log* - -node_modules -dist -dist-ssr -*.local - -# Editor directories and files -.vscode/* -!.vscode/extensions.json -!yarn.lock -.idea -.DS_Store -*.suo -*.ntvs* -*.njsproj -*.sln -*.sw? diff --git a/embed/README.md b/embed/README.md deleted file mode 100644 index af6621ba1..000000000 --- a/embed/README.md +++ /dev/null @@ -1,112 +0,0 @@ -# AnythingLLM Embedded Chat Widget - -> [!WARNING] -> The use of the AnythingLLM embed is currently in beta. Please request a feature or -> report a bug via a Github Issue if you have any issues. - -> [!WARNING] -> The core AnythingLLM team publishes a pre-built version of the script that is bundled -> with the main application. You can find it at the frontend URL `/embed/anythingllm-chat-widget.min.js`. -> You should only be working in this repo if you are wanting to build your own custom embed. - -This folder of AnythingLLM contains the source code for how the embedded version of AnythingLLM works to provide a public facing interface of your workspace. - -The AnythingLLM Embedded chat widget allows you to expose a workspace and its embedded knowledge base as a chat bubble via a ` -``` - -### `