Commit Graph

521 Commits

Author SHA1 Message Date
Timothy Carambat
20135835d0
Ollama sequential embedding (#2230)
* ollama: Switch from parallel to sequential chunk embedding

* throw error on empty embeddings

---------

Co-authored-by: John Blomberg <john.jb.blomberg@gmail.com>
2024-09-06 10:06:46 -07:00
Timothy Carambat
b8b55b5899
Feature/add searchapi web browsing (#2224)
* Add SearchApi to web browsing

* UI modifications for SearchAPI

---------

Co-authored-by: Sebastjan Prachovskij <sebastjan.prachovskij@gmail.com>
2024-09-05 10:36:46 -07:00
Mr Simon C
7594841dac
Bug/make swagger json output openapi 3 compliant (#2219)
update source to ensure swagger.json is openapi 3.0.0 compliant
2024-09-04 15:40:24 -07:00
Sean Hatfield
86ba16acdc
Add new Voyage AI embedding models (#2193)
new voyageai embedding models
2024-08-29 14:11:00 -07:00
timothycarambat
334fd9cdd0 pre-validate sessionID type for embed chats 2024-08-27 16:38:20 -07:00
timothycarambat
548da9ade3 Exception handler on embed chat middleware 2024-08-27 16:27:58 -07:00
timothycarambat
47a5c7126c Patch path traversal in move-files that can be used by administrator level attacker only 2024-08-27 16:19:12 -07:00
Timothy Carambat
29df483a27
AnythingLLM Chrome Extension (#2066)
* initial commit for chrome extension

* wip browser extension backend

* wip frontend browser extension settings

* fix typo for browserExtension route

* implement verification codes + frontend panel for browser extension keys

* reorganize + state management for all connection states

* implement embed to workspace

* add send page to anythingllm extension option + refactor

* refactor connection string auth + update context menus + organize background.js into models

* popup extension from main app and save if successful

* fix hebrew translation misspelling

* fetch custom logo inside chrome extension

* delete api keys on disconnect of extension

* use correct apiUrl constant in frontend + remove unneeded comments

* remove upload-link endpoint and send inner text html to raw text collector endpoint

* update readme

* fix readme link

* fix readme typo

* update readme

* handle deletion of browser keys with key id and DELETE endpoint

* move event string to constant

* remove tablename and writable fields from BrowserExtensionApiKey backend model

* add border-none to all buttons and inputs for desktop compatibility

* patch prisma injections

* update delete endpoints to delete keys by id

* remove unused prop

* add button to attempt browser extension connection + remove max active keys

* wip multi user mode support

* multi user mode support

* clean up backend + show created by in frotend browser extension page

* show multi user warning message on key creation + hide context menus when no workspaces

* show browser extension options to managers

* small backend changes and refactors

* extension cleanup

* rename submodule

* extension updates & docs

* dev docker build

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
2024-08-27 14:58:47 -07:00
Timothy Carambat
b4651aff35
Support gpt-4o for Azure deployments (#2182) 2024-08-26 14:35:42 -07:00
Timothy Carambat
22ecb7cb39
Add RAG agent plugin to API agent (#2171) 2024-08-23 09:32:19 -07:00
Timothy Carambat
2de9e492ec
Enabled use of @agent (and skills) via dev API calls (#2161)
* Use `@agent` via dev API

* Move EphemeralEventListener to same file as agent
2024-08-22 13:12:09 -07:00
Timothy Carambat
fdc3add53c
Api session id support (#2158)
* Refactor api endpoint chat handler to its own function
remove legacy `chatWithWorkspace` and cleanup `index.js`

* Add `sessionId` in dev API to partition chats logically statelessly
2024-08-21 15:25:47 -07:00
timothycarambat
2d2e49bc00 fix missing export 2024-08-21 14:58:56 -07:00
Timothy Carambat
1f96b837b3
Refactor api endpoint chat handler to its own function (#2157)
remove legacy `chatWithWorkspace` and cleanup `index.js`
2024-08-21 14:47:06 -07:00
timothycarambat
17abbe97e5 remove #swagger.path from API docs 2024-08-20 12:48:08 -07:00
timothycarambat
cb7cb2d976 Add 405B to perplexity 2024-08-19 12:26:22 -07:00
Timothy Carambat
a8d25c7dd3
Allow readable username passed in script widget (#2131) 2024-08-16 16:55:42 -07:00
Sean Hatfield
1dad4d9409
Fix multi-user setup bug for invalid username/password (#2130)
* fix multiuser setup bug for invalid username/password

* handle user creation failed gracefully and delete user on fail
2024-08-16 12:55:14 -07:00
Timothy Carambat
c8fe254d45
Omit invalid response.text values and prompts (#2127)
* Omit invalid `response.text` values and `prompts`
resolves #2108

* remove import
2024-08-15 14:22:27 -07:00
Timothy Carambat
99f2c25b1c
Agent Context window + context window refactor. (#2126)
* Enable agent context windows to be accurate per provider:model

* Refactor model mapping to external file
Add token count to document length instead of char-count
refernce promptWindowLimit from AIProvider in central location

* remove unused imports
2024-08-15 12:13:28 -07:00
Shahar
4365d69359
Fix TypeError by replacing this.openai.createChatCompletion with the correct function call (#2117)
fixed new api syntax
2024-08-14 14:39:48 -07:00
Timothy Carambat
4430ddb059
Encryption in JWT for single-user password mode (#2111)
* wip encrypting jwt value

* Encrypt/Decrypt pass in JWT value for verification in single-user password mode
2024-08-13 17:54:12 -07:00
PyKen
a2571024a9
Add prompt window limits for gpt-4o-* models (#2104) 2024-08-13 09:13:36 -07:00
Timothy Carambat
f06ef6180d
add exp model to v1Beta (#2082) 2024-08-09 14:19:49 -07:00
timothycarambat
c9e19264eb fix validation issue 2024-08-07 14:24:24 -07:00
Sean Hatfield
665553af5a
Developer API access for embed chat widgets (#1999)
* developer api access for embedded chat widgets and embed chats

* lint

* uncheck files

* sanitize embed api inputs

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
2024-08-07 11:36:32 -07:00
Sean Hatfield
c97066526a
New user account validations (#2037)
* force lowercase and no space for new and
 edit user modals

* edit account modal validations

* use pattern for form validation + remove validations from edit user

* revert comment deletions

* comment fix

* update validation message

* update regex
allow updating by block name changes to invalid names

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-08-07 11:35:37 -07:00
Timothy Carambat
d072875e43
Add piperTTS in-browser text-to-speech (#2052)
* Add piperTTS in-browser text-to-speech

* update vite config

* Add voice default + change prod public URL

* uncheck file

* Error handling
bump package for better quality and voices

* bump package

* Remove pre-packed WASM - will not support offline first solution for docker

* attach TTSProvider telem
2024-08-07 11:09:51 -07:00
Sean Hatfield
6666fff0c2
Support multiple preset prompts in single message (#2036)
support multiple preset prompts in single message
2024-08-06 10:27:36 -07:00
Sean Hatfield
7273c892a1
Ollama performance mode option (#2014)
* ollama performance mode option

* Change ENV prop
Move perf setting to advanced

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-08-02 13:29:17 -07:00
Timothy Carambat
ba8e4e5d3e
handle OpenRouter exceptions on streaming (#2033) 2024-08-02 12:23:39 -07:00
RahSwe
c55ef33fce
Gemini Pro 1.5, API support for 2M context and new experimental model (#2031) 2024-08-02 10:24:31 -07:00
timothycarambat
6dc3642661 Patch Groq preview models maxed to 8K tokens due to warning 2024-08-01 09:24:57 -07:00
timothycarambat
466bf7dc9c Bump Perplexity and Together AI static model list 2024-07-31 10:58:34 -07:00
Timothy Carambat
38fc181238
Add multimodality support (#2001)
* Add multimodality support

* Add Bedrock, KoboldCpp,LocalAI,and TextWebGenUI multi-modal

* temp dev build

* patch bad import

* noscrolls for windows dnd

* noscrolls for windows dnd

* update README

* update README

* add multimodal check
2024-07-31 10:47:49 -07:00
Timothy Carambat
20cd6b7481
Added more char exceptions to mapping extension (#2010) 2024-07-31 09:41:12 -07:00
Timothy Carambat
d877d2b7ad
Add drag-and-drop to chat window (#1995)
* Add drag-and-drop to chat window

* add uploader icon and remove empty space text when attachments are present

* color theme

* color update
2024-07-30 10:26:16 -07:00
Timothy Carambat
5e73dce506
Enable editing of OpenRouter stream timeout for slower connections (#1994) 2024-07-29 11:49:14 -07:00
timothycarambat
296f041564 path perplexity model ids
closes #1990
2024-07-28 16:29:18 -07:00
Timothy Carambat
ae58a2cb0d
Return threads for API workspace calls (#1978)
* Return threads for API workspace calls

* remove import
2024-07-26 14:18:40 -07:00
timothycarambat
7a2ffefdc3 update case stmt for duplicate groq model 2024-07-25 17:39:29 -07:00
timothycarambat
6bab8b5bd4 ui adjustments for order flow
back nav on order flow
fix bad schema ref
2024-07-25 14:10:17 -07:00
Timothy Carambat
88e2209959
bump jsonwebtoken version (#1971)
* bump `jsonwebtoken` version

* update dev build branch
2024-07-25 11:03:39 -07:00
Timothy Carambat
04a0fc4ec9
Remove unused deps (#1938)
* Remove unused deps

* improve dependency
2024-07-25 10:21:03 -07:00
Timothy Carambat
61e214aa8c
Add support for Groq /models endpoint (#1957)
* Add support for Groq /models endpoint

* linting
2024-07-24 08:35:52 -07:00
Timothy Carambat
23de85a3bd
Revert odbc support (#1936)
* Revert "Patch ODBC support from missing binary/headers for node-odbc"

This reverts commit 9de6b1cc26.

* Revert "OBDC Support (#1933)"

This reverts commit cd597a361e.
2024-07-23 17:27:39 -07:00
Timothy Carambat
9366e69d88
Add AWS bedrock support for LLM + agents (#1935)
add AWS bedrock support for LLM + agents
2024-07-23 16:35:37 -07:00
Timothy Carambat
cd597a361e
OBDC Support (#1933)
* add possibility to connect to SQL Base by ODBC

---------

Co-authored-by: suchaudn <nicolas.suchaud@legrand.fr>
Co-authored-by: nicho2 <nicho2@laposte.net>
2024-07-23 12:42:53 -07:00
Timothy Carambat
42235fcd8a
GitLab Hosted and Local Connector (#1932)
* Add support for GitLab repo collection as well as Github Repo collection
* Refactor for repo collectors to be more compact

---------

Co-authored-by: Emil Rofors <emirof@gmail.com>
2024-07-23 12:23:51 -07:00
Timothy Carambat
f8e54b2328
Fine tuning order flow (cloud-based) (#1923)
* WIP fine-tuning order flow

* flow patches and typo

* Refine steps
add fine tuning CTA on chat page
add fine tuning banner and validation endpoints
add finetuning banner on relevant pages with perms check

* Add prod firebase url
2024-07-22 17:59:17 -07:00
timothycarambat
d0713e15f1 patch response body for swagger 2024-07-22 15:21:32 -07:00
Timothy Carambat
76aa2a4fd4
Implement support for selecting basic keep_alive times for Ollama (#1920) 2024-07-22 14:44:47 -07:00
timothycarambat
7b42a67acf docs: update requirement for /raw-text endpoint
resolves #1893
2024-07-22 13:19:57 -07:00
timothycarambat
2185753068 patch text.substring bug from compressor 2024-07-22 12:53:11 -07:00
Timothy Carambat
3198718975
Update references to new domain (#1916) 2024-07-22 11:05:34 -07:00
timothycarambat
f15529653f patch logger for full logs 2024-07-19 18:35:41 -07:00
timothycarambat
cec1a3d585 append stacktraces to winston 2024-07-19 18:13:54 -07:00
Timothy Carambat
b5a2437bb5
patch docker scount CVE in old express-ws pkg (#1907)
* patch CVE in old express-ws pkg

* patch workflow

* remove dev-image
2024-07-19 17:40:22 -07:00
Timothy Carambat
8180787c2e
Enable editing and setting of meta-tag information (#1892)
* Enable editing and setting of meta-tag information

* cleanup

* tmp build for testing

* finally always refresh

* unset workflow

* dev build

* rm tmp build
2024-07-19 15:58:43 -07:00
timothycarambat
f9929a28cb patch lint 2024-07-19 15:27:56 -07:00
Johnny Chen
c4158b97d6
bugfix: API new user return 400 if user creation failed (#1898)
* return 400 if user creation failed

The client might not check the error field unless they realize it's a 4xx error instead of 200 success.

* Update index.js

* use one-liner style

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
2024-07-19 15:27:21 -07:00
Timothy Carambat
5df6b5f7d9
Bump perplexity models (#1905)
* Added Supported Models Free Tier - chat_models.txt

Need to fill in correct Parameter Count.

* Bump perplexity model
closes #1901
closes #1900

---------

Co-authored-by: Tim-Hoekstra <135951177+Tim-Hoekstra@users.noreply.github.com>
2024-07-19 15:11:10 -07:00
Timothy Carambat
b275008b19
Docker image updates (#1896)
* bump base image

* Bump mysql

* attestations

* attestations perms

* attestations perms

* fix permissions for attestetions for master push

* test cleanup

* repin base image

* revert base

* patch frontend-clean

* force resolve braces to fixed version

* rebump
2024-07-19 08:09:32 -07:00
Timothy Carambat
75b6710052
Extend support for some chars in slugify of workspace (#1890)
resolves #1877
2024-07-18 10:35:02 -07:00
Sean Hatfield
e909b25b29
[FEAT] Prisma injection validation (#1874)
check all prisma models/model usage and patch any potential sql injection vulns
2024-07-16 16:40:05 -07:00
timothycarambat
86a66ba569 change alpaca format to include citations and system prompt 2024-07-15 16:05:53 -07:00
Sean Hatfield
8f0af88332
[FIX] Fix manual input model pref for agents (#1848)
* patch llm providers that have manual inputs for model pref

* refactor agent model fallback
update UI to show disabled providers to stop questions about provider limitations

* patch log on startup

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-07-11 14:03:24 -07:00
Sean Hatfield
ab56bae1bb
[FEAT] Ability to delete messages from conversation history (#1812)
* implement delete message feature

* add new delete chat endpoint that just hides messages from ui and fix perms

* refactor: add delete-event hooks for delete so we dont need to pass so much
refactor: implmentation of delete endpoint PUT

* update animation
linting
remove element on animiation to prevent popin

* linting and cleanup imports

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-07-03 18:28:08 -07:00
timothycarambat
8658b1e7c7 linting 2024-07-03 18:25:44 -07:00
Timothy Carambat
29c9eeaa5c
Add winston logging for production (#1811) 2024-07-03 16:39:33 -07:00
Sean Hatfield
8b5d9ccdb3
[FEAT] Fork chat to new thread (#1788)
* implement thread forking feature

* rename thread based on forked message

* refactor bulk message create for thread fork + bump prisma version

* revert prisma version bump

* add todo to bulkCreate function in workspace chats

* cast user input to expected type to prevent prisma injection

* refactor: update order of ops for thread fork

---------

Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
2024-07-03 14:44:35 -07:00
timothycarambat
89538c3614 fix typo in agent debug logs 2024-07-03 14:02:33 -07:00
timothycarambat
fbc9c868c4 chore: silence getGitVersion for docker env runtime 2024-07-03 11:50:31 -07:00
Timothy Carambat
cb6b3340f0
Update bgworker to use fork instead of worker_thread (#1808) 2024-07-03 11:44:34 -07:00
Timothy Carambat
0b845fbb1c
Deprecate .isSafe moderation (#1790)
Add type defs to helpers
2024-06-28 15:32:30 -07:00
Sean Hatfield
910eb36cfe
[FIX] OpenAI compatible endpoints query mode developer API bug fix (#1789)
fix query mode always responding with refusal message on develop api openai compatible endpoints
2024-06-28 14:23:16 -07:00
Timothy Carambat
7a78ad3960
Apply workspace topN to agent call (#1779)
resolves #1777
2024-06-27 17:58:28 -07:00
Timothy Carambat
9bd65f1567
[CHORE] Migration from vectordb to @lancedb/lancedb NodeJS SDK (#1766)
WIP on migration to @lancedb/lancedb NodeJS SDK
2024-06-26 21:57:16 -07:00
Sean Hatfield
77916d920b
[FEAT] Remove users_can_delete_workspaces feature (#1742)
remove all references to users_can_delete_workspaces option
2024-06-25 16:46:06 -07:00
Sean Hatfield
dde8bc238b
[FIX] Remove Azure URL validation (#1758)
remove azure url validation
2024-06-25 12:10:51 -07:00
Timothy Carambat
88a0335fd2
Add OpenAI compatible API support (#1757) 2024-06-24 18:28:21 -07:00
Sean Hatfield
e72fa8b370
[FEAT] Generic OpenAI embedding provider (#1664)
* implement generic openai embedding provider

* linting

* comment & description update for generic openai embedding provider

* fix privacy for generic

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-06-21 16:27:02 -07:00
Timothy Carambat
dc4ad6b5a9
[BETA] Live document sync (#1719)
* wip bg workers for live document sync

* Add ability to re-embed specific documents across many workspaces via background queue
bgworkser is gated behind expieremental system setting flag that needs to be explictly enabled
UI for watching/unwatching docments that are embedded.
TODO: UI to easily manage all bg tasks and see run results
TODO: UI to enable this feature and background endpoints to manage it

* create frontend views and paths
Move elements to correct experimental scope

* update migration to delete runs on removal of watched document

* Add watch support to YouTube transcripts (#1716)

* Add watch support to YouTube transcripts
refactor how sync is done for supported types

* Watch specific files in Confluence space (#1718)

Add failure-prune check for runs

* create tmp workflow modifications for beta image

* create tmp workflow modifications for beta image

* create tmp workflow modifications for beta image

* dual build
update copy of alert modals

* update job interval

* Add support for live-sync of Github files

* update copy for document sync feature

* hide Experimental features from UI

* update docs links

* [FEAT] Implement new settings menu for experimental features (#1735)

* implement new settings menu for experimental features

* remove unused context save bar

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* dont run job on boot

* unset workflow changes

* Add persistent encryption service
Relay key to collector so persistent encryption can be used
Encrypt any private data in chunkSources used for replay during resync jobs

* update jsDOC

* Linting and organization

* update modal copy for feature

---------

Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
2024-06-21 13:38:50 -07:00
timothycarambat
c176fe38d2 remove async on dumpEnv 2024-06-20 20:50:14 -07:00
timothycarambat
ed53bbcf15 Merge branch 'master' of github.com:Mintplex-Labs/anything-llm 2024-06-20 18:03:33 -07:00
timothycarambat
9411791fe5 Simplify ENV management
auto write ENV on changes
Dont double-track envs that are already tracked
2024-06-20 18:03:21 -07:00
Sean Hatfield
12bf576e16
[FEAT] Workspace threads developer API endpoints (#1668)
* WIP workspace threads developer api endpoints

* fix swagger docs

* fix broken endpoints and test on single/multi user modes

* support for chatting with workspace thread by userId

* update swagger json
2024-06-20 16:27:35 -07:00
Sean Hatfield
c2523a9593
[FEAT] Persist query mode refusal responses as chat history (#1727)
* log query refusals to workspace chats but hide in ui

* linting

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-06-20 15:44:19 -07:00
Timothy Carambat
8b1ceb30c1
Add support for searXNG search for agents (#1733)
resolves #1367
2024-06-20 14:08:00 -07:00
Sean Hatfield
524edd6e69
[FEAT] Add support for Claude Sonnet 3.5 model (#1731)
add support for claude sonnet 3.5 model
2024-06-20 10:13:53 -07:00
timothycarambat
421c5c6b91 linting 2024-06-18 11:30:50 -07:00
lewismacnow
6c68bdbd51
1711 enhance workspace users api (#1712)
* Update workspace.js

adds userId to the workspaceUsers function

* Update openapi.json

Updates the workspace/users API docs to display correct 200 response example, including userId, username, role and last update datetime.
2024-06-18 09:35:40 -07:00
Sean Hatfield
1c473621aa
[FEAT] Add user management endpoints to list all users via developer API (#1708)
* add user management endpoints to list all users via developer api

* rename users to user management in swagger docs

* linting

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-06-17 16:24:41 -07:00
timothycarambat
0d5cc558c9 linting
patch UI exception with invalid return value from failed parse
resolves #1692
2024-06-17 13:55:10 -07:00
lewismacnow
38441f4b21
1693 get workspace users api (#1694)
* Enhance API - add GET for users with access to workspace

Adds GET request endpoint for retrieving a list of users with permissions to access the specified workspace

* Update Swagger for users with access to workspace

Adds swagger docs for the endpoint used to retrieve users with access to workspace. "v1/admin/workspaces/:workspaceId/users"
2024-06-17 13:52:01 -07:00
timothycarambat
05870ec8d0 chore: update no agent error message
style: update error message width
2024-06-12 13:19:06 -07:00
timothycarambat
4f15f95f53 hotfix: fix bad semantic HTML
feat: add /reset to bailout for agent session
fix: patch save-to-browser not enabling
2024-06-12 12:43:06 -07:00
Sean Hatfield
9c3014de70
[FEAT] Agent skills UI redesign (#1565)
* WIP agent settings redesign

* WIP rework new agent skill UI

* WIP save bar/agent styles

* WIP update settings fix

* desktop agent config UI implementation

* remove unneeded files

* fix sql and web browsing plugins not starting & add default badges

* fix serply merge conflict

* review: cleanup unused files/folders/components

* refactor components

* refactor components

* fix order of customized skills

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2024-06-12 10:52:32 -07:00
timothycarambat
29263dab52 patch: Fix API delete documents 2024-06-11 08:14:13 -07:00
timothycarambat
394b5aafb6 linting 2024-06-10 15:22:32 -07:00