Commit Graph

568 Commits

Author SHA1 Message Date
timothycarambat
c6f0e4cdc4 update docker instructions 2023-11-17 20:15:11 -08:00
timothycarambat
17dfeae0da dockerfile default UID/GUID 2023-11-17 19:21:47 -08:00
timothycarambat
14c56a9bfc bump build workflow 2023-11-17 19:18:26 -08:00
timothycarambat
8176381513 build docker image and push on master merge 2023-11-17 19:15:23 -08:00
timothycarambat
b583aa74fd remove prints 2023-11-16 17:17:52 -08:00
Sean Hatfield
7edfccaf9a
Adding url uploads to document picker (#375)
* WIP adding url uploads to document picker

* fix manual script for uploading url to custom-documents

* fix metadata for url scraping

* wip url parsing

* update how async link scraping works

* docker-compose defaults added
no autocomplete on URLs

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-16 17:15:01 -08:00
Timothy Carambat
d766d128a2
move embedding options to their own component (#387) 2023-11-16 15:40:39 -08:00
Sean Hatfield
5ad8a5f2d0
Allow use of any embedder for any llm/update data handling modal (#386)
* allow use of any embedder for any llm/update data handling modal

* Apply embedder override and fallback to OpenAI and Azure models

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-16 15:19:49 -08:00
Sean Hatfield
2c2543b4d7
fix settings button path on single user mode (#383)
Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-16 14:37:42 -08:00
Sean Hatfield
f40309cfdb
Add id to all metadata to prevent errors in frontend document picker (#378)
add id to all metadata to prevent errors in frontend docuemnt picker

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-16 14:36:26 -08:00
Sean Hatfield
73f342eb19
Warning about switching embedder or vectordb (#385)
* added warning modal to LLM preference

* added warning modal for changing embedder

* remove warning from LLM preference & add warning to vector database selection

* linting

* remove comments and move warning modal to component

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-16 14:35:14 -08:00
timothycarambat
1e3d82e184 patch collector script 2023-11-16 10:25:23 -08:00
timothycarambat
c5dc68633b patch link scrape tool schema 2023-11-14 16:41:39 -08:00
Timothy Carambat
085745c5e4
Prevent lone-admin from locking themselves out the system (#376)
resolves #367
2023-11-14 14:43:40 -08:00
Sean Hatfield
dd33767059
365 update invite UI to match new UI (#370)
* update NewUserModal to match ui styles

* fix bg color of invite screen/auto login after accept invitation

* fix error text color

* cleanup

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-14 13:58:00 -08:00
Sean Hatfield
1aa58dcb7b
Disable prisma logs on prod (#371)
* disable prisma logs on prod

* linting

* keep const top level

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-14 13:53:11 -08:00
Tobias Landenberger
a96a9d41a3
LocalAI for embeddings (#361)
* feature: add localAi as embedding provider

* chore: add LocalAI image

* chore: add localai embedding examples to docker .env.example

* update setting env
pull models from localai API

* update comments on embedder
Dont show cost estimation on UI

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-14 13:49:31 -08:00
timothycarambat
f269124760 fix type error for filetypes 2023-11-14 13:49:18 -08:00
Timothy Carambat
4bb99ab4bf
Support LocalAi as LLM provider by @tlandenberger (#373)
* feature: add LocalAI as llm provider

* update Onboarding/mgmt settings
Grab models from models endpoint for localai
merge with master

* update streaming for complete chunk streaming
update localAI LLM to be able to stream

* force schema on URL

---------
Co-authored-by: timothycarambat <rambat1010@gmail.com>
Co-authored-by: tlandenberger <tobiaslandenberger@gmail.com>
2023-11-14 12:31:44 -08:00
Timothy Carambat
6957bc3ec0
Robots.txt (#369)
* assume default model where appropriate

* merge with master and fix other model refs

* disallow robots

* add public file
2023-11-13 15:22:24 -08:00
Timothy Carambat
8743be679b
assume default model where appropriate (#366)
* assume default model where appropriate

* merge with master and fix other model refs
2023-11-13 15:17:22 -08:00
Timothy Carambat
c22c50cca8
Enable chat streaming for LLMs (#354)
* [Draft] Enable chat streaming for LLMs

* stream only, move sendChat to deprecated

* Update TODO deprecation comments
update console output color for streaming disabled
2023-11-13 15:07:30 -08:00
Sean Hatfield
fa29003a46
Create manager role and limit default role (#351)
* added manager role to options

* block default role from editing workspace settings on workspace and text input box

* block default user from accessing settings at all

* create manager route

* let pass through if in single user mode

* fix permissions for manager and admin roles in settings

* fix settings button for single user and remove unneeded console.logs

* rename routes and paths for clarity

* admin, manager, default roles complete

* remove unneeded comments

* consistency changes

* manage permissions for mum modes

* update sidebar for single-user mode

* update comment on middleware
Modify permission setting for admins

* update render conditional

* Add role usage hint to each role

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-13 14:51:16 -08:00
Timothy Carambat
7fcf29d769
Add better code-block hightlighting and styles (#357)
* Add better code-block hightlighting and styles

* add fallback class
2023-11-10 16:31:21 -08:00
Timothy Carambat
2b17bf26a8
Posthog telemetry updates (#356)
track subuser anon
2023-11-10 16:02:46 -08:00
Tobias Landenberger
2914c09dd5
fix: adjust return type of addDocuments in case of no additions (#353) 2023-11-10 13:27:53 -08:00
timothycarambat
6e72519caf patch onboarding 2023-11-09 15:19:52 -08:00
timothycarambat
195d3cfcc3 patch filename 2023-11-09 13:42:47 -08:00
Francisco Bischoff
f499f1ba59
Using OpenAI API locally (#335)
* Using OpenAI API locally

* Infinite prompt input and compression implementation (#332)

* WIP on continuous prompt window summary

* wip

* Move chat out of VDB
simplify chat interface
normalize LLM model interface
have compression abstraction
Cleanup compressor
TODO: Anthropic stuff

* Implement compression for Anythropic
Fix lancedb sources

* cleanup vectorDBs and check that lance, chroma, and pinecone are returning valid metadata sources

* Resolve Weaviate citation sources not working with schema

* comment cleanup

* disable import on hosted instances (#339)

* disable import on hosted instances

* Update UI on disabled import/export

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>

* Add support for gpt-4-turbo 128K model (#340)

resolves #336
Add support for gpt-4-turbo 128K model

* 315 show citations based on relevancy score (#316)

* settings for similarity score threshold and prisma schema updated

* prisma schema migration for adding similarityScore setting

* WIP

* Min score default change

* added similarityThreshold checking for all vectordb providers

* linting

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>

* rename localai to lmstudio

* forgot files that were renamed

* normalize model interface

* add model and context window limits

* update LMStudio tagline

* Fully working LMStudio integration

---------
Co-authored-by: Francisco Bischoff <984592+franzbischoff@users.noreply.github.com>
Co-authored-by: Timothy Carambat <rambat1010@gmail.com>
Co-authored-by: Sean Hatfield <seanhatfield5@gmail.com>
2023-11-09 12:33:21 -08:00
Timothy Carambat
1ec774ab2e
[Chore] replace all React-feather icons with phosphor icons fully (#349)
replace all React-feather icons with phosphor icons fully
remove package-lock.json files - yarn only
2023-11-09 08:55:20 -08:00
Sean Hatfield
458ffed0c7
added onboarding data handling modal (#342)
* added onboarding data handling modal

* adding data handling modal component

* update element to list
Update copy

* remove useEffect dep

* refactor onboarding navigation using history

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-08 19:00:12 -08:00
Sean Hatfield
997482ef8f
added JSONL export to workspace chats (#345)
* added JSONL export to workspace chats

* change permissions for workspace chat settings

* change permissions for workspace chat settings

* Show error for correct limit on fine-tune
Change sidebar position and permission
Remove check for MUM

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-08 17:36:54 -08:00
Timothy Carambat
88d4808c52
315 show citations based on relevancy score (#316)
* settings for similarity score threshold and prisma schema updated

* prisma schema migration for adding similarityScore setting

* WIP

* Min score default change

* added similarityThreshold checking for all vectordb providers

* linting

---------

Co-authored-by: shatfield4 <seanhatfield5@gmail.com>
2023-11-06 16:49:29 -08:00
Timothy Carambat
d34ec68702
Add support for gpt-4-turbo 128K model (#340)
resolves #336
Add support for gpt-4-turbo 128K model
2023-11-06 14:22:19 -08:00
Sean Hatfield
5673dc522b
disable import on hosted instances (#339)
* disable import on hosted instances

* Update UI on disabled import/export

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-06 13:23:12 -08:00
Timothy Carambat
be9d8b0397
Infinite prompt input and compression implementation (#332)
* WIP on continuous prompt window summary

* wip

* Move chat out of VDB
simplify chat interface
normalize LLM model interface
have compression abstraction
Cleanup compressor
TODO: Anthropic stuff

* Implement compression for Anythropic
Fix lancedb sources

* cleanup vectorDBs and check that lance, chroma, and pinecone are returning valid metadata sources

* Resolve Weaviate citation sources not working with schema

* comment cleanup
2023-11-06 13:13:53 -08:00
Timothy Carambat
0751fb1fdd
Fix missing import on API (#333) 2023-11-03 12:50:56 -07:00
Sean Hatfield
eebafd7327
Replace old modals to match UI (#328)
* workspace not found modal updated to match ui

* update text preview modal to match new ui

* Mobile styles

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-11-02 16:17:02 -07:00
timothycarambat
c3abbfbf27 Fix admin chat pagination 2023-11-02 16:12:29 -07:00
timothycarambat
7d3d0e03c6 bind docker to docker/.env so dumps persist between restarts 2023-11-01 22:12:30 -07:00
timothycarambat
2afe3faed8 change how node is installed on docker 2023-11-01 20:10:49 -07:00
Sean Hatfield
ab1e9d66b5
Use toast in workspace settings tab (#322)
* fix white border around user avatar

* add toast into settings menu on change/error
2023-11-01 18:21:30 -07:00
Timothy Carambat
5441717294
normalize parser struct for all file types (#321) 2023-11-01 16:44:02 -07:00
Sean Hatfield
1c5d55c425
Fix white border around user avatar glitch (#320)
fix white border around user avatar
2023-11-01 14:17:27 -07:00
timothycarambat
24823cb5e2 patch workspace chat history windows to persist most recent chats, not the top n 2023-11-01 14:12:27 -07:00
Timothy Carambat
67c85f1550
Implement retrieval and use of fine-tune models (#314)
* Implement retrieval and use of fine-tune models
Cleanup LLM selection code
resolves #311

* Cleanup from PR bot
2023-10-31 11:38:28 -07:00
timothycarambat
745d2aeaff fix import path 2023-10-30 15:49:29 -07:00
Timothy Carambat
5d56ab623b
Anthropic claude 2 support (#305)
* WIP Anythropic support for chat, chat and query w/context

* Add onboarding support for Anthropic

* cleanup

* fix Anthropic answer parsing
move embedding selector to general util
2023-10-30 15:44:03 -07:00
Sean Hatfield
669d7a396d
282 return relevancy score with similarityresponse (#304)
* include score value in similarityResponse for weaviate

* include score value in si
milarityResponse for qdrant

* include score value in si
milarityResponse for pinecone

* include score value in similarityResponse for chroma

* include score value in similarityResponse for lancedb

* distance to similarity

---------

Co-authored-by: timothycarambat <rambat1010@gmail.com>
2023-10-30 12:46:38 -07:00
Francisco Bischoff
26dba59249
mbox parsing improvements v1 (#308)
* mbox parsing improvements v1

* autobots roll out!
2023-10-30 11:57:33 -07:00