diff --git a/README.md b/README.md index 233392ed..e7cc7c80 100644 --- a/README.md +++ b/README.md @@ -74,10 +74,10 @@ Some cool features of AnythingLLM ### Technical Overview This monorepo consists of three main sections: -- `collector`: Python tools that enable you to quickly convert online resources or local documents into LLM useable format. - `frontend`: A viteJS + React frontend that you can run to easily create and manage all your content the LLM can use. -- `server`: A nodeJS + express server to handle all the interactions and do all the vectorDB management and LLM interactions. +- `server`: A NodeJS express server to handle all the interactions and do all the vectorDB management and LLM interactions. - `docker`: Docker instructions and build process + information for building from source. +- `collector`: NodeJS express server that process and parses documents from the UI. ### Minimum Requirements > [!TIP] @@ -86,7 +86,6 @@ This monorepo consists of three main sections: > you will be storing (documents, vectors, models, etc). Minimum 10GB recommended. - `yarn` and `node` on your machine -- `python` 3.9+ for running scripts in `collector/`. - access to an LLM running locally or remotely. *AnythingLLM by default uses a built-in vector database powered by [LanceDB](https://github.com/lancedb/lancedb) @@ -112,6 +111,7 @@ export STORAGE_LOCATION="/var/lib/anythingllm" && \ mkdir -p $STORAGE_LOCATION && \ touch "$STORAGE_LOCATION/.env" && \ docker run -d -p 3001:3001 \ +--cap-add SYS_ADMIN \ -v ${STORAGE_LOCATION}:/app/server/storage \ -v ${STORAGE_LOCATION}/.env:/app/server/.env \ -e STORAGE_DIR="/app/server/storage" \ @@ -141,12 +141,6 @@ To boot the frontend locally (run commands from root of repo): [Learn about vector caching](./server/storage/vector-cache/VECTOR_CACHE.md) -## Standalone scripts - -This repo contains standalone scripts you can run to collect data from a Youtube Channel, Medium articles, local text files, word documents, and the list goes on. This is where you will use the `collector/` part of the repo. - -[Go set up and run collector scripts](./collector/README.md) - ## Contributing - create issue - create PR with branch name format of `-` diff --git a/cloud-deployments/aws/cloudformation/DEPLOY.md b/cloud-deployments/aws/cloudformation/DEPLOY.md index f3af941a..a06026c4 100644 --- a/cloud-deployments/aws/cloudformation/DEPLOY.md +++ b/cloud-deployments/aws/cloudformation/DEPLOY.md @@ -1,6 +1,6 @@ # How to deploy a private AnythingLLM instance on AWS -With an AWS account you can easily deploy a private AnythingLLM instance on AWS. This will create a url that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys and they will not be exposed - however if you want your instance to be protected it is highly recommend that you set the `AUTH_TOKEN` and `JWT_SECRET` variables in the `docker/` ENV. +With an AWS account you can easily deploy a private AnythingLLM instance on AWS. This will create a url that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys and they will not be exposed - however if you want your instance to be protected it is highly recommend that you set a password one setup is complete. **Quick Launch (EASY)** 1. Log in to your AWS account @@ -30,12 +30,11 @@ The output of this cloudformation stack will be: **Requirements** - An AWS account with billing information. - - AnythingLLM (GUI + document processor) must use a t2.small minimum and 10Gib SSD hard disk volume ## Please read this notice before submitting issues about your deployment **Note:** -Your instance will not be available instantly. Depending on the instance size you launched with it can take varying amounts of time to fully boot up. +Your instance will not be available instantly. Depending on the instance size you launched with it can take 5-10 minutes to fully boot up. If you want to check the instance's progress, navigate to [your deployed EC2 instances](https://us-west-1.console.aws.amazon.com/ec2/home) and connect to your instance via SSH in browser. diff --git a/cloud-deployments/aws/cloudformation/cloudformation_create_anythingllm.json b/cloud-deployments/aws/cloudformation/cloudformation_create_anythingllm.json index 1c944f4b..313a4ecd 100644 --- a/cloud-deployments/aws/cloudformation/cloudformation_create_anythingllm.json +++ b/cloud-deployments/aws/cloudformation/cloudformation_create_anythingllm.json @@ -89,7 +89,7 @@ "touch /home/ec2-user/anythingllm/.env\n", "sudo chown ec2-user:ec2-user -R /home/ec2-user/anythingllm\n", "docker pull mintplexlabs/anythingllm:master\n", - "docker run -d -p 3001:3001 -v /home/ec2-user/anythingllm:/app/server/storage -v /home/ec2-user/anythingllm/.env:/app/server/.env -e STORAGE_DIR=\"/app/server/storage\" mintplexlabs/anythingllm:master\n", + "docker run -d -p 3001:3001 --cap-add SYS_ADMIN -v /home/ec2-user/anythingllm:/app/server/storage -v /home/ec2-user/anythingllm/.env:/app/server/.env -e STORAGE_DIR=\"/app/server/storage\" mintplexlabs/anythingllm:master\n", "echo \"Container ID: $(sudo docker ps --latest --quiet)\"\n", "export ONLINE=$(curl -Is http://localhost:3001/api/ping | head -n 1|cut -d$' ' -f2)\n", "echo \"Health check: $ONLINE\"\n", diff --git a/cloud-deployments/digitalocean/terraform/DEPLOY.md b/cloud-deployments/digitalocean/terraform/DEPLOY.md index 1e78cf82..1877abc2 100644 --- a/cloud-deployments/digitalocean/terraform/DEPLOY.md +++ b/cloud-deployments/digitalocean/terraform/DEPLOY.md @@ -1,8 +1,6 @@ # How to deploy a private AnythingLLM instance on DigitalOcean using Terraform -With a DigitalOcean account, you can easily deploy a private AnythingLLM instance using Terraform. This will create a URL that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys, and they will not be exposed. However, if you want your instance to be protected, it is highly recommended that you set the `AUTH_TOKEN` and `JWT_SECRET` variables in the `docker/` ENV. - -[Refer to .env.example](../../../docker/HOW_TO_USE_DOCKER.md) for data format. +With a DigitalOcean account, you can easily deploy a private AnythingLLM instance using Terraform. This will create a URL that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys, and they will not be exposed. However, if you want your instance to be protected, it is highly recommended that you set a password one setup is complete. The output of this Terraform configuration will be: - 1 DigitalOcean Droplet @@ -12,8 +10,6 @@ The output of this Terraform configuration will be: - An DigitalOcean account with billing information - Terraform installed on your local machine - Follow the instructions in the [official Terraform documentation](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) for your operating system. -- `.env` file that is filled out with your settings and set up in the `docker/` folder - ## How to deploy on DigitalOcean Open your terminal and navigate to the `digitalocean/terraform` folder @@ -36,7 +32,7 @@ terraform destroy ## Please read this notice before submitting issues about your deployment **Note:** -Your instance will not be available instantly. Depending on the instance size you launched with it can take anywhere from 10-20 minutes to fully boot up. +Your instance will not be available instantly. Depending on the instance size you launched with it can take anywhere from 5-10 minutes to fully boot up. If you want to check the instances progress, navigate to [your deployed instances](https://cloud.digitalocean.com/droplets) and connect to your instance via SSH in browser. diff --git a/cloud-deployments/digitalocean/terraform/user_data.tp1 b/cloud-deployments/digitalocean/terraform/user_data.tp1 index a8705be4..33455a4e 100644 --- a/cloud-deployments/digitalocean/terraform/user_data.tp1 +++ b/cloud-deployments/digitalocean/terraform/user_data.tp1 @@ -12,7 +12,7 @@ mkdir -p /home/anythingllm touch /home/anythingllm/.env sudo docker pull mintplexlabs/anythingllm:master -sudo docker run -d -p 3001:3001 -v /home/anythingllm:/app/server/storage -v /home/anythingllm/.env:/app/server/.env -e STORAGE_DIR="/app/server/storage" mintplexlabs/anythingllm:master +sudo docker run -d -p 3001:3001 --cap-add SYS_ADMIN -v /home/anythingllm:/app/server/storage -v /home/anythingllm/.env:/app/server/.env -e STORAGE_DIR="/app/server/storage" mintplexlabs/anythingllm:master echo "Container ID: $(sudo docker ps --latest --quiet)" export ONLINE=$(curl -Is http://localhost:3001/api/ping | head -n 1|cut -d$' ' -f2) diff --git a/cloud-deployments/gcp/deployment/DEPLOY.md b/cloud-deployments/gcp/deployment/DEPLOY.md index 10249096..c659f0d8 100644 --- a/cloud-deployments/gcp/deployment/DEPLOY.md +++ b/cloud-deployments/gcp/deployment/DEPLOY.md @@ -1,8 +1,6 @@ # How to deploy a private AnythingLLM instance on GCP -With a GCP account you can easily deploy a private AnythingLLM instance on GCP. This will create a url that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys and they will not be exposed - however if you want your instance to be protected it is highly recommend that you set the `AUTH_TOKEN` and `JWT_SECRET` variables in the `docker/` ENV. - -[Refer to .env.example](../../../docker/HOW_TO_USE_DOCKER.md) for data format. +With a GCP account you can easily deploy a private AnythingLLM instance on GCP. This will create a url that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys and they will not be exposed - however if you want your instance to be protected it is highly recommend that you set a password one setup is complete. The output of this cloudformation stack will be: - 1 GCP VM @@ -11,19 +9,15 @@ The output of this cloudformation stack will be: **Requirements** - An GCP account with billing information. - - AnythingLLM (GUI + document processor) must use a n1-standard-1 minimum and 10Gib SSD hard disk volume -- `.env` file that is filled out with your settings and set up in the `docker/` folder ## How to deploy on GCP Open your terminal -1. Generate your specific cloudformation document by running `yarn generate:gcp_deployment` from the project root directory. -2. This will create a new file (`gcp_deploy_anything_llm_with_env.yaml`) in the `gcp/deployment` folder. -3. Log in to your GCP account using the following command: +1. Log in to your GCP account using the following command: ``` gcloud auth login ``` -4. After successful login, Run the following command to create a deployment using the Deployment Manager CLI: +2. After successful login, Run the following command to create a deployment using the Deployment Manager CLI: ``` @@ -57,5 +51,4 @@ If you want to check the instances progress, navigate to [your deployed instance Once connected run `sudo tail -f /var/log/cloud-init-output.log` and wait for the file to conclude deployment of the docker image. - -Additionally, your use of this deployment process means you are responsible for any costs of these GCP resources fully. +Additionally, your use of this deployment process means you are responsible for any costs of these GCP resources fully. \ No newline at end of file diff --git a/cloud-deployments/gcp/deployment/gcp_deploy_anything_llm.yaml b/cloud-deployments/gcp/deployment/gcp_deploy_anything_llm.yaml index 74658327..ad9a908f 100644 --- a/cloud-deployments/gcp/deployment/gcp_deploy_anything_llm.yaml +++ b/cloud-deployments/gcp/deployment/gcp_deploy_anything_llm.yaml @@ -34,7 +34,7 @@ resources: touch /home/anythingllm/.env sudo docker pull mintplexlabs/anythingllm:master - sudo docker run -d -p 3001:3001 -v /home/anythingllm:/app/server/storage -v /home/anythingllm/.env:/app/server/.env -e STORAGE_DIR="/app/server/storage" mintplexlabs/anythingllm:master + sudo docker run -d -p 3001:3001 --cap-add SYS_ADMIN -v /home/anythingllm:/app/server/storage -v /home/anythingllm/.env:/app/server/.env -e STORAGE_DIR="/app/server/storage" mintplexlabs/anythingllm:master echo "Container ID: $(sudo docker ps --latest --quiet)" export ONLINE=$(curl -Is http://localhost:3001/api/ping | head -n 1|cut -d$' ' -f2) diff --git a/cloud-deployments/gcp/deployment/generate.mjs b/cloud-deployments/gcp/deployment/generate.mjs deleted file mode 100644 index a8868251..00000000 --- a/cloud-deployments/gcp/deployment/generate.mjs +++ /dev/null @@ -1,61 +0,0 @@ -import fs from 'fs'; -import { fileURLToPath } from 'url'; -import path, { dirname } from 'path'; -import { exit } from 'process'; -const __dirname = dirname(fileURLToPath(import.meta.url)); -const REPLACEMENT_KEY = '!SUB::USER::CONTENT!' - -const envPath = path.resolve(__dirname, `../../../docker/.env`) -const envFileExists = fs.existsSync(envPath); - -const chalk = { - redBright: function (text) { - return `\x1b[31m${text}\x1b[0m` - }, - cyan: function (text) { - return `\x1b[36m${text}\x1b[0m` - }, - greenBright: function (text) { - return `\x1b[32m${text}\x1b[0m` - }, - blueBright: function (text) { - return `\x1b[34m${text}\x1b[0m` - } -} - -if (!envFileExists) { - console.log(chalk.redBright('[ABORT]'), 'You do not have an .env file in your ./docker/ folder. You need to create it first.'); - console.log('You can start by running', chalk.cyan('cp -n ./docker/.env.example ./docker/.env')) - exit(1); -} - -// Remove comments -// Remove UID,GID,etc -// Remove empty strings -// Split into array -const settings = fs.readFileSync(envPath, "utf8") - .replace(/^#.*\n?/gm, '') - .replace(/^UID.*\n?/gm, '') - .replace(/^GID.*\n?/gm, '') - .replace(/^CLOUD_BUILD.*\n?/gm, '') - .replace(/^\s*\n/gm, "") - .split('\n') - .filter((i) => !!i); -const formattedSettings = settings.map((i, index) => index === 0 ? i + '\n' : ' ' + i).join('\n'); - -// Read the existing GCP Deployment Manager template -const templatePath = path.resolve(__dirname, `gcp_deploy_anything_llm.yaml`); -const templateString = fs.readFileSync(templatePath, "utf8"); - -// Update the metadata section with the UserData content -const updatedTemplateString = templateString.replace(REPLACEMENT_KEY, formattedSettings); - -// Save the updated GCP Deployment Manager template -const output = path.resolve(__dirname, `gcp_deploy_anything_llm_with_env.yaml`); -fs.writeFileSync(output, updatedTemplateString, "utf8"); - -console.log(chalk.greenBright('[SUCCESS]'), 'Deploy AnythingLLM on GCP Deployment Manager using your template document.'); -console.log(chalk.greenBright('File Created:'), 'gcp_deploy_anything_llm_with_env.yaml in the output directory.'); -console.log(chalk.blueBright('[INFO]'), 'Refer to the GCP Deployment Manager documentation for how to use this file.'); - -exit(); diff --git a/collector/.env.example b/collector/.env.example deleted file mode 100644 index 32e96968..00000000 --- a/collector/.env.example +++ /dev/null @@ -1 +0,0 @@ -GOOGLE_APIS_KEY= \ No newline at end of file diff --git a/collector/.gitignore b/collector/.gitignore index 3aee7f97..57436cef 100644 --- a/collector/.gitignore +++ b/collector/.gitignore @@ -1,8 +1,6 @@ -outputs/*/*.json hotdir/* -hotdir/processed/* -hotdir/failed/* !hotdir/__HOTDIR__.md -!hotdir/processed -!hotdir/failed - +yarn-error.log +!yarn.lock +outputs +scripts diff --git a/collector/.nvmrc b/collector/.nvmrc new file mode 100644 index 00000000..59f4a2f3 --- /dev/null +++ b/collector/.nvmrc @@ -0,0 +1 @@ +v18.13.0 \ No newline at end of file diff --git a/collector/README.md b/collector/README.md deleted file mode 100644 index ad72b856..00000000 --- a/collector/README.md +++ /dev/null @@ -1,62 +0,0 @@ -# How to collect data for vectorizing -This process should be run first. This will enable you to collect a ton of data across various sources. Currently the following services are supported: -- [x] YouTube Channels -- [x] Medium -- [x] Substack -- [x] Arbitrary Link -- [x] Gitbook -- [x] Local Files (.txt, .pdf, etc) [See full list](./hotdir/__HOTDIR__.md) -_these resources are under development or require PR_ -- Twitter -![Choices](../images/choices.png) - -### Requirements -- [ ] Python 3.8+ -- [ ] Google Cloud Account (for YouTube channels) -- [ ] `brew install pandoc` [pandoc](https://pandoc.org/installing.html) (for .ODT document processing) - -### Setup -This example will be using python3.9, but will work with 3.8+. Tested on MacOs. Untested on Windows -- install virtualenv for python3.8+ first before any other steps. `python3.9 -m pip install virtualenv` -- `cd collector` from root directory -- `python3.9 -m virtualenv v-env` -- `source v-env/bin/activate` -- `pip install -r requirements.txt` -- `cp .env.example .env` -- `python main.py` for interactive collection or `python watch.py` to process local documents. -- Select the option you want and follow follow the prompts - Done! -- run `deactivate` to get back to regular shell - -### Outputs -All JSON file data is cached in the `output/` folder. This is to prevent redundant API calls to services which may have rate limits to quota caps. Clearing out the `output/` folder will execute the script as if there was no cache. - -As files are processed you will see data being written to both the `collector/outputs` folder as well as the `server/documents` folder. Later in this process, once you boot up the server you will then bulk vectorize this content from a simple UI! - -If collection fails at any point in the process it will pick up where it last bailed out so you are not reusing credits. - -### Running the document processing API locally -From the `collector` directory with the `v-env` active run `flask run --host '0.0.0.0' --port 8888`. -Now uploads from the frontend will be processed as if you ran the `watch.py` script manually. - -**Docker**: If you run this application via docker the API is already started for you and no additional action is needed. - -### How to get a Google Cloud API Key (YouTube data collection only) -**required to fetch YouTube transcripts and data** -- Have a google account -- [Visit the GCP Cloud Console](https://console.cloud.google.com/welcome) -- Click on dropdown in top right > Create new project. Name it whatever you like - - ![GCP Project Bar](../images/gcp-project-bar.png) -- [Enable YouTube Data APIV3](https://console.cloud.google.com/apis/library/youtube.googleapis.com) -- Once enabled generate a Credential key for this API -- Paste your key after `GOOGLE_APIS_KEY=` in your `collector/.env` file. - -### Using ther Twitter API -***required to get data form twitter with tweepy** -- Go to https://developer.twitter.com/en/portal/dashboard with your twitter account -- Create a new Project App - - Get your 4 keys and place them in your `collector.env` file - * TW_CONSUMER_KEY - * TW_CONSUMER_SECRET - * TW_ACCESS_TOKEN - * TW_ACCESS_TOKEN_SECRET - populate the .env with the values diff --git a/collector/api.py b/collector/api.py deleted file mode 100644 index ae265fe8..00000000 --- a/collector/api.py +++ /dev/null @@ -1,32 +0,0 @@ -import os -from flask import Flask, json, request -from scripts.watch.process_single import process_single -from scripts.watch.filetypes import ACCEPTED_MIMES -from scripts.link import process_single_link -api = Flask(__name__) - -WATCH_DIRECTORY = "hotdir" -@api.route('/process', methods=['POST']) -def process_file(): - content = request.json - target_filename = os.path.normpath(content.get('filename')).lstrip(os.pardir + os.sep) - print(f"Processing {target_filename}") - success, reason = process_single(WATCH_DIRECTORY, target_filename) - return json.dumps({'filename': target_filename, 'success': success, 'reason': reason}) - -@api.route('/process-link', methods=['POST']) -async def process_link(): - content = request.json - url = content.get('link') - print(f"Processing {url}") - success, reason = await process_single_link(url) - return json.dumps({'url': url, 'success': success, 'reason': reason}) - - -@api.route('/accepts', methods=['GET']) -def get_accepted_filetypes(): - return json.dumps(ACCEPTED_MIMES) - -@api.route('/', methods=['GET']) -def root(): - return "

Use POST /process with filename key in JSON body in order to process a file. File by that name must exist in hotdir already.

" \ No newline at end of file diff --git a/collector/hotdir/__HOTDIR__.md b/collector/hotdir/__HOTDIR__.md index 2f1c8da5..8f20efc4 100644 --- a/collector/hotdir/__HOTDIR__.md +++ b/collector/hotdir/__HOTDIR__.md @@ -1,17 +1,3 @@ ### What is the "Hot directory" -This is the location where you can dump all supported file types and have them automatically converted and prepared to be digested by the vectorizing service and selected from the AnythingLLM frontend. - -Files dropped in here will only be processed when you are running `python watch.py` from the `collector` directory. - -Once converted the original file will be moved to the `hotdir/processed` folder so that the original document is still able to be linked to when referenced when attached as a source document during chatting. - -**Supported File types** -- `.md` -- `.txt` -- `.pdf` - -__requires more development__ -- `.png .jpg etc` -- `.mp3` -- `.mp4` +This is a pre-set file location that documents will be written to when uploaded by AnythingLLM. There is really no need to touch it. \ No newline at end of file diff --git a/collector/index.js b/collector/index.js new file mode 100644 index 00000000..a4506673 --- /dev/null +++ b/collector/index.js @@ -0,0 +1,78 @@ +process.env.NODE_ENV === "development" + ? require("dotenv").config({ path: `.env.${process.env.NODE_ENV}` }) + : require("dotenv").config(); + +const express = require("express"); +const bodyParser = require("body-parser"); +const cors = require("cors"); +const path = require("path"); +const { ACCEPTED_MIMES } = require("./utils/constants"); +const { reqBody } = require("./utils/http"); +const { processSingleFile } = require("./processSingleFile"); +const { processLink } = require("./processLink"); +const app = express(); + +app.use(cors({ origin: true })); +app.use( + bodyParser.text(), + bodyParser.json(), + bodyParser.urlencoded({ + extended: true, + }) +); + +app.post("/process", async function (request, response) { + const { filename } = reqBody(request); + try { + const targetFilename = path + .normalize(filename) + .replace(/^(\.\.(\/|\\|$))+/, ""); + const { success, reason } = await processSingleFile(targetFilename); + response.status(200).json({ filename: targetFilename, success, reason }); + } catch (e) { + console.error(e); + response.status(200).json({ + filename: filename, + success: false, + reason: "A processing error occurred.", + }); + } + return; +}); + +app.post("/process-link", async function (request, response) { + const { link } = reqBody(request); + try { + const { success, reason } = await processLink(link); + response.status(200).json({ url: link, success, reason }); + } catch (e) { + console.error(e); + response.status(200).json({ + url: link, + success: false, + reason: "A processing error occurred.", + }); + } + return; +}); + +app.get("/accepts", function (_, response) { + response.status(200).json(ACCEPTED_MIMES); +}); + +app.all("*", function (_, response) { + response.sendStatus(200); +}); + +app + .listen(8888, async () => { + console.log(`Document processor app listening on port 8888`); + }) + .on("error", function (_) { + process.once("SIGUSR2", function () { + process.kill(process.pid, "SIGUSR2"); + }); + process.on("SIGINT", function () { + process.kill(process.pid, "SIGINT"); + }); + }); diff --git a/collector/main.py b/collector/main.py deleted file mode 100644 index a7d3a7e7..00000000 --- a/collector/main.py +++ /dev/null @@ -1,84 +0,0 @@ -import os -from InquirerPy import inquirer -from scripts.youtube import youtube -from scripts.link import link, links, crawler -from scripts.substack import substack -from scripts.medium import medium -from scripts.gitbook import gitbook -from scripts.sitemap import sitemap -from scripts.twitter import twitter - -def main(): - if os.name == 'nt': - methods = { - '1': 'YouTube Channel', - '2': 'Article or Blog Link', - '3': 'Substack', - '4': 'Medium', - '5': 'Gitbook', - '6': 'Twitter', - '7': 'Sitemap', - } - print("There are options for data collection to make this easier for you.\nType the number of the method you wish to execute.") - print("1. YouTube Channel\n2. Article or Blog Link (Single)\n3. Substack\n4. Medium\n\n[In development]:\nTwitter\n\n") - selection = input("Your selection: ") - method = methods.get(str(selection)) - else: - method = inquirer.select( - message="What kind of data would you like to add to convert into long-term memory?", - choices=[ - {"name": "YouTube Channel", "value": "YouTube Channel"}, - {"name": "Substack", "value": "Substack"}, - {"name": "Medium", "value": "Medium"}, - {"name": "Article or Blog Link(s)", "value": "Article or Blog Link(s)"}, - {"name": "Gitbook", "value": "Gitbook"}, - {"name": "Twitter", "value": "Twitter"}, - {"name": "Sitemap", "value": "Sitemap"}, - {"name": "Abort", "value": "Abort"}, - ], - ).execute() - - if 'Article or Blog Link' in method: - method = inquirer.select( - message="Do you want to scrape a single article/blog/url or many at once?", - choices=[ - {"name": "Single URL", "value": "Single URL"}, - {"name": "Multiple URLs", "value": "Multiple URLs"}, - {"name": "URL Crawler", "value": "URL Crawler"}, - {"name": "Abort", "value": "Abort"}, - ], - ).execute() - if method == 'Single URL': - link() - exit(0) - if method == 'Multiple URLs': - links() - exit(0) - if method == 'URL Crawler': - crawler() - exit(0) - - if method == 'Abort': exit(0) - if method == 'YouTube Channel': - youtube() - exit(0) - if method == 'Substack': - substack() - exit(0) - if method == 'Medium': - medium() - exit(0) - if method == 'Gitbook': - gitbook() - exit(0) - if method == 'Sitemap': - sitemap() - exit(0) - if method == 'Twitter': - twitter() - exit(0) - print("Selection was not valid.") - exit(1) - -if __name__ == "__main__": - main() \ No newline at end of file diff --git a/collector/nodemon.json b/collector/nodemon.json new file mode 100644 index 00000000..23e9d55e --- /dev/null +++ b/collector/nodemon.json @@ -0,0 +1,3 @@ +{ + "events": {} +} \ No newline at end of file diff --git a/collector/package.json b/collector/package.json new file mode 100644 index 00000000..c85890d1 --- /dev/null +++ b/collector/package.json @@ -0,0 +1,42 @@ +{ + "name": "anything-llm-document-collector", + "version": "0.2.0", + "description": "Document collector server endpoints", + "main": "index.js", + "author": "Timothy Carambat (Mintplex Labs)", + "license": "MIT", + "private": false, + "engines": { + "node": ">=18.12.1" + }, + "scripts": { + "dev": "NODE_ENV=development nodemon --trace-warnings index.js", + "start": "NODE_ENV=production node index.js", + "lint": "yarn prettier --write ./processSingleFile ./processLink ./utils index.js" + }, + "dependencies": { + "@googleapis/youtube": "^9.0.0", + "bcrypt": "^5.1.0", + "body-parser": "^1.20.2", + "cors": "^2.8.5", + "dotenv": "^16.0.3", + "express": "^4.18.2", + "extract-zip": "^2.0.1", + "js-tiktoken": "^1.0.8", + "langchain": "0.0.201", + "mammoth": "^1.6.0", + "mbox-parser": "^1.0.1", + "mime": "^3.0.0", + "moment": "^2.29.4", + "multer": "^1.4.5-lts.1", + "officeparser": "^4.0.5", + "pdf-parse": "^1.1.1", + "puppeteer": "^21.6.1", + "slugify": "^1.6.6", + "uuid": "^9.0.0" + }, + "devDependencies": { + "nodemon": "^2.0.22", + "prettier": "^2.4.1" + } +} \ No newline at end of file diff --git a/collector/processLink/convert/generic.js b/collector/processLink/convert/generic.js new file mode 100644 index 00000000..6af41f6c --- /dev/null +++ b/collector/processLink/convert/generic.js @@ -0,0 +1,72 @@ +const { v4 } = require("uuid"); +const { + PuppeteerWebBaseLoader, +} = require("langchain/document_loaders/web/puppeteer"); +const { writeToServerDocuments } = require("../../utils/files"); +const { tokenizeString } = require("../../utils/tokenizer"); +const { default: slugify } = require("slugify"); + +async function scrapeGenericUrl(link) { + console.log(`-- Working URL ${link} --`); + const content = await getPageContent(link); + + if (!content.length) { + console.error(`Resulting URL content was empty at ${link}.`); + return { success: false, reason: `No URL content found at ${link}.` }; + } + + const url = new URL(link); + const filename = (url.host + "-" + url.pathname).replace(".", "_"); + + data = { + id: v4(), + url: "file://" + slugify(filename) + ".html", + title: slugify(filename) + ".html", + docAuthor: "no author found", + description: "No description found.", + docSource: "URL link uploaded by the user.", + chunkSource: slugify(link) + ".html", + published: new Date().toLocaleString(), + wordCount: content.split(" ").length, + pageContent: content, + token_count_estimate: tokenizeString(content).length, + }; + + writeToServerDocuments(data, `url-${slugify(filename)}-${data.id}`); + console.log(`[SUCCESS]: URL ${link} converted & ready for embedding.\n`); + return { success: true, reason: null }; +} + +async function getPageContent(link) { + try { + let pageContents = []; + const loader = new PuppeteerWebBaseLoader(link, { + launchOptions: { + headless: "new", + }, + gotoOptions: { + waitUntil: "domcontentloaded", + }, + async evaluate(page, browser) { + const result = await page.evaluate(() => document.body.innerText); + await browser.close(); + return result; + }, + }); + + const docs = await loader.load(); + + for (const doc of docs) { + pageContents.push(doc.pageContent); + } + + return pageContents.join(" "); + } catch (error) { + console.error("getPageContent failed!", error); + } + return null; +} + +module.exports = { + scrapeGenericUrl, +}; diff --git a/collector/processLink/index.js b/collector/processLink/index.js new file mode 100644 index 00000000..bd3ced19 --- /dev/null +++ b/collector/processLink/index.js @@ -0,0 +1,11 @@ +const { validURL } = require("../utils/url"); +const { scrapeGenericUrl } = require("./convert/generic"); + +async function processLink(link) { + if (!validURL(link)) return { success: false, reason: "Not a valid URL." }; + return await scrapeGenericUrl(link); +} + +module.exports = { + processLink, +}; diff --git a/collector/processSingleFile/convert/asDocx.js b/collector/processSingleFile/convert/asDocx.js new file mode 100644 index 00000000..7c336865 --- /dev/null +++ b/collector/processSingleFile/convert/asDocx.js @@ -0,0 +1,51 @@ +const { v4 } = require("uuid"); +const { DocxLoader } = require("langchain/document_loaders/fs/docx"); +const { + createdDate, + trashFile, + writeToServerDocuments, +} = require("../../utils/files"); +const { tokenizeString } = require("../../utils/tokenizer"); +const { default: slugify } = require("slugify"); + +async function asDocX({ fullFilePath = "", filename = "" }) { + const loader = new DocxLoader(fullFilePath); + + console.log(`-- Working ${filename} --`); + let pageContent = []; + const docs = await loader.load(); + for (const doc of docs) { + console.log(doc.metadata); + console.log(`-- Parsing content from docx page --`); + if (!doc.pageContent.length) continue; + pageContent.push(doc.pageContent); + } + + if (!pageContent.length) { + console.error(`Resulting text content was empty for ${filename}.`); + trashFile(fullFilePath); + return { success: false, reason: `No text content found in ${filename}.` }; + } + + const content = pageContent.join(""); + data = { + id: v4(), + url: "file://" + fullFilePath, + title: filename, + docAuthor: "no author found", + description: "No description found.", + docSource: "pdf file uploaded by the user.", + chunkSource: filename, + published: createdDate(fullFilePath), + wordCount: content.split(" ").length, + pageContent: content, + token_count_estimate: tokenizeString(content).length, + }; + + writeToServerDocuments(data, `${slugify(filename)}-${data.id}`); + trashFile(fullFilePath); + console.log(`[SUCCESS]: ${filename} converted & ready for embedding.\n`); + return { success: true, reason: null }; +} + +module.exports = asDocX; diff --git a/collector/processSingleFile/convert/asMbox.js b/collector/processSingleFile/convert/asMbox.js new file mode 100644 index 00000000..58df98cd --- /dev/null +++ b/collector/processSingleFile/convert/asMbox.js @@ -0,0 +1,65 @@ +const { v4 } = require("uuid"); +const fs = require("fs"); +const { mboxParser } = require("mbox-parser"); +const { + createdDate, + trashFile, + writeToServerDocuments, +} = require("../../utils/files"); +const { tokenizeString } = require("../../utils/tokenizer"); +const { default: slugify } = require("slugify"); + +async function asMbox({ fullFilePath = "", filename = "" }) { + console.log(`-- Working ${filename} --`); + + const mails = await mboxParser(fs.createReadStream(fullFilePath)) + .then((mails) => mails) + .catch((error) => { + console.log(`Could not parse mail items`, error); + return []; + }); + + if (!mails.length) { + console.error(`Resulting mail items was empty for ${filename}.`); + trashFile(fullFilePath); + return { success: false, reason: `No mail items found in ${filename}.` }; + } + + let item = 1; + for (const mail of mails) { + if (!mail.hasOwnProperty("text")) continue; + + const content = mail.text; + if (!content) continue; + console.log( + `-- Working on message "${mail.subject || "Unknown subject"}" --` + ); + + data = { + id: v4(), + url: "file://" + fullFilePath, + title: mail?.subject + ? slugify(mail?.subject?.replace(".", "")) + ".mbox" + : `msg_${item}-${filename}`, + docAuthor: mail?.from?.text, + description: "No description found.", + docSource: "Mbox message file uploaded by the user.", + chunkSource: filename, + published: createdDate(fullFilePath), + wordCount: content.split(" ").length, + pageContent: content, + token_count_estimate: tokenizeString(content).length, + }; + + item++; + writeToServerDocuments(data, `${slugify(filename)}-${data.id}-msg-${item}`); + } + + trashFile(fullFilePath); + console.log( + `[SUCCESS]: ${filename} messages converted & ready for embedding.\n` + ); + return { success: true, reason: null }; +} + +module.exports = asMbox; diff --git a/collector/processSingleFile/convert/asOfficeMime.js b/collector/processSingleFile/convert/asOfficeMime.js new file mode 100644 index 00000000..49a96b8b --- /dev/null +++ b/collector/processSingleFile/convert/asOfficeMime.js @@ -0,0 +1,46 @@ +const { v4 } = require("uuid"); +const officeParser = require("officeparser"); +const { + createdDate, + trashFile, + writeToServerDocuments, +} = require("../../utils/files"); +const { tokenizeString } = require("../../utils/tokenizer"); +const { default: slugify } = require("slugify"); + +async function asOfficeMime({ fullFilePath = "", filename = "" }) { + console.log(`-- Working ${filename} --`); + let content = ""; + try { + content = await officeParser.parseOfficeAsync(fullFilePath); + } catch (error) { + console.error(`Could not parse office or office-like file`, error); + } + + if (!content.length) { + console.error(`Resulting text content was empty for ${filename}.`); + trashFile(fullFilePath); + return { success: false, reason: `No text content found in ${filename}.` }; + } + + data = { + id: v4(), + url: "file://" + fullFilePath, + title: filename, + docAuthor: "no author found", + description: "No description found.", + docSource: "Office file uploaded by the user.", + chunkSource: filename, + published: createdDate(fullFilePath), + wordCount: content.split(" ").length, + pageContent: content, + token_count_estimate: tokenizeString(content).length, + }; + + writeToServerDocuments(data, `${slugify(filename)}-${data.id}`); + trashFile(fullFilePath); + console.log(`[SUCCESS]: ${filename} converted & ready for embedding.\n`); + return { success: true, reason: null }; +} + +module.exports = asOfficeMime; diff --git a/collector/processSingleFile/convert/asPDF.js b/collector/processSingleFile/convert/asPDF.js new file mode 100644 index 00000000..2509887e --- /dev/null +++ b/collector/processSingleFile/convert/asPDF.js @@ -0,0 +1,56 @@ +const { v4 } = require("uuid"); +const { PDFLoader } = require("langchain/document_loaders/fs/pdf"); +const { + createdDate, + trashFile, + writeToServerDocuments, +} = require("../../utils/files"); +const { tokenizeString } = require("../../utils/tokenizer"); +const { default: slugify } = require("slugify"); + +async function asPDF({ fullFilePath = "", filename = "" }) { + const pdfLoader = new PDFLoader(fullFilePath, { + splitPages: true, + }); + + console.log(`-- Working ${filename} --`); + const pageContent = []; + const docs = await pdfLoader.load(); + for (const doc of docs) { + console.log( + `-- Parsing content from pg ${ + doc.metadata?.loc?.pageNumber || "unknown" + } --` + ); + if (!doc.pageContent.length) continue; + pageContent.push(doc.pageContent); + } + + if (!pageContent.length) { + console.error(`Resulting text content was empty for ${filename}.`); + trashFile(fullFilePath); + return { success: false, reason: `No text content found in ${filename}.` }; + } + + const content = pageContent.join(""); + data = { + id: v4(), + url: "file://" + fullFilePath, + title: docs[0]?.metadata?.pdf?.info?.Title || filename, + docAuthor: docs[0]?.metadata?.pdf?.info?.Creator || "no author found", + description: "No description found.", + docSource: "pdf file uploaded by the user.", + chunkSource: filename, + published: createdDate(fullFilePath), + wordCount: content.split(" ").length, + pageContent: content, + token_count_estimate: tokenizeString(content).length, + }; + + writeToServerDocuments(data, `${slugify(filename)}-${data.id}`); + trashFile(fullFilePath); + console.log(`[SUCCESS]: ${filename} converted & ready for embedding.\n`); + return { success: true, reason: null }; +} + +module.exports = asPDF; diff --git a/collector/processSingleFile/convert/asTxt.js b/collector/processSingleFile/convert/asTxt.js new file mode 100644 index 00000000..229c175e --- /dev/null +++ b/collector/processSingleFile/convert/asTxt.js @@ -0,0 +1,46 @@ +const { v4 } = require("uuid"); +const fs = require("fs"); +const { tokenizeString } = require("../../utils/tokenizer"); +const { + createdDate, + trashFile, + writeToServerDocuments, +} = require("../../utils/files"); +const { default: slugify } = require("slugify"); + +async function asTxt({ fullFilePath = "", filename = "" }) { + let content = ""; + try { + content = fs.readFileSync(fullFilePath, "utf8"); + } catch (err) { + console.error("Could not read file!", err); + } + + if (!content?.length) { + console.error(`Resulting text content was empty for ${filename}.`); + trashFile(fullFilePath); + return { success: false, reason: `No text content found in ${filename}.` }; + } + + console.log(`-- Working ${filename} --`); + data = { + id: v4(), + url: "file://" + fullFilePath, + title: filename, + docAuthor: "Unknown", // TODO: Find a better author + description: "Unknown", // TODO: Find a better description + docSource: "a text file uploaded by the user.", + chunkSource: filename, + published: createdDate(fullFilePath), + wordCount: content.split(" ").length, + pageContent: content, + token_count_estimate: tokenizeString(content).length, + }; + + writeToServerDocuments(data, `${slugify(filename)}-${data.id}`); + trashFile(fullFilePath); + console.log(`[SUCCESS]: ${filename} converted & ready for embedding.\n`); + return { success: true, reason: null }; +} + +module.exports = asTxt; diff --git a/collector/processSingleFile/index.js b/collector/processSingleFile/index.js new file mode 100644 index 00000000..3884ca9b --- /dev/null +++ b/collector/processSingleFile/index.js @@ -0,0 +1,51 @@ +const path = require("path"); +const fs = require("fs"); +const { + WATCH_DIRECTORY, + SUPPORTED_FILETYPE_CONVERTERS, +} = require("../utils/constants"); +const { trashFile } = require("../utils/files"); + +RESERVED_FILES = ["__HOTDIR__.md"]; + +async function processSingleFile(targetFilename) { + const fullFilePath = path.resolve(WATCH_DIRECTORY, targetFilename); + if (RESERVED_FILES.includes(targetFilename)) + return { + success: false, + reason: "Filename is a reserved filename and cannot be processed.", + }; + if (!fs.existsSync(fullFilePath)) + return { + success: false, + reason: "File does not exist in upload directory.", + }; + + const fileExtension = path.extname(fullFilePath).toLowerCase(); + if (!fileExtension) { + return { + success: false, + reason: `No file extension found. This file cannot be processed.`, + }; + } + + if (!Object.keys(SUPPORTED_FILETYPE_CONVERTERS).includes(fileExtension)) { + trashFile(fullFilePath); + return { + success: false, + reason: `File extension ${fileExtension} not supported for parsing.`, + }; + } + + const FileTypeProcessor = require(SUPPORTED_FILETYPE_CONVERTERS[ + fileExtension + ]); + return await FileTypeProcessor({ + fullFilePath, + filename: targetFilename, + }); +} + +module.exports = { + processSingleFile, +}; diff --git a/collector/requirements.txt b/collector/requirements.txt deleted file mode 100644 index cf1137fb..00000000 --- a/collector/requirements.txt +++ /dev/null @@ -1,117 +0,0 @@ -about-time==4.2.1 -aiohttp==3.8.4 -aiosignal==1.3.1 -alive-progress==3.1.2 -anyio==3.7.0 -appdirs==1.4.4 -argilla==1.8.0 -asgiref==3.7.2 -async-timeout==4.0.2 -attrs==23.1.0 -backoff==2.2.1 -beautifulsoup4==4.12.2 -blinker==1.6.2 -bs4==0.0.1 -certifi==2023.5.7 -cffi==1.15.1 -chardet==5.1.0 -charset-normalizer==3.1.0 -click==8.1.3 -commonmark==0.9.1 -cryptography==41.0.1 -cssselect==1.2.0 -dataclasses-json==0.5.7 -Deprecated==1.2.14 -docx2txt==0.8 -et-xmlfile==1.1.0 -exceptiongroup==1.1.1 -fake-useragent==1.2.1 -Flask==2.3.2 -frozenlist==1.3.3 -grapheme==0.6.0 -greenlet==2.0.2 -gunicorn==20.1.0 -h11==0.14.0 -httpcore==0.16.3 -httpx==0.23.3 -idna==3.4 -importlib-metadata==6.6.0 -importlib-resources==5.12.0 -inquirerpy==0.3.4 -install==1.3.5 -itsdangerous==2.1.2 -Jinja2==3.1.2 -joblib==1.2.0 -langchain==0.0.189 -lxml==4.9.2 -Markdown==3.4.3 -MarkupSafe==2.1.3 -marshmallow==3.19.0 -marshmallow-enum==1.5.1 -monotonic==1.6 -msg-parser==1.2.0 -multidict==6.0.4 -mypy-extensions==1.0.0 -nltk==3.8.1 -numexpr==2.8.4 -numpy==1.23.5 -oauthlib==3.2.2 -olefile==0.46 -openapi-schema-pydantic==1.2.4 -openpyxl==3.1.2 -packaging==23.1 -pandas==1.5.3 -parse==1.19.0 -pdfminer.six==20221105 -pfzy==0.3.4 -Pillow==9.5.0 -prompt-toolkit==3.0.38 -pycparser==2.21 -pydantic==1.10.8 -pyee==8.2.2 -Pygments==2.15.1 -PyMuPDF==1.22.5 -pypandoc==1.4 -pyppeteer==1.0.2 -pyquery==2.0.0 -python-dateutil==2.8.2 -python-docx==0.8.11 -python-dotenv==0.21.1 -python-magic==0.4.27 -python-pptx==0.6.21 -python-slugify==8.0.1 -pytz==2023.3 -PyYAML==6.0 -regex==2023.5.5 -requests==2.31.0 -requests-html==0.10.0 -requests-oauthlib==1.3.1 -rfc3986==1.5.0 -rich==13.0.1 -six==1.16.0 -sniffio==1.3.0 -soupsieve==2.4.1 -SQLAlchemy==2.0.15 -tabulate==0.9.0 -tenacity==8.2.2 -text-unidecode==1.3 -tiktoken==0.4.0 -tqdm==4.65.0 -tweepy==4.14.0 -typer==0.9.0 -typing-inspect==0.9.0 -typing_extensions==4.6.3 -Unidecode==1.3.6 -unstructured==0.7.1 -urllib3==1.26.16 -uuid==1.30 -w3lib==2.1.1 -wcwidth==0.2.6 -websockets==10.4 -Werkzeug==2.3.6 -wrapt==1.14.1 -xlrd==2.0.1 -XlsxWriter==3.1.2 -yarl==1.9.2 -youtube-transcript-api==0.6.0 -zipp==3.15.0 diff --git a/collector/scripts/__init__.py b/collector/scripts/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/collector/scripts/gitbook.py b/collector/scripts/gitbook.py deleted file mode 100644 index 98625bf8..00000000 --- a/collector/scripts/gitbook.py +++ /dev/null @@ -1,44 +0,0 @@ -import os, json -from langchain.document_loaders import GitbookLoader -from urllib.parse import urlparse -from datetime import datetime -from alive_progress import alive_it -from .utils import tokenize -from uuid import uuid4 - -def gitbook(): - url = input("Enter the URL of the GitBook you want to collect: ") - if(url == ''): - print("Not a gitbook URL") - exit(1) - - primary_source = urlparse(url) - output_path = f"./outputs/gitbook-logs/{primary_source.netloc}" - transaction_output_dir = f"../server/storage/documents/gitbook-{primary_source.netloc}" - - if os.path.exists(output_path) == False:os.makedirs(output_path) - if os.path.exists(transaction_output_dir) == False: os.makedirs(transaction_output_dir) - loader = GitbookLoader(url, load_all_paths= primary_source.path in ['','/']) - for doc in alive_it(loader.load()): - metadata = doc.metadata - content = doc.page_content - source = urlparse(metadata.get('source')) - name = 'home' if source.path in ['','/'] else source.path.replace('/','_') - output_filename = f"doc-{name}.json" - transaction_output_filename = f"doc-{name}.json" - data = { - 'id': str(uuid4()), - 'url': metadata.get('source'), - 'title': metadata.get('title'), - 'description': metadata.get('title'), - 'published': datetime.today().strftime('%Y-%m-%d %H:%M:%S'), - 'wordCount': len(content), - 'pageContent': content, - 'token_count_estimate': len(tokenize(content)) - } - - with open(f"{output_path}/{output_filename}", 'w', encoding='utf-8') as file: - json.dump(data, file, ensure_ascii=True, indent=4) - - with open(f"{transaction_output_dir}/{transaction_output_filename}", 'w', encoding='utf-8') as file: - json.dump(data, file, ensure_ascii=True, indent=4) diff --git a/collector/scripts/link.py b/collector/scripts/link.py deleted file mode 100644 index aa5b32df..00000000 --- a/collector/scripts/link.py +++ /dev/null @@ -1,222 +0,0 @@ -import os, json, tempfile -from urllib.parse import urlparse -from requests_html import HTMLSession -from langchain.document_loaders import UnstructuredHTMLLoader -from .link_utils import append_meta, AsyncHTMLSessionFixed -from .utils import tokenize, ada_v2_cost -import requests -from bs4 import BeautifulSoup - -# Example Channel URL https://tim.blog/2022/08/09/nft-insider-trading-policy/ -def link(): - totalTokens = 0 - print("[NOTICE]: The first time running this process it will download supporting libraries.\n\n") - fqdn_link = input("Paste in the URL of an online article or blog: ") - if(len(fqdn_link) == 0): - print("Invalid URL!") - exit(1) - - session = HTMLSession() - req = session.get(fqdn_link) - if(req.ok == False): - print("Could not reach this url!") - exit(1) - - req.html.render() - full_text = None - with tempfile.NamedTemporaryFile(mode = "w") as tmp: - tmp.write(req.html.html) - tmp.seek(0) - loader = UnstructuredHTMLLoader(tmp.name) - data = loader.load()[0] - full_text = data.page_content - tmp.close() - - link = append_meta(req, full_text, True) - if(len(full_text) > 0): - totalTokens += len(tokenize(full_text)) - source = urlparse(req.url) - output_filename = f"website-{source.netloc}-{source.path.replace('/','_')}.json" - output_path = f"./outputs/website-logs" - - transaction_output_filename = f"website-{source.path.replace('/','_')}.json" - transaction_output_dir = f"../server/storage/documents/custom-documents" - - if os.path.isdir(output_path) == False: - os.makedirs(output_path) - - if os.path.isdir(transaction_output_dir) == False: - os.makedirs(transaction_output_dir) - - full_text = append_meta(req, full_text) - with open(f"{output_path}/{output_filename}", 'w', encoding='utf-8') as file: - json.dump(link, file, ensure_ascii=True, indent=4) - - with open(f"{transaction_output_dir}/{transaction_output_filename}", 'w', encoding='utf-8') as file: - json.dump(link, file, ensure_ascii=True, indent=4) - else: - print("Could not parse any meaningful data from this link or url.") - exit(1) - - print(f"\n\n[Success]: article or link content fetched!") - print(f"////////////////////////////") - print(f"Your estimated cost to embed this data using OpenAI's text-embedding-ada-002 model at $0.0004 / 1K tokens will cost {ada_v2_cost(totalTokens)} using {totalTokens} tokens.") - print(f"////////////////////////////") - exit(0) - -async def process_single_link(url): - session = None - try: - print(f"Working on {url}...") - session = AsyncHTMLSessionFixed() - req = await session.get(url) - await req.html.arender() - await session.close() - - if not req.ok: - return False, "Could not reach this URL." - - full_text = None - with tempfile.NamedTemporaryFile(mode = "w") as tmp: - tmp.write(req.html.html) - tmp.seek(0) - loader = UnstructuredHTMLLoader(tmp.name) - data = loader.load()[0] - full_text = data.page_content - tmp.close() - - if full_text: - link_meta = append_meta(req, full_text, True) - - source = urlparse(req.url) - transaction_output_dir = "../server/storage/documents/custom-documents" - transaction_output_filename = f"website-{source.netloc}-{source.path.replace('/', '_')}.json" - - if not os.path.isdir(transaction_output_dir): - os.makedirs(transaction_output_dir) - - file_path = os.path.join(transaction_output_dir, transaction_output_filename) - with open(file_path, 'w', encoding='utf-8') as file: - json.dump(link_meta, file, ensure_ascii=False, indent=4) - - - return True, "Content fetched and saved." - - else: - return False, "Could not parse any meaningful data from this URL." - - except Exception as e: - if session is not None: - session.close() # Kill hanging session. - return False, str(e) - -def crawler(): - prompt = "Paste in root URI of the pages of interest: " - new_link = input(prompt) - filter_value = input("Add a filter value for the url to ensure links don't wander too far. eg: 'my-domain.com': ") - #extract this from the uri provided - root_site = urlparse(new_link).scheme + "://" + urlparse(new_link).hostname - links = [] - urls = new_link - links.append(new_link) - grab = requests.get(urls) - soup = BeautifulSoup(grab.text, 'html.parser') - - # traverse paragraphs from soup - for link in soup.find_all("a"): - data = link.get('href') - if (data is not None): - fullpath = data if data[0] != '/' else f"{root_site}{data}" - try: - destination = urlparse(fullpath).scheme + "://" + urlparse(fullpath).hostname + (urlparse(fullpath).path if urlparse(fullpath).path is not None else '') - if filter_value in destination: - data = destination.strip() - print (data) - links.append(data) - else: - print (data + " does not apply for linking...") - except: - print (data + " does not apply for linking...") - #parse the links found - parse_links(links) - -def links(): - links = [] - prompt = "Paste in the URL of an online article or blog: " - done = False - - while(done == False): - new_link = input(prompt) - if(len(new_link) == 0): - done = True - links = [*set(links)] - continue - - links.append(new_link) - prompt = f"\n{len(links)} links in queue. Submit an empty value when done pasting in links to execute collection.\nPaste in the next URL of an online article or blog: " - - if(len(links) == 0): - print("No valid links provided!") - exit(1) - - parse_links(links) - - -# parse links from array -def parse_links(links): - totalTokens = 0 - for link in links: - print(f"Working on {link}...") - session = HTMLSession() - - req = session.get(link, timeout=20) - - if not req.ok: - print(f"Could not reach {link} - skipping!") - continue - - req.html.render(timeout=10) - - full_text = None - with tempfile.NamedTemporaryFile(mode="w") as tmp: - tmp.write(req.html.html) - tmp.seek(0) - loader = UnstructuredHTMLLoader(tmp.name) - data = loader.load()[0] - full_text = data.page_content - tmp.close() - - link = append_meta(req, full_text, True) - if len(full_text) > 0: - source = urlparse(req.url) - output_filename = f"website-{source.netloc}-{source.path.replace('/','_')}.json" - output_path = f"./outputs/website-logs" - - transaction_output_filename = f"website-{source.path.replace('/','_')}.json" - transaction_output_dir = f"../server/storage/documents/custom-documents" - - if not os.path.isdir(output_path): - os.makedirs(output_path) - - if not os.path.isdir(transaction_output_dir): - os.makedirs(transaction_output_dir) - - full_text = append_meta(req, full_text) - tokenCount = len(tokenize(full_text)) - totalTokens += tokenCount - - with open(f"{output_path}/{output_filename}", 'w', encoding='utf-8') as file: - json.dump(link, file, ensure_ascii=True, indent=4) - - with open(f"{transaction_output_dir}/{transaction_output_filename}", 'w', encoding='utf-8') as file: - json.dump(link, file, ensure_ascii=True, indent=4) - - req.session.close() - else: - print(f"Could not parse any meaningful data from {link}.") - continue - - print(f"\n\n[Success]: {len(links)} article or link contents fetched!") - print(f"////////////////////////////") - print(f"Your estimated cost to embed this data using OpenAI's text-embedding-ada-002 model at $0.0004 / 1K tokens will cost {ada_v2_cost(totalTokens)} using {totalTokens} tokens.") - print(f"////////////////////////////") \ No newline at end of file diff --git a/collector/scripts/link_utils.py b/collector/scripts/link_utils.py deleted file mode 100644 index 6afe05a0..00000000 --- a/collector/scripts/link_utils.py +++ /dev/null @@ -1,45 +0,0 @@ -import json, pyppeteer -from datetime import datetime -from .watch.utils import guid -from dotenv import load_dotenv -from .watch.utils import guid -from .utils import tokenize -from requests_html import AsyncHTMLSession - -load_dotenv() - -def normalize_url(url): - if(url.endswith('.web')): - return url - return f"{url}.web" - -def append_meta(request, text, metadata_only = False): - meta = { - 'id': guid(), - 'url': normalize_url(request.url), - 'title': request.html.find('title', first=True).text if len(request.html.find('title')) != 0 else '', - 'docAuthor': 'N/A', - 'description': request.html.find('meta[name="description"]', first=True).attrs.get('content') if request.html.find('meta[name="description"]', first=True) != None else '', - 'docSource': 'web page', - 'chunkSource': request.url, - 'published':request.html.find('meta[property="article:published_time"]', first=True).attrs.get('content') if request.html.find('meta[property="article:published_time"]', first=True) != None else datetime.today().strftime('%Y-%m-%d %H:%M:%S'), - 'wordCount': len(text.split(' ')), - 'pageContent': text, - 'token_count_estimate':len(tokenize(text)), - } - return "Article JSON Metadata:\n"+json.dumps(meta)+"\n\n\nText Content:\n" + text if metadata_only == False else meta - -class AsyncHTMLSessionFixed(AsyncHTMLSession): - """ - pip3 install websockets==6.0 --force-reinstall - """ - def __init__(self, **kwargs): - super(AsyncHTMLSessionFixed, self).__init__(**kwargs) - self.__browser_args = kwargs.get("browser_args", ["--no-sandbox"]) - - @property - async def browser(self): - if not hasattr(self, "_browser"): - self._browser = await pyppeteer.launch(ignoreHTTPSErrors=not(self.verify), headless=True, handleSIGINT=False, handleSIGTERM=False, handleSIGHUP=False, args=self.__browser_args) - - return self._browser \ No newline at end of file diff --git a/collector/scripts/medium.py b/collector/scripts/medium.py deleted file mode 100644 index e3228ce1..00000000 --- a/collector/scripts/medium.py +++ /dev/null @@ -1,71 +0,0 @@ -import os, json -from urllib.parse import urlparse -from .utils import tokenize, ada_v2_cost -from .medium_utils import get_username, fetch_recent_publications, append_meta -from alive_progress import alive_it - -# Example medium URL: https://medium.com/@yujiangtham or https://davidall.medium.com -def medium(): - print("[NOTICE]: This method will only get the 10 most recent publishings.") - author_url = input("Enter the medium URL of the author you want to collect: ") - if(author_url == ''): - print("Not a valid medium.com/@author URL") - exit(1) - - handle = get_username(author_url) - if(handle is None): - print("This does not appear to be a valid medium.com/@author URL") - exit(1) - - publications = fetch_recent_publications(handle) - if(len(publications)==0): - print("There are no public or free publications by this creator - nothing to collect.") - exit(1) - - totalTokenCount = 0 - transaction_output_dir = f"../server/storage/documents/medium-{handle}" - if os.path.isdir(transaction_output_dir) == False: - os.makedirs(transaction_output_dir) - - for publication in alive_it(publications): - pub_file_path = transaction_output_dir + f"/publication-{publication.get('id')}.json" - if os.path.exists(pub_file_path) == True: continue - - full_text = publication.get('pageContent') - if full_text is None or len(full_text) == 0: continue - - full_text = append_meta(publication, full_text) - item = { - 'id': publication.get('id'), - 'url': publication.get('url'), - 'title': publication.get('title'), - 'published': publication.get('published'), - 'wordCount': len(full_text.split(' ')), - 'pageContent': full_text, - } - - tokenCount = len(tokenize(full_text)) - item['token_count_estimate'] = tokenCount - - totalTokenCount += tokenCount - with open(pub_file_path, 'w', encoding='utf-8') as file: - json.dump(item, file, ensure_ascii=True, indent=4) - - print(f"[Success]: {len(publications)} scraped and fetched!") - print(f"\n\n////////////////////////////") - print(f"Your estimated cost to embed all of this data using OpenAI's text-embedding-ada-002 model at $0.0004 / 1K tokens will cost {ada_v2_cost(totalTokenCount)} using {totalTokenCount} tokens.") - print(f"////////////////////////////\n\n") - exit(0) - - - - - - - - - - - - - \ No newline at end of file diff --git a/collector/scripts/medium_utils.py b/collector/scripts/medium_utils.py deleted file mode 100644 index 37e6ab86..00000000 --- a/collector/scripts/medium_utils.py +++ /dev/null @@ -1,71 +0,0 @@ -import os, json, requests, re -from bs4 import BeautifulSoup - -def get_username(author_url): - if '@' in author_url: - pattern = r"medium\.com/@([\w-]+)" - match = re.search(pattern, author_url) - return match.group(1) if match else None - else: - # Given subdomain - pattern = r"([\w-]+).medium\.com" - match = re.search(pattern, author_url) - return match.group(1) if match else None - -def get_docid(medium_docpath): - pattern = r"medium\.com/p/([\w-]+)" - match = re.search(pattern, medium_docpath) - return match.group(1) if match else None - -def fetch_recent_publications(handle): - rss_link = f"https://medium.com/feed/@{handle}" - response = requests.get(rss_link) - if(response.ok == False): - print(f"Could not fetch RSS results for author.") - return [] - - xml = response.content - soup = BeautifulSoup(xml, 'xml') - items = soup.find_all('item') - publications = [] - - if os.path.isdir("./outputs/medium-logs") == False: - os.makedirs("./outputs/medium-logs") - - file_path = f"./outputs/medium-logs/medium-{handle}.json" - - if os.path.exists(file_path): - with open(file_path, "r") as file: - print(f"Returning cached data for Author {handle}. If you do not wish to use stored data then delete the file for this author to allow refetching.") - return json.load(file) - - for item in items: - tags = [] - for tag in item.find_all('category'): tags.append(tag.text) - content = BeautifulSoup(item.find('content:encoded').text, 'html.parser') - data = { - 'id': get_docid(item.find('guid').text), - 'title': item.find('title').text, - 'url': item.find('link').text.split('?')[0], - 'tags': ','.join(tags), - 'published': item.find('pubDate').text, - 'pageContent': content.get_text() - } - publications.append(data) - - with open(file_path, 'w+', encoding='utf-8') as json_file: - json.dump(publications, json_file, ensure_ascii=True, indent=2) - print(f"{len(publications)} articles found for author medium.com/@{handle}. Saved to medium-logs/medium-{handle}.json") - - return publications - -def append_meta(publication, text): - meta = { - 'url': publication.get('url'), - 'tags': publication.get('tags'), - 'title': publication.get('title'), - 'createdAt': publication.get('published'), - 'wordCount': len(text.split(' ')) - } - return "Article Metadata:\n"+json.dumps(meta)+"\n\nArticle Content:\n" + text - diff --git a/collector/scripts/sitemap.py b/collector/scripts/sitemap.py deleted file mode 100644 index f1c3b45c..00000000 --- a/collector/scripts/sitemap.py +++ /dev/null @@ -1,39 +0,0 @@ -import requests -import xml.etree.ElementTree as ET -from scripts.link import parse_links -import re - -def parse_sitemap(url): - response = requests.get(url) - root = ET.fromstring(response.content) - - urls = [] - for element in root.iter('{http://www.sitemaps.org/schemas/sitemap/0.9}url'): - for loc in element.iter('{http://www.sitemaps.org/schemas/sitemap/0.9}loc'): - if not has_extension_to_ignore(loc.text): - urls.append(loc.text) - else: - print(f"Skipping filetype: {loc.text}") - - return urls - -# Example sitemap URL https://www.nerdwallet.com/blog/wp-sitemap-news-articles-1.xml -def sitemap(): - sitemap_url = input("Enter the URL of the sitemap: ") - - if(len(sitemap_url) == 0): - print("No valid sitemap provided!") - exit(1) - - url_array = parse_sitemap(sitemap_url) - - #parse links from array - parse_links(url_array) - -def has_extension_to_ignore(string): - image_extensions = ['.jpg', '.jpeg', '.png', '.gif', '.bmp', '.pdf'] - - pattern = r'\b(' + '|'.join(re.escape(ext) for ext in image_extensions) + r')\b' - match = re.search(pattern, string, re.IGNORECASE) - - return match is not None \ No newline at end of file diff --git a/collector/scripts/substack.py b/collector/scripts/substack.py deleted file mode 100644 index 9a596584..00000000 --- a/collector/scripts/substack.py +++ /dev/null @@ -1,78 +0,0 @@ -import os, json -from urllib.parse import urlparse -from .utils import tokenize, ada_v2_cost -from .substack_utils import fetch_all_publications, only_valid_publications, get_content, append_meta -from alive_progress import alive_it - -# Example substack URL: https://swyx.substack.com/ -def substack(): - author_url = input("Enter the substack URL of the author you want to collect: ") - if(author_url == ''): - print("Not a valid author.substack.com URL") - exit(1) - - source = urlparse(author_url) - if('substack.com' not in source.netloc or len(source.netloc.split('.')) != 3): - print("This does not appear to be a valid author.substack.com URL") - exit(1) - - subdomain = source.netloc.split('.')[0] - publications = fetch_all_publications(subdomain) - valid_publications = only_valid_publications(publications) - - if(len(valid_publications)==0): - print("There are no public or free preview newsletters by this creator - nothing to collect.") - exit(1) - - print(f"{len(valid_publications)} of {len(publications)} publications are readable publically text posts - collecting those.") - - totalTokenCount = 0 - transaction_output_dir = f"../server/storage/documents/substack-{subdomain}" - if os.path.isdir(transaction_output_dir) == False: - os.makedirs(transaction_output_dir) - - for publication in alive_it(valid_publications): - pub_file_path = transaction_output_dir + f"/publication-{publication.get('id')}.json" - if os.path.exists(pub_file_path) == True: continue - - full_text = get_content(publication.get('canonical_url')) - if full_text is None or len(full_text) == 0: continue - - full_text = append_meta(publication, full_text) - item = { - 'id': publication.get('id'), - 'url': publication.get('canonical_url'), - 'thumbnail': publication.get('cover_image'), - 'title': publication.get('title'), - 'subtitle': publication.get('subtitle'), - 'description': publication.get('description'), - 'published': publication.get('post_date'), - 'wordCount': publication.get('wordcount'), - 'pageContent': full_text, - } - - tokenCount = len(tokenize(full_text)) - item['token_count_estimate'] = tokenCount - - totalTokenCount += tokenCount - with open(pub_file_path, 'w', encoding='utf-8') as file: - json.dump(item, file, ensure_ascii=True, indent=4) - - print(f"[Success]: {len(valid_publications)} scraped and fetched!") - print(f"\n\n////////////////////////////") - print(f"Your estimated cost to embed all of this data using OpenAI's text-embedding-ada-002 model at $0.0004 / 1K tokens will cost {ada_v2_cost(totalTokenCount)} using {totalTokenCount} tokens.") - print(f"////////////////////////////\n\n") - exit(0) - - - - - - - - - - - - - \ No newline at end of file diff --git a/collector/scripts/substack_utils.py b/collector/scripts/substack_utils.py deleted file mode 100644 index b6b5f083..00000000 --- a/collector/scripts/substack_utils.py +++ /dev/null @@ -1,88 +0,0 @@ -import os, json, requests, tempfile -from requests_html import HTMLSession -from langchain.document_loaders import UnstructuredHTMLLoader -from .watch.utils import guid - -def fetch_all_publications(subdomain): - file_path = f"./outputs/substack-logs/substack-{subdomain}.json" - - if os.path.isdir("./outputs/substack-logs") == False: - os.makedirs("./outputs/substack-logs") - - if os.path.exists(file_path): - with open(file_path, "r") as file: - print(f"Returning cached data for substack {subdomain}.substack.com. If you do not wish to use stored data then delete the file for this newsletter to allow refetching.") - return json.load(file) - - collecting = True - offset = 0 - publications = [] - - while collecting is True: - url = f"https://{subdomain}.substack.com/api/v1/archive?sort=new&offset={offset}" - response = requests.get(url) - if(response.ok == False): - print("Bad response - exiting collection") - collecting = False - continue - - data = response.json() - - if(len(data) ==0 ): - collecting = False - continue - - for publication in data: - publications.append(publication) - offset = len(publications) - - with open(file_path, 'w+', encoding='utf-8') as json_file: - json.dump(publications, json_file, ensure_ascii=True, indent=2) - print(f"{len(publications)} publications found for author {subdomain}.substack.com. Saved to substack-logs/channel-{subdomain}.json") - - return publications - -def only_valid_publications(publications= []): - valid_publications = [] - for publication in publications: - is_paid = publication.get('audience') != 'everyone' - if (is_paid and publication.get('should_send_free_preview') != True) or publication.get('type') != 'newsletter': continue - valid_publications.append(publication) - return valid_publications - -def get_content(article_link): - print(f"Fetching {article_link}") - if(len(article_link) == 0): - print("Invalid URL!") - return None - - session = HTMLSession() - req = session.get(article_link) - if(req.ok == False): - print("Could not reach this url!") - return None - - req.html.render() - - full_text = None - with tempfile.NamedTemporaryFile(mode = "w") as tmp: - tmp.write(req.html.html) - tmp.seek(0) - loader = UnstructuredHTMLLoader(tmp.name) - data = loader.load()[0] - full_text = data.page_content - tmp.close() - return full_text - -def append_meta(publication, text): - meta = { - 'id': guid(), - 'url': publication.get('canonical_url'), - 'thumbnail': publication.get('cover_image'), - 'title': publication.get('title'), - 'subtitle': publication.get('subtitle'), - 'description': publication.get('description'), - 'createdAt': publication.get('post_date'), - 'wordCount': publication.get('wordcount') - } - return "Newsletter Metadata:\n"+json.dumps(meta)+"\n\nArticle Content:\n" + text \ No newline at end of file diff --git a/collector/scripts/twitter.py b/collector/scripts/twitter.py deleted file mode 100644 index 4c085071..00000000 --- a/collector/scripts/twitter.py +++ /dev/null @@ -1,103 +0,0 @@ -""" -Tweepy implementation of twitter reader. Requires the 4 twitter keys to operate. -""" - -import tweepy -import os, time -import pandas as pd -import json -from .utils import tokenize, ada_v2_cost -from .watch.utils import guid - -def twitter(): - #get user and number of tweets to read - username = input("user timeline to read from (blank to ignore): ") - searchQuery = input("Search term, or leave blank to get user tweets (blank to ignore): ") - tweetCount = input("Gather the last number of tweets: ") - - # Read your API keys to call the API. - consumer_key = os.environ.get("TW_CONSUMER_KEY") - consumer_secret = os.environ.get("TW_CONSUMER_SECRET") - access_token = os.environ.get("TW_ACCESS_TOKEN") - access_token_secret = os.environ.get("TW_ACCESS_TOKEN_SECRET") - - # Check if any of the required environment variables is missing. - if not consumer_key or not consumer_secret or not access_token or not access_token_secret: - raise EnvironmentError("One of the twitter API environment variables are missing.") - - # Pass in our twitter API authentication key - auth = tweepy.OAuth1UserHandler( - consumer_key, consumer_secret, access_token, access_token_secret - ) - - # Instantiate the tweepy API - api = tweepy.API(auth, wait_on_rate_limit=True) - - try: - if (searchQuery == ''): - tweets = api.user_timeline(screen_name=username, tweet_mode = 'extended', count=tweetCount) - else: - tweets = api.search_tweets(q=searchQuery, tweet_mode = 'extended', count=tweetCount) - - # Pulling Some attributes from the tweet - attributes_container = [ - [tweet.id, tweet.user.screen_name, tweet.created_at, tweet.favorite_count, tweet.source, tweet.full_text] - for tweet in tweets - ] - - # Creation of column list to rename the columns in the dataframe - columns = ["id", "Screen Name", "Date Created", "Number of Likes", "Source of Tweet", "Tweet"] - - # Creation of Dataframe - tweets_df = pd.DataFrame(attributes_container, columns=columns) - - totalTokens = 0 - for index, row in tweets_df.iterrows(): - meta_link = twitter_meta(row, True) - output_filename = f"twitter-{username}-{row['Date Created']}.json" - output_path = f"./outputs/twitter-logs" - - transaction_output_filename = f"tweet-{username}-{row['id']}.json" - transaction_output_dir = f"../server/storage/documents/twitter-{username}" - - if not os.path.isdir(output_path): - os.makedirs(output_path) - - if not os.path.isdir(transaction_output_dir): - os.makedirs(transaction_output_dir) - - full_text = twitter_meta(row) - tokenCount = len(tokenize(full_text)) - meta_link['pageContent'] = full_text - meta_link['token_count_estimate'] = tokenCount - totalTokens += tokenCount - - with open(f"{output_path}/{output_filename}", 'w', encoding='utf-8') as file: - json.dump(meta_link, file, ensure_ascii=True, indent=4) - - with open(f"{transaction_output_dir}/{transaction_output_filename}", 'w', encoding='utf-8') as file: - json.dump(meta_link, file, ensure_ascii=True, indent=4) - - # print(f"{transaction_output_dir}/{transaction_output_filename}") - - print(f"{tokenCount} tokens written over {tweets_df.shape[0]} records.") - - except BaseException as e: - print("Status Failed: ", str(e)) - time.sleep(3) - - -def twitter_meta(row, metadata_only = False): - # Note that /anyuser is a known twitter hack for not knowing the user's handle - # https://stackoverflow.com/questions/897107/can-i-fetch-the-tweet-from-twitter-if-i-know-the-tweets-id - url = f"http://twitter.com/anyuser/status/{row['id']}" - title = f"Tweet {row['id']}" - meta = { - 'id': guid(), - 'url': url, - 'title': title, - 'description': 'Tweet from ' + row["Screen Name"], - 'published': row["Date Created"].strftime('%Y-%m-%d %H:%M:%S'), - 'wordCount': len(row["Tweet"]), - } - return "Tweet JSON Metadata:\n"+json.dumps(meta)+"\n\n\nText Content:\n" + row["Tweet"] if metadata_only == False else meta diff --git a/collector/scripts/utils.py b/collector/scripts/utils.py deleted file mode 100644 index 288da0f2..00000000 --- a/collector/scripts/utils.py +++ /dev/null @@ -1,10 +0,0 @@ -import tiktoken -encoder = tiktoken.encoding_for_model("text-embedding-ada-002") - -def tokenize(fullText): - return encoder.encode(fullText) - -def ada_v2_cost(tokenCount): - rate_per = 0.0004 / 1_000 # $0.0004 / 1K tokens - total = tokenCount * rate_per - return '${:,.2f}'.format(total) if total >= 0.01 else '< $0.01' \ No newline at end of file diff --git a/collector/scripts/watch/__init__.py b/collector/scripts/watch/__init__.py deleted file mode 100644 index e69de29b..00000000 diff --git a/collector/scripts/watch/convert/as_docx.py b/collector/scripts/watch/convert/as_docx.py deleted file mode 100644 index b3778617..00000000 --- a/collector/scripts/watch/convert/as_docx.py +++ /dev/null @@ -1,78 +0,0 @@ -import os -from langchain.document_loaders import Docx2txtLoader, UnstructuredODTLoader -from slugify import slugify -from ..utils import guid, file_creation_time, write_to_server_documents, move_source -from ...utils import tokenize - -# Process all text-related documents. -def as_docx(**kwargs): - parent_dir = kwargs.get('directory', 'hotdir') - filename = kwargs.get('filename') - ext = kwargs.get('ext', '.txt') - remove = kwargs.get('remove_on_complete', False) - fullpath = f"{parent_dir}/{filename}{ext}" - - loader = Docx2txtLoader(fullpath) - data = loader.load()[0] - content = data.page_content - - if len(content) == 0: - print(f"Resulting text content was empty for {filename}{ext}.") - return(False, f"No text content found in {filename}{ext}") - - print(f"-- Working {fullpath} --") - data = { - 'id': guid(), - 'url': "file://"+os.path.abspath(f"{parent_dir}/processed/{filename}{ext}"), - 'title': f"{filename}{ext}", - 'docAuthor': 'Unknown', # TODO: Find a better author - 'description': 'Unknown', # TODO: Find a better bescription - 'docSource': 'Docx Text file uploaded by the user.', - 'chunkSource': f"{filename}{ext}", - 'published': file_creation_time(fullpath), - 'wordCount': len(content), - 'pageContent': content, - 'token_count_estimate': len(tokenize(content)) - } - - write_to_server_documents(data, f"{slugify(filename)}-{data.get('id')}") - move_source(parent_dir, f"{filename}{ext}", remove=remove) - - print(f"[SUCCESS]: {filename}{ext} converted & ready for embedding.\n") - return(True, None) - -def as_odt(**kwargs): - parent_dir = kwargs.get('directory', 'hotdir') - filename = kwargs.get('filename') - ext = kwargs.get('ext', '.txt') - remove = kwargs.get('remove_on_complete', False) - fullpath = f"{parent_dir}/{filename}{ext}" - - loader = UnstructuredODTLoader(fullpath) - data = loader.load()[0] - content = data.page_content - - if len(content) == 0: - print(f"Resulting text content was empty for {filename}{ext}.") - return(False, f"No text content found in {filename}{ext}") - - print(f"-- Working {fullpath} --") - data = { - 'id': guid(), - 'url': "file://"+os.path.abspath(f"{parent_dir}/processed/{filename}{ext}"), - 'title': f"{filename}{ext}", - 'docAuthor': 'Unknown', # TODO: Find a better author - 'description': 'Unknown', # TODO: Find a better bescription - 'docSource': 'ODT Text file uploaded by the user.', - 'chunkSource': f"{filename}{ext}", - 'published': file_creation_time(fullpath), - 'wordCount': len(content), - 'pageContent': content, - 'token_count_estimate': len(tokenize(content)) - } - - write_to_server_documents(data, f"{slugify(filename)}-{data.get('id')}") - move_source(parent_dir, f"{filename}{ext}", remove=remove) - - print(f"[SUCCESS]: {filename}{ext} converted & ready for embedding.\n") - return(True, None) \ No newline at end of file diff --git a/collector/scripts/watch/convert/as_html.py b/collector/scripts/watch/convert/as_html.py deleted file mode 100644 index a70ddb56..00000000 --- a/collector/scripts/watch/convert/as_html.py +++ /dev/null @@ -1,42 +0,0 @@ -import os, re -from slugify import slugify -from langchain.document_loaders import BSHTMLLoader -from ..utils import guid, file_creation_time, write_to_server_documents, move_source -from ...utils import tokenize - -# Process all html-related documents. -def as_html(**kwargs): - parent_dir = kwargs.get('directory', 'hotdir') - filename = kwargs.get('filename') - ext = kwargs.get('ext', '.html') - remove = kwargs.get('remove_on_complete', False) - fullpath = f"{parent_dir}/{filename}{ext}" - - loader = BSHTMLLoader(fullpath) - document = loader.load()[0] - content = re.sub(r"\n+", "\n", document.page_content) - - if len(content) == 0: - print(f"Resulting text content was empty for {filename}{ext}.") - return(False, f"No text content found in {filename}{ext}") - - print(f"-- Working {fullpath} --") - data = { - 'id': guid(), - 'url': "file://"+os.path.abspath(f"{parent_dir}/processed/{filename}{ext}"), - 'title': document.metadata.get('title', f"{filename}{ext}"), - 'docAuthor': 'Unknown', # TODO: Find a better author - 'description': 'Unknown', # TODO: Find a better description - 'docSource': 'an HTML file uploaded by the user.', - 'chunkSource': f"{filename}{ext}", - 'published': file_creation_time(fullpath), - 'wordCount': len(content), - 'pageContent': content, - 'token_count_estimate': len(tokenize(content)) - } - - write_to_server_documents(data, f"{slugify(filename)}-{data.get('id')}") - move_source(parent_dir, f"{filename}{ext}", remove=remove) - - print(f"[SUCCESS]: {filename}{ext} converted & ready for embedding.\n") - return(True, None) diff --git a/collector/scripts/watch/convert/as_markdown.py b/collector/scripts/watch/convert/as_markdown.py deleted file mode 100644 index b7d9fbf6..00000000 --- a/collector/scripts/watch/convert/as_markdown.py +++ /dev/null @@ -1,42 +0,0 @@ -import os -from langchain.document_loaders import UnstructuredMarkdownLoader -from slugify import slugify -from ..utils import guid, file_creation_time, write_to_server_documents, move_source -from ...utils import tokenize - -# Process all text-related documents. -def as_markdown(**kwargs): - parent_dir = kwargs.get('directory', 'hotdir') - filename = kwargs.get('filename') - ext = kwargs.get('ext', '.txt') - remove = kwargs.get('remove_on_complete', False) - fullpath = f"{parent_dir}/{filename}{ext}" - - loader = UnstructuredMarkdownLoader(fullpath) - data = loader.load()[0] - content = data.page_content - - if len(content) == 0: - print(f"Resulting page content was empty - no text could be extracted from {filename}{ext}.") - return(False, f"No text could be extracted from {filename}{ext}.") - - print(f"-- Working {fullpath} --") - data = { - 'id': guid(), - 'url': "file://"+os.path.abspath(f"{parent_dir}/processed/{filename}{ext}"), - 'title': f"{filename}", # TODO: find a better metadata - 'docAuthor': 'Unknown', # TODO: find a better metadata - 'description': 'Unknown', # TODO: find a better metadata - 'docSource': 'markdown file uploaded by the user.', - 'chunkSource': f"{filename}{ext}", - 'published': file_creation_time(fullpath), - 'wordCount': len(content), - 'pageContent': content, - 'token_count_estimate': len(tokenize(content)) - } - - write_to_server_documents(data, f"{slugify(filename)}-{data.get('id')}") - move_source(parent_dir, f"{filename}{ext}", remove=remove) - - print(f"[SUCCESS]: {filename}{ext} converted & ready for embedding.\n") - return(True, None) diff --git a/collector/scripts/watch/convert/as_mbox.py b/collector/scripts/watch/convert/as_mbox.py deleted file mode 100644 index 2d7c08e6..00000000 --- a/collector/scripts/watch/convert/as_mbox.py +++ /dev/null @@ -1,124 +0,0 @@ -import os -import datetime -import email.utils -import re -import quopri -import base64 -from mailbox import mbox, mboxMessage -from slugify import slugify -from bs4 import BeautifulSoup -from scripts.watch.utils import ( - guid, - file_creation_time, - write_to_server_documents, - move_source, -) -from scripts.utils import tokenize - - -def get_content(message: mboxMessage) -> str: - content = "None" - # if message.is_multipart(): - for part in message.walk(): - if part.get_content_type() == "text/plain": - content = part.get_payload(decode=True) - break - elif part.get_content_type() == "text/html": - soup = BeautifulSoup(part.get_payload(decode=True), "html.parser") - content = soup.get_text() - - if isinstance(content, bytes): - try: - content = content.decode("utf-8") - except UnicodeDecodeError: - content = content.decode("latin-1") - - return content - - -def parse_subject(subject: str) -> str: - # Check if subject is Quoted-Printable encoded - if subject.startswith("=?") and subject.endswith("?="): - # Extract character set and encoding information - match = re.match(r"=\?(.+)\?(.)\?(.+)\?=", subject) - if match: - charset = match.group(1) - encoding = match.group(2) - encoded_text = match.group(3) - is_quoted_printable = encoding.upper() == "Q" - is_base64 = encoding.upper() == "B" - if is_quoted_printable: - # Decode Quoted-Printable encoded text - subject = quopri.decodestring(encoded_text).decode(charset) - elif is_base64: - # Decode Base64 encoded text - subject = base64.b64decode(encoded_text).decode(charset) - - return subject - - -# Process all mbox-related documents. -def as_mbox(**kwargs): - parent_dir = kwargs.get("directory", "hotdir") - filename = kwargs.get("filename") - ext = kwargs.get("ext", ".mbox") - remove = kwargs.get("remove_on_complete", False) - - if filename is not None: - filename = str(filename) - else: - print("[ERROR]: No filename provided.") - return (False, "No filename provided.") - - fullpath = f"{parent_dir}/{filename}{ext}" - - print(f"-- Working {fullpath} --") - box = mbox(fullpath) - - for message in box: - content = get_content(message) - content = content.strip().replace("\r\n", "\n") - - if len(content) == 0: - print("[WARNING]: Mail with no content. Ignored.") - continue - - date_tuple = email.utils.parsedate_tz(message["Date"]) - if date_tuple: - local_date = datetime.datetime.fromtimestamp( - email.utils.mktime_tz(date_tuple) - ) - date_sent = local_date.strftime("%a, %d %b %Y %H:%M:%S") - else: - date_sent = None - - subject = message["Subject"] - - if subject is None: - print("[WARNING]: Mail with no subject. But has content.") - subject = "None" - else: - subject = parse_subject(subject) - - abs_path = os.path.abspath( - f"{parent_dir}/processed/{slugify(filename)}-{guid()}{ext}" - ) - data = { - "id": guid(), - "url": f"file://{abs_path}", - "title": subject, - "docAuthor": message["From"], - "description": f"email from {message['From']} to {message['To']}", - "docSource": "mbox file uploaded by the user.", - "chunkSource": subject, - "published": file_creation_time(fullpath), - "wordCount": len(content), - "pageContent": content, - "token_count_estimate": len(tokenize(content)), - } - - write_to_server_documents(data, f"{slugify(filename)}-{data.get('id')}") - - move_source(parent_dir, f"{filename}{ext}", remove=remove) - print(f"[SUCCESS]: {filename}{ext} converted & ready for embedding.\n") - return (True, None) diff --git a/collector/scripts/watch/convert/as_pdf.py b/collector/scripts/watch/convert/as_pdf.py deleted file mode 100644 index 7199d6c5..00000000 --- a/collector/scripts/watch/convert/as_pdf.py +++ /dev/null @@ -1,58 +0,0 @@ -import os, fitz -from langchain.document_loaders import PyMuPDFLoader # better UTF support and metadata -from slugify import slugify -from ..utils import guid, file_creation_time, write_to_server_documents, move_source -from ...utils import tokenize - -# Process all PDF-related documents. -def as_pdf(**kwargs): - parent_dir = kwargs.get('directory', 'hotdir') - filename = kwargs.get('filename') - ext = kwargs.get('ext', '.txt') - remove = kwargs.get('remove_on_complete', False) - fullpath = f"{parent_dir}/{filename}{ext}" - - print(f"-- Working {fullpath} --") - loader = PyMuPDFLoader(fullpath) - pages = loader.load() - - if len(pages) == 0: - print(f"{fullpath} parsing resulted in no pages - nothing to do.") - return(False, f"No pages found for {filename}{ext}!") - - # Set doc to the first page so we can still get the metadata from PyMuPDF but without all the unicode issues. - doc = pages[0] - del loader - del pages - - page_content = '' - for page in fitz.open(fullpath): - print(f"-- Parsing content from pg {page.number} --") - page_content += str(page.get_text('text')) - - if len(page_content) == 0: - print(f"Resulting page content was empty - no text could be extracted from the document.") - return(False, f"No text content could be extracted from {filename}{ext}!") - - title = doc.metadata.get('title') - author = doc.metadata.get('author') - subject = doc.metadata.get('subject') - data = { - 'id': guid(), - 'url': "file://"+os.path.abspath(f"{parent_dir}/processed/{filename}{ext}"), - 'title': title if title else f"{filename}{ext}", - 'docAuthor': author if author else 'No author found', - 'description': subject if subject else 'No description found.', - 'docSource': 'pdf file uploaded by the user.', - 'chunkSource': f"{filename}{ext}", - 'published': file_creation_time(fullpath), - 'wordCount': len(page_content), # Technically a letter count :p - 'pageContent': page_content, - 'token_count_estimate': len(tokenize(page_content)) - } - - write_to_server_documents(data, f"{slugify(filename)}-{data.get('id')}") - move_source(parent_dir, f"{filename}{ext}", remove=remove) - - print(f"[SUCCESS]: {filename}{ext} converted & ready for embedding.\n") - return(True, None) diff --git a/collector/scripts/watch/convert/as_text.py b/collector/scripts/watch/convert/as_text.py deleted file mode 100644 index 1b897874..00000000 --- a/collector/scripts/watch/convert/as_text.py +++ /dev/null @@ -1,38 +0,0 @@ -import os -from slugify import slugify -from ..utils import guid, file_creation_time, write_to_server_documents, move_source -from ...utils import tokenize - -# Process all text-related documents. -def as_text(**kwargs): - parent_dir = kwargs.get('directory', 'hotdir') - filename = kwargs.get('filename') - ext = kwargs.get('ext', '.txt') - remove = kwargs.get('remove_on_complete', False) - fullpath = f"{parent_dir}/{filename}{ext}" - content = open(fullpath).read() - - if len(content) == 0: - print(f"Resulting text content was empty for {filename}{ext}.") - return(False, f"No text content found in {filename}{ext}") - - print(f"-- Working {fullpath} --") - data = { - 'id': guid(), - 'url': "file://"+os.path.abspath(f"{parent_dir}/processed/{filename}{ext}"), - 'title': f"{filename}{ext}", - 'docAuthor': 'Unknown', # TODO: Find a better author - 'description': 'Unknown', # TODO: Find a better description - 'docSource': 'a text file uploaded by the user.', - 'chunkSource': f"{filename}{ext}", - 'published': file_creation_time(fullpath), - 'wordCount': len(content), - 'pageContent': content, - 'token_count_estimate': len(tokenize(content)) - } - - write_to_server_documents(data, f"{slugify(filename)}-{data.get('id')}") - move_source(parent_dir, f"{filename}{ext}", remove=remove) - - print(f"[SUCCESS]: {filename}{ext} converted & ready for embedding.\n") - return(True, None) diff --git a/collector/scripts/watch/filetypes.py b/collector/scripts/watch/filetypes.py deleted file mode 100644 index ebc11a6e..00000000 --- a/collector/scripts/watch/filetypes.py +++ /dev/null @@ -1,25 +0,0 @@ -from .convert.as_text import as_text -from .convert.as_markdown import as_markdown -from .convert.as_pdf import as_pdf -from .convert.as_docx import as_docx, as_odt -from .convert.as_mbox import as_mbox -from .convert.as_html import as_html - -FILETYPES = { - '.txt': as_text, - '.md': as_markdown, - '.pdf': as_pdf, - '.docx': as_docx, - '.odt': as_odt, - '.mbox': as_mbox, - '.html': as_html, -} - -ACCEPTED_MIMES = { - 'text/plain': ['.txt', '.md'], - 'text/html': ['.html'], - 'application/vnd.openxmlformats-officedocument.wordprocessingml.document': ['.docx'], - 'application/vnd.oasis.opendocument.text': ['.odt'], - 'application/pdf': ['.pdf'], - 'application/mbox': ['.mbox'], -} diff --git a/collector/scripts/watch/main.py b/collector/scripts/watch/main.py deleted file mode 100644 index 152e13bc..00000000 --- a/collector/scripts/watch/main.py +++ /dev/null @@ -1,22 +0,0 @@ -import os -from .filetypes import FILETYPES -from .utils import move_source - -RESERVED = ['__HOTDIR__.md'] -def watch_for_changes(directory): - for raw_doc in os.listdir(directory): - if os.path.isdir(f"{directory}/{raw_doc}") or raw_doc in RESERVED: continue - - filename, fileext = os.path.splitext(raw_doc) - if filename in ['.DS_Store'] or fileext == '': continue - - if fileext not in FILETYPES.keys(): - print(f"{fileext} not a supported file type for conversion. Removing from hot directory.") - move_source(new_destination_filename=raw_doc, failed=True) - continue - - FILETYPES[fileext]( - directory=directory, - filename=filename, - ext=fileext, - ) \ No newline at end of file diff --git a/collector/scripts/watch/process_single.py b/collector/scripts/watch/process_single.py deleted file mode 100644 index 19fc3978..00000000 --- a/collector/scripts/watch/process_single.py +++ /dev/null @@ -1,35 +0,0 @@ -import os -from .filetypes import FILETYPES -from .utils import move_source - -RESERVED = ['__HOTDIR__.md'] - -# This script will do a one-off processing of a specific document that exists in hotdir. -# For this function we remove the original source document since there is no need to keep it and it will -# only occupy additional disk space. -def process_single(directory, target_doc): - if os.path.isdir(f"{directory}/{target_doc}") or target_doc in RESERVED: return (False, "Not a file") - - if os.path.exists(f"{directory}/{target_doc}") is False: - print(f"{directory}/{target_doc} does not exist.") - return (False, f"{directory}/{target_doc} does not exist.") - - filename, fileext = os.path.splitext(target_doc) - if filename in ['.DS_Store'] or fileext == '': return False - if fileext == '.lock': - print(f"{filename} is locked - skipping until unlocked") - return (False, f"{filename} is locked - skipping until unlocked") - - if fileext not in FILETYPES.keys(): - print(f"{fileext} not a supported file type for conversion. It will not be processed.") - move_source(new_destination_filename=target_doc, failed=True, remove=True) - return (False, f"{fileext} not a supported file type for conversion. It will not be processed.") - - # Returns Tuple of (Boolean, String|None) of success status and possible error message. - # Error message will display to user. - return FILETYPES[fileext]( - directory=directory, - filename=filename, - ext=fileext, - remove_on_complete=True # remove source document to save disk space. - ) \ No newline at end of file diff --git a/collector/scripts/watch/utils.py b/collector/scripts/watch/utils.py deleted file mode 100644 index f61e3943..00000000 --- a/collector/scripts/watch/utils.py +++ /dev/null @@ -1,35 +0,0 @@ -import os, json -from datetime import datetime -from uuid import uuid4 - -def guid(): - return str(uuid4()) - -def file_creation_time(path_to_file): - try: - if os.name == 'nt': - return datetime.fromtimestamp(os.path.getctime(path_to_file)).strftime('%Y-%m-%d %H:%M:%S') - else: - stat = os.stat(path_to_file) - return datetime.fromtimestamp(stat.st_birthtime).strftime('%Y-%m-%d %H:%M:%S') - except AttributeError: - return datetime.today().strftime('%Y-%m-%d %H:%M:%S') - -def move_source(working_dir='hotdir', new_destination_filename='', failed=False, remove=False): - if remove and os.path.exists(f"{working_dir}/{new_destination_filename}"): - print(f"{new_destination_filename} deleted from filesystem") - os.remove(f"{working_dir}/{new_destination_filename}") - return - - destination = f"{working_dir}/processed" if not failed else f"{working_dir}/failed" - if os.path.exists(destination) == False: - os.mkdir(destination) - - os.replace(f"{working_dir}/{new_destination_filename}", f"{destination}/{new_destination_filename}") - return - -def write_to_server_documents(data, filename, override_destination = None): - destination = f"../server/storage/documents/custom-documents" if override_destination == None else override_destination - if os.path.exists(destination) == False: os.makedirs(destination) - with open(f"{destination}/{filename}.json", 'w', encoding='utf-8') as file: - json.dump(data, file, ensure_ascii=True, indent=4) diff --git a/collector/scripts/youtube.py b/collector/scripts/youtube.py deleted file mode 100644 index aea76e0f..00000000 --- a/collector/scripts/youtube.py +++ /dev/null @@ -1,55 +0,0 @@ -import os, json -from youtube_transcript_api import YouTubeTranscriptApi -from youtube_transcript_api.formatters import TextFormatter, JSONFormatter -from .utils import tokenize, ada_v2_cost -from .yt_utils import fetch_channel_video_information, get_channel_id, clean_text, append_meta, get_duration -from alive_progress import alive_it - -# Example Channel URL https://www.youtube.com/channel/UCmWbhBB96ynOZuWG7LfKong -# Example Channel URL https://www.youtube.com/@mintplex - -def youtube(): - channel_link = input("Paste in the URL of a YouTube channel: ") - channel_id = get_channel_id(channel_link) - - if channel_id == None or len(channel_id) == 0: - print("Invalid input - must be full YouTube channel URL") - exit(1) - - channel_data = fetch_channel_video_information(channel_id) - transaction_output_dir = f"../server/storage/documents/youtube-{channel_data.get('channelTitle')}" - - if os.path.isdir(transaction_output_dir) == False: - os.makedirs(transaction_output_dir) - - print(f"\nFetching transcripts for {len(channel_data.get('items'))} videos - please wait.\nStopping and restarting will not refetch known transcripts in case there is an error.\nSaving results to: {transaction_output_dir}.") - totalTokenCount = 0 - for video in alive_it(channel_data.get('items')): - video_file_path = transaction_output_dir + f"/video-{video.get('id')}.json" - if os.path.exists(video_file_path) == True: - continue - - formatter = TextFormatter() - json_formatter = JSONFormatter() - try: - transcript = YouTubeTranscriptApi.get_transcript(video.get('id')) - raw_text = clean_text(formatter.format_transcript(transcript)) - duration = get_duration(json_formatter.format_transcript(transcript)) - - if(len(raw_text) > 0): - fullText = append_meta(video, duration, raw_text) - tokenCount = len(tokenize(fullText)) - video['pageContent'] = fullText - video['token_count_estimate'] = tokenCount - totalTokenCount += tokenCount - with open(video_file_path, 'w', encoding='utf-8') as file: - json.dump(video, file, ensure_ascii=True, indent=4) - except: - print("There was an issue getting the transcription of a video in the list - likely because captions are disabled. Skipping") - continue - - print(f"[Success]: {len(channel_data.get('items'))} video transcripts fetched!") - print(f"\n\n////////////////////////////") - print(f"Your estimated cost to embed all of this data using OpenAI's text-embedding-ada-002 model at $0.0004 / 1K tokens will cost {ada_v2_cost(totalTokenCount)} using {totalTokenCount} tokens.") - print(f"////////////////////////////\n\n") - exit(0) diff --git a/collector/scripts/yt_utils.py b/collector/scripts/yt_utils.py deleted file mode 100644 index c13fb26e..00000000 --- a/collector/scripts/yt_utils.py +++ /dev/null @@ -1,122 +0,0 @@ -import json, requests, os, re -from slugify import slugify -from dotenv import load_dotenv -from .watch.utils import guid -load_dotenv() - -def is_yt_short(videoId): - url = 'https://www.youtube.com/shorts/' + videoId - ret = requests.head(url) - return ret.status_code == 200 - -def get_channel_id(channel_link): - if('@' in channel_link): - pattern = r'https?://www\.youtube\.com/(@\w+)/?' - match = re.match(pattern, channel_link) - if match is False: return None - handle = match.group(1) - print('Need to map username to channelId - this can take a while sometimes.') - response = requests.get(f"https://yt.lemnoslife.com/channels?handle={handle}", timeout=20) - - if(response.ok == False): - print("Handle => ChannelId mapping endpoint is too slow - use regular youtube.com/channel URL") - return None - - json_data = response.json() - return json_data.get('items')[0].get('id') - else: - pattern = r"youtube\.com/channel/([\w-]+)" - match = re.search(pattern, channel_link) - return match.group(1) if match else None - - -def clean_text(text): - return re.sub(r"\[.*?\]", "", text) - -def append_meta(video, duration, text): - meta = { - 'id': guid(), - 'youtubeURL': f"https://youtube.com/watch?v={video.get('id')}", - 'thumbnail': video.get('thumbnail'), - 'description': video.get('description'), - 'createdAt': video.get('published'), - 'videoDurationInSeconds': duration, - } - return "Video JSON Metadata:\n"+json.dumps(meta, indent=4)+"\n\n\nAudio Transcript:\n" + text - -def get_duration(json_str): - data = json.loads(json_str) - return data[-1].get('start') - -def fetch_channel_video_information(channel_id, windowSize = 50): - if channel_id == None or len(channel_id) == 0: - print("No channel id provided!") - exit(1) - - if os.path.isdir("./outputs/channel-logs") == False: - os.makedirs("./outputs/channel-logs") - - file_path = f"./outputs/channel-logs/channel-{channel_id}.json" - if os.path.exists(file_path): - with open(file_path, "r") as file: - print(f"Returning cached data for channel {channel_id}. If you do not wish to use stored data then delete the file for this channel to allow refetching.") - return json.load(file) - - if(os.getenv('GOOGLE_APIS_KEY') == None): - print("GOOGLE_APIS_KEY env variable not set!") - exit(1) - - done = False - currentPage = None - pageTokens = [] - items = [] - data = { - 'id': channel_id, - } - - print("Fetching first page of results...") - while(done == False): - url = f"https://www.googleapis.com/youtube/v3/search?key={os.getenv('GOOGLE_APIS_KEY')}&channelId={channel_id}&part=snippet,id&order=date&type=video&maxResults={windowSize}" - if(currentPage != None): - print(f"Fetching page ${currentPage}") - url += f"&pageToken={currentPage}" - - req = requests.get(url) - if(req.ok == False): - print("Could not fetch channel_id items!") - exit(1) - - response = req.json() - currentPage = response.get('nextPageToken') - if currentPage in pageTokens: - print('All pages iterated and logged!') - done = True - break - - for item in response.get('items'): - if 'id' in item and 'videoId' in item.get('id'): - if is_yt_short(item.get('id').get('videoId')): - print(f"Filtering out YT Short {item.get('id').get('videoId')}") - continue - - if data.get('channelTitle') is None: - data['channelTitle'] = slugify(item.get('snippet').get('channelTitle')) - - newItem = { - 'id': item.get('id').get('videoId'), - 'url': f"https://youtube.com/watch?v={item.get('id').get('videoId')}", - 'title': item.get('snippet').get('title'), - 'description': item.get('snippet').get('description'), - 'thumbnail': item.get('snippet').get('thumbnails').get('high').get('url'), - 'published': item.get('snippet').get('publishTime'), - } - items.append(newItem) - - pageTokens.append(currentPage) - - data['items'] = items - with open(file_path, 'w+', encoding='utf-8') as json_file: - json.dump(data, json_file, ensure_ascii=True, indent=2) - print(f"{len(items)} videos found for channel {data.get('channelTitle')}. Saved to channel-logs/channel-{channel_id}.json") - - return data \ No newline at end of file diff --git a/collector/utils/asDocx.js b/collector/utils/asDocx.js new file mode 100644 index 00000000..cd81a5a5 --- /dev/null +++ b/collector/utils/asDocx.js @@ -0,0 +1,50 @@ +const { v4 } = require("uuid"); +const { DocxLoader } = require("langchain/document_loaders/fs/docx"); +const { + createdDate, + trashFile, + writeToServerDocuments, +} = require("../../utils/files"); +const { tokenizeString } = require("../../utils/tokenizer"); +const { default: slugify } = require("slugify"); + +async function asDocX({ fullFilePath = "", filename = "" }) { + const loader = new DocxLoader(fullFilePath); + + console.log(`-- Working ${filename} --`); + let pageContent = []; + const docs = await loader.load(); + for (const doc of docs) { + console.log(doc.metadata); + console.log(`-- Parsing content from docx page --`); + if (!doc.pageContent.length) continue; + pageContent.push(doc.pageContent); + } + + if (!pageContent.length) { + console.error(`Resulting text content was empty for ${filename}.`); + return { success: false, reason: `No text content found in ${filename}.` }; + } + + const content = pageContent.join(""); + data = { + id: v4(), + url: "file://" + fullFilePath, + title: filename, + docAuthor: "no author found", + description: "No description found.", + docSource: "pdf file uploaded by the user.", + chunkSource: filename, + published: createdDate(fullFilePath), + wordCount: content.split(" ").length, + pageContent: content, + token_count_estimate: tokenizeString(content).length, + }; + + writeToServerDocuments(data, `${slugify(filename)}-${data.id}`); + trashFile(fullFilePath); + console.log(`[SUCCESS]: ${filename} converted & ready for embedding.\n`); + return { success: true, reason: null }; +} + +module.exports = asDocX; diff --git a/collector/utils/constants.js b/collector/utils/constants.js new file mode 100644 index 00000000..ca59836c --- /dev/null +++ b/collector/utils/constants.js @@ -0,0 +1,40 @@ +const WATCH_DIRECTORY = require("path").resolve(__dirname, "../hotdir"); + +const ACCEPTED_MIMES = { + "text/plain": [".txt", ".md"], + "text/html": [".html"], + + "application/vnd.openxmlformats-officedocument.wordprocessingml.document": [ + ".docx", + ], + "application/vnd.openxmlformats-officedocument.presentationml.presentation": [ + ".pptx", + ], + + "application/vnd.oasis.opendocument.text": [".odt"], + "application/vnd.oasis.opendocument.presentation": [".odp"], + + "application/pdf": [".pdf"], + "application/mbox": [".mbox"], +}; + +const SUPPORTED_FILETYPE_CONVERTERS = { + ".txt": "./convert/asTxt.js", + ".md": "./convert/asTxt.js", + ".html": "./convert/asTxt.js", + ".pdf": "./convert/asPDF.js", + + ".docx": "./convert/asDocx.js", + ".pptx": "./convert/asOfficeMime.js", + + ".odt": "./convert/asOfficeMime.js", + ".odp": "./convert/asOfficeMime.js", + + ".mbox": "./convert/asMbox.js", +}; + +module.exports = { + SUPPORTED_FILETYPE_CONVERTERS, + WATCH_DIRECTORY, + ACCEPTED_MIMES, +}; diff --git a/collector/utils/files/index.js b/collector/utils/files/index.js new file mode 100644 index 00000000..edcdd54f --- /dev/null +++ b/collector/utils/files/index.js @@ -0,0 +1,55 @@ +const fs = require("fs"); +const path = require("path"); + +function trashFile(filepath) { + if (!fs.existsSync(filepath)) return; + + try { + const isDir = fs.lstatSync(filepath).isDirectory(); + if (isDir) return; + } catch { + return; + } + + fs.rmSync(filepath); + return; +} + +function createdDate(filepath) { + try { + const { birthtimeMs, birthtime } = fs.statSync(filepath); + if (birthtimeMs === 0) throw new Error("Invalid stat for file!"); + return birthtime.toLocaleString(); + } catch { + return "unknown"; + } +} + +function writeToServerDocuments( + data = {}, + filename, + destinationOverride = null +) { + const destination = destinationOverride + ? path.resolve(destinationOverride) + : path.resolve( + __dirname, + "../../../server/storage/documents/custom-documents" + ); + if (!fs.existsSync(destination)) + fs.mkdirSync(destination, { recursive: true }); + const destinationFilePath = path.resolve(destination, filename); + + fs.writeFileSync( + destinationFilePath + ".json", + JSON.stringify(data, null, 4), + { encoding: "utf-8" } + ); + return; +} + +module.exports = { + trashFile, + createdDate, + writeToServerDocuments, +}; diff --git a/collector/utils/http/index.js b/collector/utils/http/index.js new file mode 100644 index 00000000..2283c069 --- /dev/null +++ b/collector/utils/http/index.js @@ -0,0 +1,18 @@ +process.env.NODE_ENV === "development" + ? require("dotenv").config({ path: `.env.${process.env.NODE_ENV}` }) + : require("dotenv").config(); + +function reqBody(request) { + return typeof request.body === "string" + ? JSON.parse(request.body) + : request.body; +} + +function queryParams(request) { + return request.query; +} + +module.exports = { + reqBody, + queryParams, +}; diff --git a/collector/utils/tokenizer/index.js b/collector/utils/tokenizer/index.js new file mode 100644 index 00000000..618a7cdc --- /dev/null +++ b/collector/utils/tokenizer/index.js @@ -0,0 +1,15 @@ +const { getEncoding } = require("js-tiktoken"); + +function tokenizeString(input = "") { + try { + const encoder = getEncoding("cl100k_base"); + return encoder.encode(input); + } catch (e) { + console.error("Could not tokenize string!"); + return []; + } +} + +module.exports = { + tokenizeString, +}; diff --git a/collector/utils/url/index.js b/collector/utils/url/index.js new file mode 100644 index 00000000..3a7f9072 --- /dev/null +++ b/collector/utils/url/index.js @@ -0,0 +1,11 @@ +function validURL(url) { + try { + new URL(url); + return true; + } catch {} + return false; +} + +module.exports = { + validURL, +}; diff --git a/collector/watch.py b/collector/watch.py deleted file mode 100644 index 1223ae2a..00000000 --- a/collector/watch.py +++ /dev/null @@ -1,21 +0,0 @@ -import _thread, time -from scripts.watch.main import watch_for_changes - -a_list = [] -WATCH_DIRECTORY = "hotdir" -def input_thread(a_list): - input() - a_list.append(True) - -def main(): - _thread.start_new_thread(input_thread, (a_list,)) - print(f"Watching '{WATCH_DIRECTORY}/' for new files.\n\nUpload files into this directory while this script is running to convert them.\nPress enter or crtl+c to exit script.") - while not a_list: - watch_for_changes(WATCH_DIRECTORY) - time.sleep(1) - - print("Stopping watching of hot directory.") - exit(1) - -if __name__ == "__main__": - main() \ No newline at end of file diff --git a/collector/wsgi.py b/collector/wsgi.py deleted file mode 100644 index a6f402e6..00000000 --- a/collector/wsgi.py +++ /dev/null @@ -1,4 +0,0 @@ -from api import api - -if __name__ == '__main__': - api.run(debug=False) \ No newline at end of file diff --git a/collector/yarn.lock b/collector/yarn.lock new file mode 100644 index 00000000..b277d7ac --- /dev/null +++ b/collector/yarn.lock @@ -0,0 +1,2925 @@ +# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY. +# yarn lockfile v1 + + +"@anthropic-ai/sdk@^0.9.1": + version "0.9.1" + resolved "https://registry.yarnpkg.com/@anthropic-ai/sdk/-/sdk-0.9.1.tgz#b2d2b7bf05c90dce502c9a2e869066870f69ba88" + integrity sha512-wa1meQ2WSfoY8Uor3EdrJq0jTiZJoKoSii2ZVWRY1oN4Tlr5s59pADg9T79FTbPe1/se5c3pBeZgJL63wmuoBA== + dependencies: + "@types/node" "^18.11.18" + "@types/node-fetch" "^2.6.4" + abort-controller "^3.0.0" + agentkeepalive "^4.2.1" + digest-fetch "^1.3.0" + form-data-encoder "1.7.2" + formdata-node "^4.3.2" + node-fetch "^2.6.7" + web-streams-polyfill "^3.2.1" + +"@babel/code-frame@^7.0.0": + version "7.23.5" + resolved "https://registry.yarnpkg.com/@babel/code-frame/-/code-frame-7.23.5.tgz#9009b69a8c602293476ad598ff53e4562e15c244" + integrity sha512-CgH3s1a96LipHCmSUmYFPwY7MNx8C3avkq7i4Wl3cfa662ldtUe4VM1TPXX70pfmrlWTb6jLqTYrZyT2ZTJBgA== + dependencies: + "@babel/highlight" "^7.23.4" + chalk "^2.4.2" + +"@babel/helper-validator-identifier@^7.22.20": + version "7.22.20" + resolved "https://registry.yarnpkg.com/@babel/helper-validator-identifier/-/helper-validator-identifier-7.22.20.tgz#c4ae002c61d2879e724581d96665583dbc1dc0e0" + integrity sha512-Y4OZ+ytlatR8AI+8KZfKuL5urKp7qey08ha31L8b3BwewJAoJamTzyvxPR/5D+KkdJCGPq/+8TukHBlY10FX9A== + +"@babel/highlight@^7.23.4": + version "7.23.4" + resolved "https://registry.yarnpkg.com/@babel/highlight/-/highlight-7.23.4.tgz#edaadf4d8232e1a961432db785091207ead0621b" + integrity sha512-acGdbYSfp2WheJoJm/EBBBLh/ID8KDc64ISZ9DYtBmC8/Q204PZJLHyzeB5qMzJ5trcOkybd78M4x2KWsUq++A== + dependencies: + "@babel/helper-validator-identifier" "^7.22.20" + chalk "^2.4.2" + js-tokens "^4.0.0" + +"@googleapis/youtube@^9.0.0": + version "9.0.0" + resolved "https://registry.yarnpkg.com/@googleapis/youtube/-/youtube-9.0.0.tgz#e45f6f5f7eac198c6391782b94b3ca54bacf0b63" + integrity sha512-abCi9o1nfODVsPDqiq/QCHte2rXgMXS7ssHoeJIWiRkbvECNNyYFtmXz96oWGMw8Pnpm3gZzMXsmC9cSdMDnvg== + dependencies: + googleapis-common "^6.0.3" + +"@langchain/core@~0.0.8": + version "0.0.9" + resolved "https://registry.yarnpkg.com/@langchain/core/-/core-0.0.9.tgz#3b4e74693eddcfa5f6e649857b561d3a9b60d1d4" + integrity sha512-TpGOvSY/nvpTx3ArVOJPqogfWroxnuD2eGnXvKRroB+DLyKKTxH7EODzgpLpf52C1x1aSnhYNlXoDAbePNNg4Q== + dependencies: + ansi-styles "^5.0.0" + camelcase "6" + decamelize "1.2.0" + js-tiktoken "^1.0.8" + langsmith "^0.0.48" + p-queue "^6.6.2" + p-retry "4" + uuid "^9.0.0" + zod "^3.22.3" + +"@mapbox/node-pre-gyp@^1.0.10": + version "1.0.11" + resolved "https://registry.yarnpkg.com/@mapbox/node-pre-gyp/-/node-pre-gyp-1.0.11.tgz#417db42b7f5323d79e93b34a6d7a2a12c0df43fa" + integrity sha512-Yhlar6v9WQgUp/He7BdgzOz8lqMQ8sU+jkCq7Wx8Myc5YFJLbEe7lgui/V7G1qB1DJykHSGwreceSaD60Y0PUQ== + dependencies: + detect-libc "^2.0.0" + https-proxy-agent "^5.0.0" + make-dir "^3.1.0" + node-fetch "^2.6.7" + nopt "^5.0.0" + npmlog "^5.0.1" + rimraf "^3.0.2" + semver "^7.3.5" + tar "^6.1.11" + +"@puppeteer/browsers@1.9.0": + version "1.9.0" + resolved "https://registry.yarnpkg.com/@puppeteer/browsers/-/browsers-1.9.0.tgz#dfd0aad0bdc039572f1b57648f189525d627b7ff" + integrity sha512-QwguOLy44YBGC8vuPP2nmpX4MUN2FzWbsnvZJtiCzecU3lHmVZkaC1tq6rToi9a200m8RzlVtDyxCS0UIDrxUg== + dependencies: + debug "4.3.4" + extract-zip "2.0.1" + progress "2.0.3" + proxy-agent "6.3.1" + tar-fs "3.0.4" + unbzip2-stream "1.4.3" + yargs "17.7.2" + +"@selderee/plugin-htmlparser2@^0.11.0": + version "0.11.0" + resolved "https://registry.yarnpkg.com/@selderee/plugin-htmlparser2/-/plugin-htmlparser2-0.11.0.tgz#d5b5e29a7ba6d3958a1972c7be16f4b2c188c517" + integrity sha512-P33hHGdldxGabLFjPPpaTxVolMrzrcegejx+0GxjrIb9Zv48D8yAIA/QTDR2dFl7Uz7urX8aX6+5bCZslr+gWQ== + dependencies: + domhandler "^5.0.3" + selderee "^0.11.0" + +"@tokenizer/token@^0.3.0": + version "0.3.0" + resolved "https://registry.yarnpkg.com/@tokenizer/token/-/token-0.3.0.tgz#fe98a93fe789247e998c75e74e9c7c63217aa276" + integrity sha512-OvjF+z51L3ov0OyAU0duzsYuvO01PH7x4t6DJx+guahgTnBHkhJdG7soQeTSFLWN3efnHyibZ4Z8l2EuWwJN3A== + +"@tootallnate/quickjs-emscripten@^0.23.0": + version "0.23.0" + resolved "https://registry.yarnpkg.com/@tootallnate/quickjs-emscripten/-/quickjs-emscripten-0.23.0.tgz#db4ecfd499a9765ab24002c3b696d02e6d32a12c" + integrity sha512-C5Mc6rdnsaJDjO3UpGW/CQTHtCKaYlScZTly4JIu97Jxo/odCiH0ITnDXSJPTOrEKk/ycSZ0AOgTmkDtkOsvIA== + +"@types/mailparser@^3.0.0": + version "3.4.4" + resolved "https://registry.yarnpkg.com/@types/mailparser/-/mailparser-3.4.4.tgz#0bd71e205573b9dd9a445e10a8b8cb0e45420998" + integrity sha512-C6Znp2QVS25JqtuPyxj38Qh+QoFcLycdxsvcc6IZCGekhaMBzbdTXzwGzhGoYb3TfKu8IRCNV0sV1o3Od97cEQ== + dependencies: + "@types/node" "*" + iconv-lite "^0.6.3" + +"@types/node-fetch@^2.6.4": + version "2.6.7" + resolved "https://registry.yarnpkg.com/@types/node-fetch/-/node-fetch-2.6.7.tgz#a1abe2ce24228b58ad97f99480fdcf9bbc6ab16d" + integrity sha512-lX17GZVpJ/fuCjguZ5b3TjEbSENxmEk1B2z02yoXSK9WMEWRivhdSY73wWMn6bpcCDAOh6qAdktpKHIlkDk2lg== + dependencies: + "@types/node" "*" + form-data "^4.0.0" + +"@types/node@*": + version "20.4.2" + resolved "https://registry.yarnpkg.com/@types/node/-/node-20.4.2.tgz#129cc9ae69f93824f92fac653eebfb4812ab4af9" + integrity sha512-Dd0BYtWgnWJKwO1jkmTrzofjK2QXXcai0dmtzvIBhcA+RsG5h8R3xlyta0kGOZRNfL9GuRtb1knmPEhQrePCEw== + +"@types/node@^18.11.18": + version "18.18.7" + resolved "https://registry.yarnpkg.com/@types/node/-/node-18.18.7.tgz#bb3a7068dc4ba421b6968f2a259298b3a4e129e8" + integrity sha512-bw+lEsxis6eqJYW8Ql6+yTqkE6RuFtsQPSe5JxXbqYRFQEER5aJA9a5UH9igqDWm3X4iLHIKOHlnAXLM4mi7uQ== + dependencies: + undici-types "~5.26.4" + +"@types/retry@0.12.0": + version "0.12.0" + resolved "https://registry.yarnpkg.com/@types/retry/-/retry-0.12.0.tgz#2b35eccfcee7d38cd72ad99232fbd58bffb3c84d" + integrity sha512-wWKOClTTiizcZhXnPY4wikVAwmdYHp8q6DmC+EJUzAMsycb7HB32Kh9RN4+0gExjmPmZSAQjgURXIGATPegAvA== + +"@types/uuid@^9.0.1": + version "9.0.7" + resolved "https://registry.yarnpkg.com/@types/uuid/-/uuid-9.0.7.tgz#b14cebc75455eeeb160d5fe23c2fcc0c64f724d8" + integrity sha512-WUtIVRUZ9i5dYXefDEAI7sh9/O7jGvHg7Df/5O/gtH3Yabe5odI3UWopVR1qbPXQtvOxWu3mM4XxlYeZtMWF4g== + +"@types/yauzl@^2.9.1": + version "2.10.0" + resolved "https://registry.yarnpkg.com/@types/yauzl/-/yauzl-2.10.0.tgz#b3248295276cf8c6f153ebe6a9aba0c988cb2599" + integrity sha512-Cn6WYCm0tXv8p6k+A8PvbDG763EDpBoTzHdA+Q/MF6H3sapGjCm9NzoaJncJS9tUKSuCoDs9XHxYYsQDgxR6kw== + dependencies: + "@types/node" "*" + +"@xmldom/xmldom@^0.8.10", "@xmldom/xmldom@^0.8.6": + version "0.8.10" + resolved "https://registry.yarnpkg.com/@xmldom/xmldom/-/xmldom-0.8.10.tgz#a1337ca426aa61cef9fe15b5b28e340a72f6fa99" + integrity sha512-2WALfTl4xo2SkGCYRt6rDTFfk9R1czmBvUQy12gK2KuRKIpWEhcbbzy8EZXtz/jkRqHX8bFEc6FC1HjX4TUWYw== + +abbrev@1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/abbrev/-/abbrev-1.1.1.tgz#f8f2c887ad10bf67f634f005b6987fed3179aac8" + integrity sha512-nne9/IiQ/hzIhY6pdDnbBtz7DjPTKrY00P/zvPSm5pOFkl6xuGrGnXn/VtTNNfNtAfZ9/1RtehkszU9qcTii0Q== + +abort-controller@^3.0.0: + version "3.0.0" + resolved "https://registry.yarnpkg.com/abort-controller/-/abort-controller-3.0.0.tgz#eaf54d53b62bae4138e809ca225c8439a6efb392" + integrity sha512-h8lQ8tacZYnR3vNQTgibj+tODHI5/+l06Au2Pcriv/Gmet0eaj4TwWH41sO9wnHDiQsEj19q0drzdWdeAHtweg== + dependencies: + event-target-shim "^5.0.0" + +accepts@~1.3.8: + version "1.3.8" + resolved "https://registry.yarnpkg.com/accepts/-/accepts-1.3.8.tgz#0bf0be125b67014adcb0b0921e62db7bffe16b2e" + integrity sha512-PYAthTa2m2VKxuvSD3DPC/Gy+U+sOA1LAuT8mkmRuvw+NACSaeXEQ+NHcVF7rONl6qcaxV3Uuemwawk+7+SJLw== + dependencies: + mime-types "~2.1.34" + negotiator "0.6.3" + +agent-base@6: + version "6.0.2" + resolved "https://registry.yarnpkg.com/agent-base/-/agent-base-6.0.2.tgz#49fff58577cfee3f37176feab4c22e00f86d7f77" + integrity sha512-RZNwNclF7+MS/8bDg70amg32dyeZGZxiDuQmZxKLAlQjr3jGyLx+4Kkk58UO7D2QdgFIQCovuSuZESne6RG6XQ== + dependencies: + debug "4" + +agent-base@^7.0.2, agent-base@^7.1.0: + version "7.1.0" + resolved "https://registry.yarnpkg.com/agent-base/-/agent-base-7.1.0.tgz#536802b76bc0b34aa50195eb2442276d613e3434" + integrity sha512-o/zjMZRhJxny7OyEF+Op8X+efiELC7k7yOjMzgfzVqOzXqkBkWI79YoTdOtsuWd5BWhAGAuOY/Xa6xpiaWXiNg== + dependencies: + debug "^4.3.4" + +agentkeepalive@^4.2.1: + version "4.5.0" + resolved "https://registry.yarnpkg.com/agentkeepalive/-/agentkeepalive-4.5.0.tgz#2673ad1389b3c418c5a20c5d7364f93ca04be923" + integrity sha512-5GG/5IbQQpC9FpkRGsSvZI5QYeSCzlJHdpBQntCsuTOxhKD8lqKhrleg2Yi7yvMIf82Ycmmqln9U8V9qwEiJew== + dependencies: + humanize-ms "^1.2.1" + +ansi-regex@^5.0.1: + version "5.0.1" + resolved "https://registry.yarnpkg.com/ansi-regex/-/ansi-regex-5.0.1.tgz#082cb2c89c9fe8659a311a53bd6a4dc5301db304" + integrity sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ== + +ansi-styles@^3.2.1: + version "3.2.1" + resolved "https://registry.yarnpkg.com/ansi-styles/-/ansi-styles-3.2.1.tgz#41fbb20243e50b12be0f04b8dedbf07520ce841d" + integrity sha512-VT0ZI6kZRdTh8YyJw3SMbYm/u+NqfsAxEpWO0Pf9sq8/e94WxxOpPKx9FR1FlyCtOVDNOQ+8ntlqFxiRc+r5qA== + dependencies: + color-convert "^1.9.0" + +ansi-styles@^4.0.0: + version "4.3.0" + resolved "https://registry.yarnpkg.com/ansi-styles/-/ansi-styles-4.3.0.tgz#edd803628ae71c04c85ae7a0906edad34b648937" + integrity sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg== + dependencies: + color-convert "^2.0.1" + +ansi-styles@^5.0.0: + version "5.2.0" + resolved "https://registry.yarnpkg.com/ansi-styles/-/ansi-styles-5.2.0.tgz#07449690ad45777d1924ac2abb2fc8895dba836b" + integrity sha512-Cxwpt2SfTzTtXcfOlzGEee8O+c+MmUgGrNiBcXnuWxuFJHe6a5Hz7qwhwe5OgaSYI0IJvkLqWX1ASG+cJOkEiA== + +anymatch@~3.1.2: + version "3.1.3" + resolved "https://registry.yarnpkg.com/anymatch/-/anymatch-3.1.3.tgz#790c58b19ba1720a84205b57c618d5ad8524973e" + integrity sha512-KMReFUr0B4t+D+OBkjR3KYqvocp2XaSzO55UcB6mgQMd3KbcE+mWTyvVV7D/zsdEbNnV6acZUutkiHQXvTr1Rw== + dependencies: + normalize-path "^3.0.0" + picomatch "^2.0.4" + +append-field@^1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/append-field/-/append-field-1.0.0.tgz#1e3440e915f0b1203d23748e78edd7b9b5b43e56" + integrity sha512-klpgFSWLW1ZEs8svjfb7g4qWY0YS5imI82dTg+QahUvJ8YqAY0P10Uk8tTyh9ZGuYEZEMaeJYCF5BFuX552hsw== + +"aproba@^1.0.3 || ^2.0.0": + version "2.0.0" + resolved "https://registry.yarnpkg.com/aproba/-/aproba-2.0.0.tgz#52520b8ae5b569215b354efc0caa3fe1e45a8adc" + integrity sha512-lYe4Gx7QT+MKGbDsA+Z+he/Wtef0BiwDOlK/XkBrdfsh9J/jPPXbX0tE9x9cl27Tmu5gg3QUbUrQYa/y+KOHPQ== + +are-we-there-yet@^2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/are-we-there-yet/-/are-we-there-yet-2.0.0.tgz#372e0e7bd279d8e94c653aaa1f67200884bf3e1c" + integrity sha512-Ci/qENmwHnsYo9xKIcUJN5LeDKdJ6R1Z1j9V/J5wyq8nh/mYPEpIKJbBZXtZjG04HiK7zV/p6Vs9952MrMeUIw== + dependencies: + delegates "^1.0.0" + readable-stream "^3.6.0" + +argparse@^2.0.1: + version "2.0.1" + resolved "https://registry.yarnpkg.com/argparse/-/argparse-2.0.1.tgz#246f50f3ca78a3240f6c997e8a9bd1eac49e4b38" + integrity sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q== + +argparse@~1.0.3: + version "1.0.10" + resolved "https://registry.yarnpkg.com/argparse/-/argparse-1.0.10.tgz#bcd6791ea5ae09725e17e5ad988134cd40b3d911" + integrity sha512-o5Roy6tNG4SL/FOkCAN6RzjiakZS25RLYFrcMttJqbdd8BWrnA+fGz57iN5Pb06pvBGvl5gQ0B48dJlslXvoTg== + dependencies: + sprintf-js "~1.0.2" + +array-flatten@1.1.1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/array-flatten/-/array-flatten-1.1.1.tgz#9a5f699051b1e7073328f2a008968b64ea2955d2" + integrity sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg== + +arrify@^2.0.0: + version "2.0.1" + resolved "https://registry.yarnpkg.com/arrify/-/arrify-2.0.1.tgz#c9655e9331e0abcd588d2a7cad7e9956f66701fa" + integrity sha512-3duEwti880xqi4eAMN8AyR4a0ByT90zoYdLlevfrvU43vb0YZwZVfxOgxWrLXXXpyugL0hNZc9G6BiB5B3nUug== + +ast-types@^0.13.4: + version "0.13.4" + resolved "https://registry.yarnpkg.com/ast-types/-/ast-types-0.13.4.tgz#ee0d77b343263965ecc3fb62da16e7222b2b6782" + integrity sha512-x1FCFnFifvYDDzTaLII71vG5uvDwgtmDTEVWAxrgeiR8VjMONcCXJx7E+USjDtHlwFmt9MysbqgF9b9Vjr6w+w== + dependencies: + tslib "^2.0.1" + +asynckit@^0.4.0: + version "0.4.0" + resolved "https://registry.yarnpkg.com/asynckit/-/asynckit-0.4.0.tgz#c79ed97f7f34cb8f2ba1bc9790bcc366474b4b79" + integrity sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q== + +b4a@^1.6.4: + version "1.6.4" + resolved "https://registry.yarnpkg.com/b4a/-/b4a-1.6.4.tgz#ef1c1422cae5ce6535ec191baeed7567443f36c9" + integrity sha512-fpWrvyVHEKyeEvbKZTVOeZF3VSKKWtJxFIxX/jaVPf+cLbGUSitjb49pHLqPV2BUNNZ0LcoeEGfE/YCpyDYHIw== + +balanced-match@^1.0.0: + version "1.0.2" + resolved "https://registry.yarnpkg.com/balanced-match/-/balanced-match-1.0.2.tgz#e83e3a7e3f300b34cb9d87f615fa0cbf357690ee" + integrity sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw== + +base-64@^0.1.0: + version "0.1.0" + resolved "https://registry.yarnpkg.com/base-64/-/base-64-0.1.0.tgz#780a99c84e7d600260361511c4877613bf24f6bb" + integrity sha512-Y5gU45svrR5tI2Vt/X9GPd3L0HNIKzGu202EjxrXMpuc2V2CiKgemAbUUsqYmZJvPtCXoUKjNZwBJzsNScUbXA== + +base64-js@^1.3.0, base64-js@^1.3.1, base64-js@^1.5.1: + version "1.5.1" + resolved "https://registry.yarnpkg.com/base64-js/-/base64-js-1.5.1.tgz#1b1b440160a5bf7ad40b650f095963481903930a" + integrity sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA== + +basic-ftp@^5.0.2: + version "5.0.4" + resolved "https://registry.yarnpkg.com/basic-ftp/-/basic-ftp-5.0.4.tgz#28aeab7bfbbde5f5d0159cd8bb3b8e633bbb091d" + integrity sha512-8PzkB0arJFV4jJWSGOYR+OEic6aeKMu/osRhBULN6RY0ykby6LKhbmuQ5ublvaas5BOwboah5D87nrHyuh8PPA== + +bcrypt@^5.1.0: + version "5.1.0" + resolved "https://registry.yarnpkg.com/bcrypt/-/bcrypt-5.1.0.tgz#bbb27665dbc400480a524d8991ac7434e8529e17" + integrity sha512-RHBS7HI5N5tEnGTmtR/pppX0mmDSBpQ4aCBsj7CEQfYXDcO74A8sIBYcJMuCsis2E81zDxeENYhv66oZwLiA+Q== + dependencies: + "@mapbox/node-pre-gyp" "^1.0.10" + node-addon-api "^5.0.0" + +bignumber.js@^9.0.0: + version "9.1.1" + resolved "https://registry.yarnpkg.com/bignumber.js/-/bignumber.js-9.1.1.tgz#c4df7dc496bd849d4c9464344c1aa74228b4dac6" + integrity sha512-pHm4LsMJ6lzgNGVfZHjMoO8sdoRhOzOH4MLmY65Jg70bpxCKu5iOHNJyfF6OyvYw7t8Fpf35RuzUyqnQsj8Vig== + +binary-extensions@^2.0.0, binary-extensions@^2.2.0: + version "2.2.0" + resolved "https://registry.yarnpkg.com/binary-extensions/-/binary-extensions-2.2.0.tgz#75f502eeaf9ffde42fc98829645be4ea76bd9e2d" + integrity sha512-jDctJ/IVQbZoJykoeHbhXpOlNBqGNcwXJKJog42E5HDPUwQTSdjCHdihjj0DlnheQ7blbT6dHOafNAiS8ooQKA== + +binary-search@^1.3.5: + version "1.3.6" + resolved "https://registry.yarnpkg.com/binary-search/-/binary-search-1.3.6.tgz#e32426016a0c5092f0f3598836a1c7da3560565c" + integrity sha512-nbE1WxOTTrUWIfsfZ4aHGYu5DOuNkbxGokjV6Z2kxfJK3uaAb8zNK1muzOeipoLHZjInT4Br88BHpzevc681xA== + +bl@^1.0.0: + version "1.2.3" + resolved "https://registry.yarnpkg.com/bl/-/bl-1.2.3.tgz#1e8dd80142eac80d7158c9dccc047fb620e035e7" + integrity sha512-pvcNpa0UU69UT341rO6AYy4FVAIkUHuZXRIWbq+zHnsVcRzDDjIAhGuuYoi0d//cwIwtt4pkpKycWEfjdV+vww== + dependencies: + readable-stream "^2.3.5" + safe-buffer "^5.1.1" + +bluebird@~3.4.0: + version "3.4.7" + resolved "https://registry.yarnpkg.com/bluebird/-/bluebird-3.4.7.tgz#f72d760be09b7f76d08ed8fae98b289a8d05fab3" + integrity sha512-iD3898SR7sWVRHbiQv+sHUtHnMvC1o3nW5rAcqnq3uOn07DSAppZYUkIGslDz6gXC7HfunPe7YVBgoEJASPcHA== + +body-parser@1.20.1: + version "1.20.1" + resolved "https://registry.yarnpkg.com/body-parser/-/body-parser-1.20.1.tgz#b1812a8912c195cd371a3ee5e66faa2338a5c668" + integrity sha512-jWi7abTbYwajOytWCQc37VulmWiRae5RyTpaCyDcS5/lMdtwSz5lOpDE67srw/HYe35f1z3fDQw+3txg7gNtWw== + dependencies: + bytes "3.1.2" + content-type "~1.0.4" + debug "2.6.9" + depd "2.0.0" + destroy "1.2.0" + http-errors "2.0.0" + iconv-lite "0.4.24" + on-finished "2.4.1" + qs "6.11.0" + raw-body "2.5.1" + type-is "~1.6.18" + unpipe "1.0.0" + +body-parser@^1.20.2: + version "1.20.2" + resolved "https://registry.yarnpkg.com/body-parser/-/body-parser-1.20.2.tgz#6feb0e21c4724d06de7ff38da36dad4f57a747fd" + integrity sha512-ml9pReCu3M61kGlqoTm2umSXTlRTuGTx0bfYj+uIUKKYycG5NtSbeetV3faSU6R7ajOPw0g/J1PvK4qNy7s5bA== + dependencies: + bytes "3.1.2" + content-type "~1.0.5" + debug "2.6.9" + depd "2.0.0" + destroy "1.2.0" + http-errors "2.0.0" + iconv-lite "0.4.24" + on-finished "2.4.1" + qs "6.11.0" + raw-body "2.5.2" + type-is "~1.6.18" + unpipe "1.0.0" + +brace-expansion@^1.1.7: + version "1.1.11" + resolved "https://registry.yarnpkg.com/brace-expansion/-/brace-expansion-1.1.11.tgz#3c7fcbf529d87226f3d2f52b966ff5271eb441dd" + integrity sha512-iCuPHDFgrHX7H2vEI/5xpz07zSHB00TpugqhmYtVmMO6518mCuRMoOYFldEBl0g187ufozdaHgWKcYFb61qGiA== + dependencies: + balanced-match "^1.0.0" + concat-map "0.0.1" + +braces@~3.0.2: + version "3.0.2" + resolved "https://registry.yarnpkg.com/braces/-/braces-3.0.2.tgz#3454e1a462ee8d599e236df336cd9ea4f8afe107" + integrity sha512-b8um+L1RzM3WDSzvhm6gIz1yfTbBt6YTlcEKAvsmqCZZFw46z626lVj9j1yEPW33H5H+lBQpZMP1k8l+78Ha0A== + dependencies: + fill-range "^7.0.1" + +buffer-alloc-unsafe@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/buffer-alloc-unsafe/-/buffer-alloc-unsafe-1.1.0.tgz#bd7dc26ae2972d0eda253be061dba992349c19f0" + integrity sha512-TEM2iMIEQdJ2yjPJoSIsldnleVaAk1oW3DBVUykyOLsEsFmEc9kn+SFFPz+gl54KQNxlDnAwCXosOS9Okx2xAg== + +buffer-alloc@^1.2.0: + version "1.2.0" + resolved "https://registry.yarnpkg.com/buffer-alloc/-/buffer-alloc-1.2.0.tgz#890dd90d923a873e08e10e5fd51a57e5b7cce0ec" + integrity sha512-CFsHQgjtW1UChdXgbyJGtnm+O/uLQeZdtbDo8mfUgYXCHSM1wgrVxXm6bSyrUuErEb+4sYVGCzASBRot7zyrow== + dependencies: + buffer-alloc-unsafe "^1.1.0" + buffer-fill "^1.0.0" + +buffer-crc32@~0.2.3: + version "0.2.13" + resolved "https://registry.yarnpkg.com/buffer-crc32/-/buffer-crc32-0.2.13.tgz#0d333e3f00eac50aa1454abd30ef8c2a5d9a7242" + integrity sha512-VO9Ht/+p3SN7SKWqcrgEzjGbRSJYTx+Q1pTQC0wrWqHx0vpJraQ6GtHx8tvcg1rlK1byhU5gccxgOgj7B0TDkQ== + +buffer-equal-constant-time@1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz#f8e71132f7ffe6e01a5c9697a4c6f3e48d5cc819" + integrity sha512-zRpUiDwd/xk6ADqPMATG8vc9VPrkck7T07OIx0gnjmJAnHnTVXNQG3vfvWNuiZIkwu9KrKdA1iJKfsfTVxE6NA== + +buffer-fill@^1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/buffer-fill/-/buffer-fill-1.0.0.tgz#f8f78b76789888ef39f205cd637f68e702122b2c" + integrity sha512-T7zexNBwiiaCOGDg9xNX9PBmjrubblRkENuptryuI64URkXDFum9il/JGL8Lm8wYfAXpredVXXZz7eMHilimiQ== + +buffer-from@^1.0.0: + version "1.1.2" + resolved "https://registry.yarnpkg.com/buffer-from/-/buffer-from-1.1.2.tgz#2b146a6fd72e80b4f55d255f35ed59a3a9a41bd5" + integrity sha512-E+XQCRwSbaaiChtv6k6Dwgc+bx+Bs6vuKJHHl5kox/BaKbhiXzqQOwK4cO22yElGp2OCmjwVhT3HmxgyPGnJfQ== + +buffer@^5.2.1: + version "5.7.1" + resolved "https://registry.yarnpkg.com/buffer/-/buffer-5.7.1.tgz#ba62e7c13133053582197160851a8f648e99eed0" + integrity sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ== + dependencies: + base64-js "^1.3.1" + ieee754 "^1.1.13" + +busboy@^1.0.0: + version "1.6.0" + resolved "https://registry.yarnpkg.com/busboy/-/busboy-1.6.0.tgz#966ea36a9502e43cdb9146962523b92f531f6893" + integrity sha512-8SFQbg/0hQ9xy3UNTB0YEnsNBbWfhf7RtnzpL7TkBiTBRfrQ9Fxcnz7VJsleJpyp6rVLvXiuORqjlHi5q+PYuA== + dependencies: + streamsearch "^1.1.0" + +bytes@3.1.2: + version "3.1.2" + resolved "https://registry.yarnpkg.com/bytes/-/bytes-3.1.2.tgz#8b0beeb98605adf1b128fa4386403c009e0221a5" + integrity sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg== + +call-bind@^1.0.0: + version "1.0.2" + resolved "https://registry.yarnpkg.com/call-bind/-/call-bind-1.0.2.tgz#b1d4e89e688119c3c9a903ad30abb2f6a919be3c" + integrity sha512-7O+FbCihrB5WGbFYesctwmTKae6rOiIzmz1icreWJ+0aA7LJfuqhEso2T9ncpcFtzMQtzXf2QGGueWJGTYsqrA== + dependencies: + function-bind "^1.1.1" + get-intrinsic "^1.0.2" + +callsites@^3.0.0: + version "3.1.0" + resolved "https://registry.yarnpkg.com/callsites/-/callsites-3.1.0.tgz#b3630abd8943432f54b3f0519238e33cd7df2f73" + integrity sha512-P8BjAsXvZS+VIDUI11hHCQEv74YT67YUi5JJFNWIqL235sBmjX4+qx9Muvls5ivyNENctx46xQLQ3aTuE7ssaQ== + +camelcase@6: + version "6.3.0" + resolved "https://registry.yarnpkg.com/camelcase/-/camelcase-6.3.0.tgz#5685b95eb209ac9c0c177467778c9c84df58ba9a" + integrity sha512-Gmy6FhYlCY7uOElZUSbxo2UCDH8owEk996gkbrpsgGtrJLM3J7jGxl9Ic7Qwwj4ivOE5AWZWRMecDdF7hqGjFA== + +chalk@^2.4.2: + version "2.4.2" + resolved "https://registry.yarnpkg.com/chalk/-/chalk-2.4.2.tgz#cd42541677a54333cf541a49108c1432b44c9424" + integrity sha512-Mti+f9lpJNcwF4tWV8/OrTTtF1gZi+f8FqlyAdouralcFWFQWF2+NgCHShjkCb+IFBLq9buZwE1xckQU4peSuQ== + dependencies: + ansi-styles "^3.2.1" + escape-string-regexp "^1.0.5" + supports-color "^5.3.0" + +charenc@0.0.2: + version "0.0.2" + resolved "https://registry.yarnpkg.com/charenc/-/charenc-0.0.2.tgz#c0a1d2f3a7092e03774bfa83f14c0fc5790a8667" + integrity sha512-yrLQ/yVUFXkzg7EDQsPieE/53+0RlaWTs+wBrvW36cyilJ2SaDWfl4Yj7MtLTXleV9uEKefbAGUPv2/iWSooRA== + +chokidar@^3.5.2: + version "3.5.3" + resolved "https://registry.yarnpkg.com/chokidar/-/chokidar-3.5.3.tgz#1cf37c8707b932bd1af1ae22c0432e2acd1903bd" + integrity sha512-Dr3sfKRP6oTcjf2JmUmFJfeVMvXBdegxB0iVQ5eb2V10uFJUCAS8OByZdVAyVb8xXNz3GjjTgj9kLWsZTqE6kw== + dependencies: + anymatch "~3.1.2" + braces "~3.0.2" + glob-parent "~5.1.2" + is-binary-path "~2.1.0" + is-glob "~4.0.1" + normalize-path "~3.0.0" + readdirp "~3.6.0" + optionalDependencies: + fsevents "~2.3.2" + +chownr@^2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/chownr/-/chownr-2.0.0.tgz#15bfbe53d2eab4cf70f18a8cd68ebe5b3cb1dece" + integrity sha512-bIomtDF5KGpdogkLd9VspvFzk9KfpyyGlS8YFVZl7TGPBHL5snIOnxeshwVgPteQ9b4Eydl+pVbIyE1DcvCWgQ== + +chromium-bidi@0.5.1: + version "0.5.1" + resolved "https://registry.yarnpkg.com/chromium-bidi/-/chromium-bidi-0.5.1.tgz#390c1af350c4887824a33d82190de1cc5c5680fc" + integrity sha512-dcCqOgq9fHKExc2R4JZs/oKbOghWpUNFAJODS8WKRtLhp3avtIH5UDCBrutdqZdh3pARogH8y1ObXm87emwb3g== + dependencies: + mitt "3.0.1" + urlpattern-polyfill "9.0.0" + +cliui@^8.0.1: + version "8.0.1" + resolved "https://registry.yarnpkg.com/cliui/-/cliui-8.0.1.tgz#0c04b075db02cbfe60dc8e6cf2f5486b1a3608aa" + integrity sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ== + dependencies: + string-width "^4.2.0" + strip-ansi "^6.0.1" + wrap-ansi "^7.0.0" + +color-convert@^1.9.0: + version "1.9.3" + resolved "https://registry.yarnpkg.com/color-convert/-/color-convert-1.9.3.tgz#bb71850690e1f136567de629d2d5471deda4c1e8" + integrity sha512-QfAUtd+vFdAtFQcC8CCyYt1fYWxSqAiK2cSD6zDB8N3cpsEBAvRxp9zOGg6G/SHHJYAT88/az/IuDGALsNVbGg== + dependencies: + color-name "1.1.3" + +color-convert@^2.0.1: + version "2.0.1" + resolved "https://registry.yarnpkg.com/color-convert/-/color-convert-2.0.1.tgz#72d3a68d598c9bdb3af2ad1e84f21d896abd4de3" + integrity sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ== + dependencies: + color-name "~1.1.4" + +color-name@1.1.3: + version "1.1.3" + resolved "https://registry.yarnpkg.com/color-name/-/color-name-1.1.3.tgz#a7d0558bd89c42f795dd42328f740831ca53bc25" + integrity sha512-72fSenhMw2HZMTVHeCA9KCmpEIbzWiQsjN+BHcBbS9vr1mtt+vJjPdksIBNUmKAW8TFUDPJK5SUU3QhE9NEXDw== + +color-name@~1.1.4: + version "1.1.4" + resolved "https://registry.yarnpkg.com/color-name/-/color-name-1.1.4.tgz#c2a09a87acbde69543de6f63fa3995c826c536a2" + integrity sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA== + +color-support@^1.1.2: + version "1.1.3" + resolved "https://registry.yarnpkg.com/color-support/-/color-support-1.1.3.tgz#93834379a1cc9a0c61f82f52f0d04322251bd5a2" + integrity sha512-qiBjkpbMLO/HL68y+lh4q0/O1MZFj2RX6X/KmMa3+gJD3z+WwI1ZzDHysvqHGS3mP6mznPckpXmw1nI9cJjyRg== + +combined-stream@^1.0.8: + version "1.0.8" + resolved "https://registry.yarnpkg.com/combined-stream/-/combined-stream-1.0.8.tgz#c3d45a8b34fd730631a110a8a2520682b31d5a7f" + integrity sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg== + dependencies: + delayed-stream "~1.0.0" + +commander@^10.0.1: + version "10.0.1" + resolved "https://registry.yarnpkg.com/commander/-/commander-10.0.1.tgz#881ee46b4f77d1c1dccc5823433aa39b022cbe06" + integrity sha512-y4Mg2tXshplEbSGzx7amzPwKKOCGuoSRP/CjEdwwk0FOGlUbq6lKuoyDZTNZkmxHdJtp54hdfY/JUrdL7Xfdug== + +commander@^2.8.1: + version "2.20.3" + resolved "https://registry.yarnpkg.com/commander/-/commander-2.20.3.tgz#fd485e84c03eb4881c20722ba48035e8531aeb33" + integrity sha512-GpVkmM8vF2vQUkj2LvZmD35JxeJOLCwJ9cUkugyk2nuhbv3+mJvpLYYt+0+USMxE+oj+ey/lJEnhZw75x/OMcQ== + +concat-map@0.0.1: + version "0.0.1" + resolved "https://registry.yarnpkg.com/concat-map/-/concat-map-0.0.1.tgz#d8a96bd77fd68df7793a73036a3ba0d5405d477b" + integrity sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg== + +concat-stream@^1.5.2: + version "1.6.2" + resolved "https://registry.yarnpkg.com/concat-stream/-/concat-stream-1.6.2.tgz#904bdf194cd3122fc675c77fc4ac3d4ff0fd1a34" + integrity sha512-27HBghJxjiZtIk3Ycvn/4kbJk/1uZuJFfuPEns6LaEvpvG1f0hTea8lilrouyo9mVc2GWdcEZ8OLoGmSADlrCw== + dependencies: + buffer-from "^1.0.0" + inherits "^2.0.3" + readable-stream "^2.2.2" + typedarray "^0.0.6" + +console-control-strings@^1.0.0, console-control-strings@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/console-control-strings/-/console-control-strings-1.1.0.tgz#3d7cf4464db6446ea644bf4b39507f9851008e8e" + integrity sha512-ty/fTekppD2fIwRvnZAVdeOiGd1c7YXEixbgJTNzqcxJWKQnjJ/V1bNEEE6hygpM3WjwHFUVK6HTjWSzV4a8sQ== + +content-disposition@0.5.4: + version "0.5.4" + resolved "https://registry.yarnpkg.com/content-disposition/-/content-disposition-0.5.4.tgz#8b82b4efac82512a02bb0b1dcec9d2c5e8eb5bfe" + integrity sha512-FveZTNuGw04cxlAiWbzi6zTAL/lhehaWbTtgluJh4/E95DqMwTmha3KZN1aAWA8cFIhHzMZUvLevkw5Rqk+tSQ== + dependencies: + safe-buffer "5.2.1" + +content-type@~1.0.4, content-type@~1.0.5: + version "1.0.5" + resolved "https://registry.yarnpkg.com/content-type/-/content-type-1.0.5.tgz#8b773162656d1d1086784c8f23a54ce6d73d7918" + integrity sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA== + +cookie-signature@1.0.6: + version "1.0.6" + resolved "https://registry.yarnpkg.com/cookie-signature/-/cookie-signature-1.0.6.tgz#e303a882b342cc3ee8ca513a79999734dab3ae2c" + integrity sha512-QADzlaHc8icV8I7vbaJXJwod9HWYp8uCqf1xa4OfNu1T7JVxQIrUgOWtHdNDtPiywmFbiS12VjotIXLrKM3orQ== + +cookie@0.5.0: + version "0.5.0" + resolved "https://registry.yarnpkg.com/cookie/-/cookie-0.5.0.tgz#d1f5d71adec6558c58f389987c366aa47e994f8b" + integrity sha512-YZ3GUyn/o8gfKJlnlX7g7xq4gyO6OSuhGPKaaGssGB2qgDUS0gPgtTvoyZLTt9Ab6dC4hfc9dV5arkvc/OCmrw== + +core-util-is@~1.0.0: + version "1.0.3" + resolved "https://registry.yarnpkg.com/core-util-is/-/core-util-is-1.0.3.tgz#a6042d3634c2b27e9328f837b965fac83808db85" + integrity sha512-ZQBvi1DcpJ4GDqanjucZ2Hj3wEO5pZDS89BWbkcrvdxksJorwUDDZamX9ldFkp9aw2lmBDLgkObEA4DWNJ9FYQ== + +cors@^2.8.5: + version "2.8.5" + resolved "https://registry.yarnpkg.com/cors/-/cors-2.8.5.tgz#eac11da51592dd86b9f06f6e7ac293b3df875d29" + integrity sha512-KIHbLJqu73RGr/hnbrO9uBeixNGuvSQjul/jdFvS/KFSIH1hWVd1ng7zOHx+YrEfInLG7q4n6GHQ9cDtxv/P6g== + dependencies: + object-assign "^4" + vary "^1" + +cosmiconfig@8.3.6: + version "8.3.6" + resolved "https://registry.yarnpkg.com/cosmiconfig/-/cosmiconfig-8.3.6.tgz#060a2b871d66dba6c8538ea1118ba1ac16f5fae3" + integrity sha512-kcZ6+W5QzcJ3P1Mt+83OUv/oHFqZHIx8DuxG6eZ5RGMERoLqp4BuGjhHLYGK+Kf5XVkQvqBSmAy/nGWN3qDgEA== + dependencies: + import-fresh "^3.3.0" + js-yaml "^4.1.0" + parse-json "^5.2.0" + path-type "^4.0.0" + +cross-fetch@4.0.0: + version "4.0.0" + resolved "https://registry.yarnpkg.com/cross-fetch/-/cross-fetch-4.0.0.tgz#f037aef1580bb3a1a35164ea2a848ba81b445983" + integrity sha512-e4a5N8lVvuLgAWgnCrLr2PP0YyDOTHa9H/Rj54dirp61qXnNq46m82bRhNqIA5VccJtWBvPTFRV3TtvHUKPB1g== + dependencies: + node-fetch "^2.6.12" + +crypt@0.0.2: + version "0.0.2" + resolved "https://registry.yarnpkg.com/crypt/-/crypt-0.0.2.tgz#88d7ff7ec0dfb86f713dc87bbb42d044d3e6c41b" + integrity sha512-mCxBlsHFYh9C+HVpiEacem8FEBnMXgU9gy4zmNC+SXAZNB/1idgp/aulFJ4FgCi7GPEVbfyng092GqL2k2rmow== + +data-uri-to-buffer@^6.0.0: + version "6.0.1" + resolved "https://registry.yarnpkg.com/data-uri-to-buffer/-/data-uri-to-buffer-6.0.1.tgz#540bd4c8753a25ee129035aebdedf63b078703c7" + integrity sha512-MZd3VlchQkp8rdend6vrx7MmVDJzSNTBvghvKjirLkD+WTChA3KUf0jkE68Q4UyctNqI11zZO9/x2Yx+ub5Cvg== + +debug@2.6.9: + version "2.6.9" + resolved "https://registry.yarnpkg.com/debug/-/debug-2.6.9.tgz#5d128515df134ff327e90a4c93f4e077a536341f" + integrity sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA== + dependencies: + ms "2.0.0" + +debug@4, debug@4.3.4, debug@^4.1.1, debug@^4.3.4: + version "4.3.4" + resolved "https://registry.yarnpkg.com/debug/-/debug-4.3.4.tgz#1319f6579357f2338d3337d2cdd4914bb5dcc865" + integrity sha512-PRWFHuSU3eDtQJPvnNY7Jcket1j0t5OuOsFzPPzsekD52Zl8qUfFIPEiswXqIvHWGVHOgX+7G/vCNNhehwxfkQ== + dependencies: + ms "2.1.2" + +debug@^3.1.0, debug@^3.2.7: + version "3.2.7" + resolved "https://registry.yarnpkg.com/debug/-/debug-3.2.7.tgz#72580b7e9145fb39b6676f9c5e5fb100b934179a" + integrity sha512-CFjzYYAi4ThfiQvizrFQevTTXHtnCqWfe7x1AhgEscTz6ZbLbfoLRLPugTQyBth6f8ZERVUSyWHFD/7Wu4t1XQ== + dependencies: + ms "^2.1.1" + +decamelize@1.2.0: + version "1.2.0" + resolved "https://registry.yarnpkg.com/decamelize/-/decamelize-1.2.0.tgz#f6534d15148269b20352e7bee26f501f9a191290" + integrity sha512-z2S+W9X73hAUUki+N+9Za2lBlun89zigOyGrsax+KUQ6wKW4ZoWpEYBkGhQjwAjjDCkWxhY0VKEhk8wzY7F5cA== + +decompress-tar@^4.0.0, decompress-tar@^4.1.0, decompress-tar@^4.1.1: + version "4.1.1" + resolved "https://registry.yarnpkg.com/decompress-tar/-/decompress-tar-4.1.1.tgz#718cbd3fcb16209716e70a26b84e7ba4592e5af1" + integrity sha512-JdJMaCrGpB5fESVyxwpCx4Jdj2AagLmv3y58Qy4GE6HMVjWz1FeVQk1Ct4Kye7PftcdOo/7U7UKzYBJgqnGeUQ== + dependencies: + file-type "^5.2.0" + is-stream "^1.1.0" + tar-stream "^1.5.2" + +decompress-tarbz2@^4.0.0: + version "4.1.1" + resolved "https://registry.yarnpkg.com/decompress-tarbz2/-/decompress-tarbz2-4.1.1.tgz#3082a5b880ea4043816349f378b56c516be1a39b" + integrity sha512-s88xLzf1r81ICXLAVQVzaN6ZmX4A6U4z2nMbOwobxkLoIIfjVMBg7TeguTUXkKeXni795B6y5rnvDw7rxhAq9A== + dependencies: + decompress-tar "^4.1.0" + file-type "^6.1.0" + is-stream "^1.1.0" + seek-bzip "^1.0.5" + unbzip2-stream "^1.0.9" + +decompress-targz@^4.0.0: + version "4.1.1" + resolved "https://registry.yarnpkg.com/decompress-targz/-/decompress-targz-4.1.1.tgz#c09bc35c4d11f3de09f2d2da53e9de23e7ce1eee" + integrity sha512-4z81Znfr6chWnRDNfFNqLwPvm4db3WuZkqV+UgXQzSngG3CEKdBkw5jrv3axjjL96glyiiKjsxJG3X6WBZwX3w== + dependencies: + decompress-tar "^4.1.1" + file-type "^5.2.0" + is-stream "^1.1.0" + +decompress-unzip@^4.0.1: + version "4.0.1" + resolved "https://registry.yarnpkg.com/decompress-unzip/-/decompress-unzip-4.0.1.tgz#deaaccdfd14aeaf85578f733ae8210f9b4848f69" + integrity sha512-1fqeluvxgnn86MOh66u8FjbtJpAFv5wgCT9Iw8rcBqQcCo5tO8eiJw7NNTrvt9n4CRBVq7CstiS922oPgyGLrw== + dependencies: + file-type "^3.8.0" + get-stream "^2.2.0" + pify "^2.3.0" + yauzl "^2.4.2" + +decompress@^4.2.0: + version "4.2.1" + resolved "https://registry.yarnpkg.com/decompress/-/decompress-4.2.1.tgz#007f55cc6a62c055afa37c07eb6a4ee1b773f118" + integrity sha512-e48kc2IjU+2Zw8cTb6VZcJQ3lgVbS4uuB1TfCHbiZIP/haNXm+SVyhu+87jts5/3ROpd82GSVCoNs/z8l4ZOaQ== + dependencies: + decompress-tar "^4.0.0" + decompress-tarbz2 "^4.0.0" + decompress-targz "^4.0.0" + decompress-unzip "^4.0.1" + graceful-fs "^4.1.10" + make-dir "^1.0.0" + pify "^2.3.0" + strip-dirs "^2.0.0" + +deepmerge@^4.3.1: + version "4.3.1" + resolved "https://registry.yarnpkg.com/deepmerge/-/deepmerge-4.3.1.tgz#44b5f2147cd3b00d4b56137685966f26fd25dd4a" + integrity sha512-3sUqbMEc77XqpdNO7FRyRog+eW3ph+GYCbj+rK+uYyRMuwsVy0rMiVtPn+QJlKFvWP/1PYpapqYn0Me2knFn+A== + +degenerator@^5.0.0: + version "5.0.1" + resolved "https://registry.yarnpkg.com/degenerator/-/degenerator-5.0.1.tgz#9403bf297c6dad9a1ece409b37db27954f91f2f5" + integrity sha512-TllpMR/t0M5sqCXfj85i4XaAzxmS5tVA16dqvdkMwGmzI+dXLXnw3J+3Vdv7VKw+ThlTMboK6i9rnZ6Nntj5CQ== + dependencies: + ast-types "^0.13.4" + escodegen "^2.1.0" + esprima "^4.0.1" + +delayed-stream@~1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/delayed-stream/-/delayed-stream-1.0.0.tgz#df3ae199acadfb7d440aaae0b29e2272b24ec619" + integrity sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ== + +delegates@^1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/delegates/-/delegates-1.0.0.tgz#84c6e159b81904fdca59a0ef44cd870d31250f9a" + integrity sha512-bd2L678uiWATM6m5Z1VzNCErI3jiGzt6HGY8OVICs40JQq/HALfbyNJmp0UDakEY4pMMaN0Ly5om/B1VI/+xfQ== + +depd@2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/depd/-/depd-2.0.0.tgz#b696163cc757560d09cf22cc8fad1571b79e76df" + integrity sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw== + +destroy@1.2.0: + version "1.2.0" + resolved "https://registry.yarnpkg.com/destroy/-/destroy-1.2.0.tgz#4803735509ad8be552934c67df614f94e66fa015" + integrity sha512-2sJGJTaXIIaR1w4iJSNoN0hnMY7Gpc/n8D4qSCJw8QqFWXf7cuAgnEHxBpweaVcPevC2l3KpjYCx3NypQQgaJg== + +detect-libc@^2.0.0: + version "2.0.2" + resolved "https://registry.yarnpkg.com/detect-libc/-/detect-libc-2.0.2.tgz#8ccf2ba9315350e1241b88d0ac3b0e1fbd99605d" + integrity sha512-UX6sGumvvqSaXgdKGUsgZWqcUyIXZ/vZTrlRT/iobiKhGL0zL4d3osHj3uqllWJK+i+sixDS/3COVEOFbupFyw== + +devtools-protocol@0.0.1203626: + version "0.0.1203626" + resolved "https://registry.yarnpkg.com/devtools-protocol/-/devtools-protocol-0.0.1203626.tgz#4366a4c81a7e0d4fd6924e9182c67f1e5941e820" + integrity sha512-nEzHZteIUZfGCZtTiS1fRpC8UZmsfD1SiyPvaUNvS13dvKf666OAm8YTi0+Ca3n1nLEyu49Cy4+dPWpaHFJk9g== + +digest-fetch@^1.3.0: + version "1.3.0" + resolved "https://registry.yarnpkg.com/digest-fetch/-/digest-fetch-1.3.0.tgz#898e69264d00012a23cf26e8a3e40320143fc661" + integrity sha512-CGJuv6iKNM7QyZlM2T3sPAdZWd/p9zQiRNS9G+9COUCwzWFTs0Xp8NF5iePx7wtvhDykReiRRrSeNb4oMmB8lA== + dependencies: + base-64 "^0.1.0" + md5 "^2.3.0" + +dingbat-to-unicode@^1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/dingbat-to-unicode/-/dingbat-to-unicode-1.0.1.tgz#5091dd673241453e6b5865e26e5a4452cdef5c83" + integrity sha512-98l0sW87ZT58pU4i61wa2OHwxbiYSbuxsCBozaVnYX2iCnr3bLM3fIes1/ej7h1YdOKuKt/MLs706TVnALA65w== + +dom-serializer@^2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/dom-serializer/-/dom-serializer-2.0.0.tgz#e41b802e1eedf9f6cae183ce5e622d789d7d8e53" + integrity sha512-wIkAryiqt/nV5EQKqQpo3SToSOV9J0DnbJqwK7Wv/Trc92zIAYZ4FlMu+JPFW1DfGFt81ZTCGgDEabffXeLyJg== + dependencies: + domelementtype "^2.3.0" + domhandler "^5.0.2" + entities "^4.2.0" + +domelementtype@^2.3.0: + version "2.3.0" + resolved "https://registry.yarnpkg.com/domelementtype/-/domelementtype-2.3.0.tgz#5c45e8e869952626331d7aab326d01daf65d589d" + integrity sha512-OLETBj6w0OsagBwdXnPdN0cnMfF9opN69co+7ZrbfPGrdpPVNBUj02spi6B1N7wChLQiPn4CSH/zJvXw56gmHw== + +domhandler@^5.0.2, domhandler@^5.0.3: + version "5.0.3" + resolved "https://registry.yarnpkg.com/domhandler/-/domhandler-5.0.3.tgz#cc385f7f751f1d1fc650c21374804254538c7d31" + integrity sha512-cgwlv/1iFQiFnU96XXgROh8xTeetsnJiDsTc7TYCLFd9+/WNkIqPTxiM/8pSd8VIrhXGTf1Ny1q1hquVqDJB5w== + dependencies: + domelementtype "^2.3.0" + +domutils@^3.0.1: + version "3.1.0" + resolved "https://registry.yarnpkg.com/domutils/-/domutils-3.1.0.tgz#c47f551278d3dc4b0b1ab8cbb42d751a6f0d824e" + integrity sha512-H78uMmQtI2AhgDJjWeQmHwJJ2bLPD3GMmO7Zja/ZZh84wkm+4ut+IUnUdRa8uCGX88DiVx1j6FRe1XfxEgjEZA== + dependencies: + dom-serializer "^2.0.0" + domelementtype "^2.3.0" + domhandler "^5.0.3" + +dotenv@^16.0.3: + version "16.3.1" + resolved "https://registry.yarnpkg.com/dotenv/-/dotenv-16.3.1.tgz#369034de7d7e5b120972693352a3bf112172cc3e" + integrity sha512-IPzF4w4/Rd94bA9imS68tZBaYyBWSCE47V1RGuMrB94iyTOIEwRmVL2x/4An+6mETpLrKJ5hQkB8W4kFAadeIQ== + +duck@^0.1.12: + version "0.1.12" + resolved "https://registry.yarnpkg.com/duck/-/duck-0.1.12.tgz#de7adf758421230b6d7aee799ce42670586b9efa" + integrity sha512-wkctla1O6VfP89gQ+J/yDesM0S7B7XLXjKGzXxMDVFg7uEn706niAtyYovKbyq1oT9YwDcly721/iUWoc8MVRg== + dependencies: + underscore "^1.13.1" + +ecdsa-sig-formatter@1.0.11, ecdsa-sig-formatter@^1.0.11: + version "1.0.11" + resolved "https://registry.yarnpkg.com/ecdsa-sig-formatter/-/ecdsa-sig-formatter-1.0.11.tgz#ae0f0fa2d85045ef14a817daa3ce9acd0489e5bf" + integrity sha512-nagl3RYrbNv6kQkeJIpt6NJZy8twLB/2vtz6yN9Z4vRKHN4/QZJIEbqohALSgwKdnksuY3k5Addp5lg8sVoVcQ== + dependencies: + safe-buffer "^5.0.1" + +ee-first@1.1.1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/ee-first/-/ee-first-1.1.1.tgz#590c61156b0ae2f4f0255732a158b266bc56b21d" + integrity sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow== + +emoji-regex@^8.0.0: + version "8.0.0" + resolved "https://registry.yarnpkg.com/emoji-regex/-/emoji-regex-8.0.0.tgz#e818fd69ce5ccfcb404594f842963bf53164cc37" + integrity sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A== + +encodeurl@~1.0.2: + version "1.0.2" + resolved "https://registry.yarnpkg.com/encodeurl/-/encodeurl-1.0.2.tgz#ad3ff4c86ec2d029322f5a02c3a9a606c95b3f59" + integrity sha512-TPJXq8JqFaVYm2CWmPvnP2Iyo4ZSM7/QKcSmuMLDObfpH5fi7RUGmd/rTDf+rut/saiDiQEeVTNgAmJEdAOx0w== + +encoding-japanese@2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/encoding-japanese/-/encoding-japanese-2.0.0.tgz#fa0226e5469e7b5b69a04fea7d5481bd1fa56936" + integrity sha512-++P0RhebUC8MJAwJOsT93dT+5oc5oPImp1HubZpAuCZ5kTLnhuuBhKHj2jJeO/Gj93idPBWmIuQ9QWMe5rX3pQ== + +end-of-stream@^1.0.0, end-of-stream@^1.1.0: + version "1.4.4" + resolved "https://registry.yarnpkg.com/end-of-stream/-/end-of-stream-1.4.4.tgz#5ae64a5f45057baf3626ec14da0ca5e4b2431eb0" + integrity sha512-+uw1inIHVPQoaVuHzRyXd21icM+cnt4CzD5rW+NC1wjOUSTOs+Te7FOv7AhN7vS9x/oIyhLP5PR1H+phQAHu5Q== + dependencies: + once "^1.4.0" + +entities@^4.2.0, entities@^4.4.0: + version "4.5.0" + resolved "https://registry.yarnpkg.com/entities/-/entities-4.5.0.tgz#5d268ea5e7113ec74c4d033b79ea5a35a488fb48" + integrity sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw== + +error-ex@^1.3.1: + version "1.3.2" + resolved "https://registry.yarnpkg.com/error-ex/-/error-ex-1.3.2.tgz#b4ac40648107fdcdcfae242f428bea8a14d4f1bf" + integrity sha512-7dFHNmqeFSEt2ZBsCriorKnn3Z2pj+fd9kmI6QoWw4//DL+icEBfc0U7qJCisqrTsKTjw4fNFy2pW9OqStD84g== + dependencies: + is-arrayish "^0.2.1" + +escalade@^3.1.1: + version "3.1.1" + resolved "https://registry.yarnpkg.com/escalade/-/escalade-3.1.1.tgz#d8cfdc7000965c5a0174b4a82eaa5c0552742e40" + integrity sha512-k0er2gUkLf8O0zKJiAhmkTnJlTvINGv7ygDNPbeIsX/TJjGJZHuh9B2UxbsaEkmlEo9MfhrSzmhIlhRlI2GXnw== + +escape-html@~1.0.3: + version "1.0.3" + resolved "https://registry.yarnpkg.com/escape-html/-/escape-html-1.0.3.tgz#0258eae4d3d0c0974de1c169188ef0051d1d1988" + integrity sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow== + +escape-string-regexp@^1.0.5: + version "1.0.5" + resolved "https://registry.yarnpkg.com/escape-string-regexp/-/escape-string-regexp-1.0.5.tgz#1b61c0562190a8dff6ae3bb2cf0200ca130b86d4" + integrity sha512-vbRorB5FUQWvla16U8R/qgaFIya2qGzwDrNmCZuYKrbdSUMG6I1ZCGQRefkRVhuOkIGVne7BQ35DSfo1qvJqFg== + +escodegen@^2.1.0: + version "2.1.0" + resolved "https://registry.yarnpkg.com/escodegen/-/escodegen-2.1.0.tgz#ba93bbb7a43986d29d6041f99f5262da773e2e17" + integrity sha512-2NlIDTwUWJN0mRPQOdtQBzbUHvdGY2P1VXSyU83Q3xKxM7WHX2Ql8dKq782Q9TgQUNOLEzEYu9bzLNj1q88I5w== + dependencies: + esprima "^4.0.1" + estraverse "^5.2.0" + esutils "^2.0.2" + optionalDependencies: + source-map "~0.6.1" + +esprima@^4.0.1: + version "4.0.1" + resolved "https://registry.yarnpkg.com/esprima/-/esprima-4.0.1.tgz#13b04cdb3e6c5d19df91ab6987a8695619b0aa71" + integrity sha512-eGuFFw7Upda+g4p+QHvnW0RyTX/SVeJBDM/gCtMARO0cLuT2HcEKnTPvhjV6aGeqrCB/sbNop0Kszm0jsaWU4A== + +estraverse@^5.2.0: + version "5.3.0" + resolved "https://registry.yarnpkg.com/estraverse/-/estraverse-5.3.0.tgz#2eea5290702f26ab8fe5370370ff86c965d21123" + integrity sha512-MMdARuVEQziNTeJD8DgMqmhwR11BRQ/cBP+pLtYdSTnf3MIO8fFeiINEbX36ZdNlfU/7A9f3gUw49B3oQsvwBA== + +esutils@^2.0.2: + version "2.0.3" + resolved "https://registry.yarnpkg.com/esutils/-/esutils-2.0.3.tgz#74d2eb4de0b8da1293711910d50775b9b710ef64" + integrity sha512-kVscqXk4OCp68SZ0dkgEKVi6/8ij300KBWTJq32P/dYeWTSwK41WyTxalN1eRmA5Z9UU/LX9D7FWSmV9SAYx6g== + +etag@~1.8.1: + version "1.8.1" + resolved "https://registry.yarnpkg.com/etag/-/etag-1.8.1.tgz#41ae2eeb65efa62268aebfea83ac7d79299b0887" + integrity sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg== + +event-target-shim@^5.0.0: + version "5.0.1" + resolved "https://registry.yarnpkg.com/event-target-shim/-/event-target-shim-5.0.1.tgz#5d4d3ebdf9583d63a5333ce2deb7480ab2b05789" + integrity sha512-i/2XbnSz/uxRCU6+NdVJgKWDTM427+MqYbkQzD321DuCQJUqOuJKIA0IM2+W2xtYHdKOmZ4dR6fExsd4SXL+WQ== + +eventemitter3@^4.0.4: + version "4.0.7" + resolved "https://registry.yarnpkg.com/eventemitter3/-/eventemitter3-4.0.7.tgz#2de9b68f6528d5644ef5c59526a1b4a07306169f" + integrity sha512-8guHBZCwKnFhYdHr2ysuRWErTwhoN2X8XELRlrRwpmfeY2jjuUN4taQMsULKUVo1K4DvZl+0pgfyoysHxvmvEw== + +expr-eval@^2.0.2: + version "2.0.2" + resolved "https://registry.yarnpkg.com/expr-eval/-/expr-eval-2.0.2.tgz#fa6f044a7b0c93fde830954eb9c5b0f7fbc7e201" + integrity sha512-4EMSHGOPSwAfBiibw3ndnP0AvjDWLsMvGOvWEZ2F96IGk0bIVdjQisOHxReSkE13mHcfbuCiXw+G4y0zv6N8Eg== + +express@^4.18.2: + version "4.18.2" + resolved "https://registry.yarnpkg.com/express/-/express-4.18.2.tgz#3fabe08296e930c796c19e3c516979386ba9fd59" + integrity sha512-5/PsL6iGPdfQ/lKM1UuielYgv3BUoJfz1aUwU9vHZ+J7gyvwdQXFEBIEIaxeGf0GIcreATNyBExtalisDbuMqQ== + dependencies: + accepts "~1.3.8" + array-flatten "1.1.1" + body-parser "1.20.1" + content-disposition "0.5.4" + content-type "~1.0.4" + cookie "0.5.0" + cookie-signature "1.0.6" + debug "2.6.9" + depd "2.0.0" + encodeurl "~1.0.2" + escape-html "~1.0.3" + etag "~1.8.1" + finalhandler "1.2.0" + fresh "0.5.2" + http-errors "2.0.0" + merge-descriptors "1.0.1" + methods "~1.1.2" + on-finished "2.4.1" + parseurl "~1.3.3" + path-to-regexp "0.1.7" + proxy-addr "~2.0.7" + qs "6.11.0" + range-parser "~1.2.1" + safe-buffer "5.2.1" + send "0.18.0" + serve-static "1.15.0" + setprototypeof "1.2.0" + statuses "2.0.1" + type-is "~1.6.18" + utils-merge "1.0.1" + vary "~1.1.2" + +extend@^3.0.2: + version "3.0.2" + resolved "https://registry.yarnpkg.com/extend/-/extend-3.0.2.tgz#f8b1136b4071fbd8eb140aff858b1019ec2915fa" + integrity sha512-fjquC59cD7CyW6urNXK0FBufkZcoiGG80wTuPujX590cB5Ttln20E2UB4S/WARVqhXffZl2LNgS+gQdPIIim/g== + +extract-zip@2.0.1, extract-zip@^2.0.1: + version "2.0.1" + resolved "https://registry.yarnpkg.com/extract-zip/-/extract-zip-2.0.1.tgz#663dca56fe46df890d5f131ef4a06d22bb8ba13a" + integrity sha512-GDhU9ntwuKyGXdZBUgTIe+vXnWj0fppUEtMDL0+idd5Sta8TGpHssn/eusA9mrPr9qNDym6SxAYZjNvCn/9RBg== + dependencies: + debug "^4.1.1" + get-stream "^5.1.0" + yauzl "^2.10.0" + optionalDependencies: + "@types/yauzl" "^2.9.1" + +fast-fifo@^1.1.0, fast-fifo@^1.2.0: + version "1.3.2" + resolved "https://registry.yarnpkg.com/fast-fifo/-/fast-fifo-1.3.2.tgz#286e31de96eb96d38a97899815740ba2a4f3640c" + integrity sha512-/d9sfos4yxzpwkDkuN7k2SqFKtYNmCTzgfEpz82x34IM9/zc8KGxQoXg1liNC/izpRM/MBdt44Nmx41ZWqk+FQ== + +fast-text-encoding@^1.0.0: + version "1.0.6" + resolved "https://registry.yarnpkg.com/fast-text-encoding/-/fast-text-encoding-1.0.6.tgz#0aa25f7f638222e3396d72bf936afcf1d42d6867" + integrity sha512-VhXlQgj9ioXCqGstD37E/HBeqEGV/qOD/kmbVG8h5xKBYvM1L3lR1Zn4555cQ8GkYbJa8aJSipLPndE1k6zK2w== + +fd-slicer@~1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/fd-slicer/-/fd-slicer-1.1.0.tgz#25c7c89cb1f9077f8891bbe61d8f390eae256f1e" + integrity sha512-cE1qsB/VwyQozZ+q1dGxR8LBYNZeofhEdUNGSMbQD3Gw2lAzX9Zb3uIU6Ebc/Fmyjo9AWWfnn0AUCHqtevs/8g== + dependencies: + pend "~1.2.0" + +file-type@^16.5.4: + version "16.5.4" + resolved "https://registry.yarnpkg.com/file-type/-/file-type-16.5.4.tgz#474fb4f704bee427681f98dd390058a172a6c2fd" + integrity sha512-/yFHK0aGjFEgDJjEKP0pWCplsPFPhwyfwevf/pVxiN0tmE4L9LmwWxWukdJSHdoCli4VgQLehjJtwQBnqmsKcw== + dependencies: + readable-web-to-node-stream "^3.0.0" + strtok3 "^6.2.4" + token-types "^4.1.1" + +file-type@^3.8.0: + version "3.9.0" + resolved "https://registry.yarnpkg.com/file-type/-/file-type-3.9.0.tgz#257a078384d1db8087bc449d107d52a52672b9e9" + integrity sha512-RLoqTXE8/vPmMuTI88DAzhMYC99I8BWv7zYP4A1puo5HIjEJ5EX48ighy4ZyKMG9EDXxBgW6e++cn7d1xuFghA== + +file-type@^5.2.0: + version "5.2.0" + resolved "https://registry.yarnpkg.com/file-type/-/file-type-5.2.0.tgz#2ddbea7c73ffe36368dfae49dc338c058c2b8ad6" + integrity sha512-Iq1nJ6D2+yIO4c8HHg4fyVb8mAJieo1Oloy1mLLaB2PvezNedhBVm+QU7g0qM42aiMbRXTxKKwGD17rjKNJYVQ== + +file-type@^6.1.0: + version "6.2.0" + resolved "https://registry.yarnpkg.com/file-type/-/file-type-6.2.0.tgz#e50cd75d356ffed4e306dc4f5bcf52a79903a919" + integrity sha512-YPcTBDV+2Tm0VqjybVd32MHdlEGAtuxS3VAYsumFokDSMG+ROT5wawGlnHDoz7bfMcMDt9hxuXvXwoKUx2fkOg== + +fill-range@^7.0.1: + version "7.0.1" + resolved "https://registry.yarnpkg.com/fill-range/-/fill-range-7.0.1.tgz#1919a6a7c75fe38b2c7c77e5198535da9acdda40" + integrity sha512-qOo9F+dMUmC2Lcb4BbVvnKJxTPjCm+RRpe4gDuGrzkL7mEVl/djYSu2OdQ2Pa302N4oqkSg9ir6jaLWJ2USVpQ== + dependencies: + to-regex-range "^5.0.1" + +finalhandler@1.2.0: + version "1.2.0" + resolved "https://registry.yarnpkg.com/finalhandler/-/finalhandler-1.2.0.tgz#7d23fe5731b207b4640e4fcd00aec1f9207a7b32" + integrity sha512-5uXcUVftlQMFnWC9qu/svkWv3GTd2PfUhK/3PLkYNAe7FbqJMt3515HaxE6eRL74GdsriiwujiawdaB1BpEISg== + dependencies: + debug "2.6.9" + encodeurl "~1.0.2" + escape-html "~1.0.3" + on-finished "2.4.1" + parseurl "~1.3.3" + statuses "2.0.1" + unpipe "~1.0.0" + +flat@^5.0.2: + version "5.0.2" + resolved "https://registry.yarnpkg.com/flat/-/flat-5.0.2.tgz#8ca6fe332069ffa9d324c327198c598259ceb241" + integrity sha512-b6suED+5/3rTpUBdG1gupIl8MPFCAMA0QXwmljLhvCUKcUvdE4gWky9zpuGCcXHOsz4J9wPGNWq6OKpmIzz3hQ== + +form-data-encoder@1.7.2: + version "1.7.2" + resolved "https://registry.yarnpkg.com/form-data-encoder/-/form-data-encoder-1.7.2.tgz#1f1ae3dccf58ed4690b86d87e4f57c654fbab040" + integrity sha512-qfqtYan3rxrnCk1VYaA4H+Ms9xdpPqvLZa6xmMgFvhO32x7/3J/ExcTd6qpxM0vH2GdMI+poehyBZvqfMTto8A== + +form-data@^4.0.0: + version "4.0.0" + resolved "https://registry.yarnpkg.com/form-data/-/form-data-4.0.0.tgz#93919daeaf361ee529584b9b31664dc12c9fa452" + integrity sha512-ETEklSGi5t0QMZuiXoA/Q6vcnxcLQP5vdugSpuAyi6SVGi2clPPp+xgEhuMaHC+zGgn31Kd235W35f7Hykkaww== + dependencies: + asynckit "^0.4.0" + combined-stream "^1.0.8" + mime-types "^2.1.12" + +formdata-node@^4.3.2: + version "4.4.1" + resolved "https://registry.yarnpkg.com/formdata-node/-/formdata-node-4.4.1.tgz#23f6a5cb9cb55315912cbec4ff7b0f59bbd191e2" + integrity sha512-0iirZp3uVDjVGt9p49aTaqjk84TrglENEDuqfdlZQ1roC9CWlPk6Avf8EEnZNcAqPonwkG35x4n3ww/1THYAeQ== + dependencies: + node-domexception "1.0.0" + web-streams-polyfill "4.0.0-beta.3" + +forwarded@0.2.0: + version "0.2.0" + resolved "https://registry.yarnpkg.com/forwarded/-/forwarded-0.2.0.tgz#2269936428aad4c15c7ebe9779a84bf0b2a81811" + integrity sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow== + +fresh@0.5.2: + version "0.5.2" + resolved "https://registry.yarnpkg.com/fresh/-/fresh-0.5.2.tgz#3d8cadd90d976569fa835ab1f8e4b23a105605a7" + integrity sha512-zJ2mQYM18rEFOudeV4GShTGIQ7RbzA7ozbU9I/XBpm7kqgMywgmylMwXHxZJmkVoYkna9d2pVXVXPdYTP9ej8Q== + +fs-constants@^1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/fs-constants/-/fs-constants-1.0.0.tgz#6be0de9be998ce16af8afc24497b9ee9b7ccd9ad" + integrity sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow== + +fs-extra@^8.1.0: + version "8.1.0" + resolved "https://registry.yarnpkg.com/fs-extra/-/fs-extra-8.1.0.tgz#49d43c45a88cd9677668cb7be1b46efdb8d2e1c0" + integrity sha512-yhlQgA6mnOJUKOsRUFsgJdQCvkKhcz8tlZG5HBQfReYZy46OwLcY+Zia0mtdHsOo9y/hP+CxMN0TU9QxoOtG4g== + dependencies: + graceful-fs "^4.2.0" + jsonfile "^4.0.0" + universalify "^0.1.0" + +fs-minipass@^2.0.0: + version "2.1.0" + resolved "https://registry.yarnpkg.com/fs-minipass/-/fs-minipass-2.1.0.tgz#7f5036fdbf12c63c169190cbe4199c852271f9fb" + integrity sha512-V/JgOLFCS+R6Vcq0slCuaeWEdNC3ouDlJMNIsacH2VtALiu9mV4LPrHc5cDl8k5aw6J8jwgWWpiTo5RYhmIzvg== + dependencies: + minipass "^3.0.0" + +fs.realpath@^1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/fs.realpath/-/fs.realpath-1.0.0.tgz#1504ad2523158caa40db4a2787cb01411994ea4f" + integrity sha512-OO0pH2lK6a0hZnAdau5ItzHPI6pUlvI7jMVnxUQRtw4owF2wk8lOSabtGDCTP4Ggrg2MbGnWO9X8K1t4+fGMDw== + +fsevents@~2.3.2: + version "2.3.2" + resolved "https://registry.yarnpkg.com/fsevents/-/fsevents-2.3.2.tgz#8a526f78b8fdf4623b709e0b975c52c24c02fd1a" + integrity sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA== + +function-bind@^1.1.1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/function-bind/-/function-bind-1.1.1.tgz#a56899d3ea3c9bab874bb9773b7c5ede92f4895d" + integrity sha512-yIovAzMX49sF8Yl58fSCWJ5svSLuaibPxXQJFLmBObTuCr0Mf1KiPopGM9NiFjiYBCbfaa2Fh6breQ6ANVTI0A== + +gauge@^3.0.0: + version "3.0.2" + resolved "https://registry.yarnpkg.com/gauge/-/gauge-3.0.2.tgz#03bf4441c044383908bcfa0656ad91803259b395" + integrity sha512-+5J6MS/5XksCuXq++uFRsnUd7Ovu1XenbeuIuNRJxYWjgQbPuFhT14lAvsWfqfAmnwluf1OwMjz39HjfLPci0Q== + dependencies: + aproba "^1.0.3 || ^2.0.0" + color-support "^1.1.2" + console-control-strings "^1.0.0" + has-unicode "^2.0.1" + object-assign "^4.1.1" + signal-exit "^3.0.0" + string-width "^4.2.3" + strip-ansi "^6.0.1" + wide-align "^1.1.2" + +gaxios@^5.0.0, gaxios@^5.0.1: + version "5.1.3" + resolved "https://registry.yarnpkg.com/gaxios/-/gaxios-5.1.3.tgz#f7fa92da0fe197c846441e5ead2573d4979e9013" + integrity sha512-95hVgBRgEIRQQQHIbnxBXeHbW4TqFk4ZDJW7wmVtvYar72FdhRIo1UGOLS2eRAKCPEdPBWu+M7+A33D9CdX9rA== + dependencies: + extend "^3.0.2" + https-proxy-agent "^5.0.0" + is-stream "^2.0.0" + node-fetch "^2.6.9" + +gcp-metadata@^5.3.0: + version "5.3.0" + resolved "https://registry.yarnpkg.com/gcp-metadata/-/gcp-metadata-5.3.0.tgz#6f45eb473d0cb47d15001476b48b663744d25408" + integrity sha512-FNTkdNEnBdlqF2oatizolQqNANMrcqJt6AAYt99B3y1aLLC8Hc5IOBb+ZnnzllodEEf6xMBp6wRcBbc16fa65w== + dependencies: + gaxios "^5.0.0" + json-bigint "^1.0.0" + +get-caller-file@^2.0.5: + version "2.0.5" + resolved "https://registry.yarnpkg.com/get-caller-file/-/get-caller-file-2.0.5.tgz#4f94412a82db32f36e3b0b9741f8a97feb031f7e" + integrity sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg== + +get-intrinsic@^1.0.2: + version "1.2.1" + resolved "https://registry.yarnpkg.com/get-intrinsic/-/get-intrinsic-1.2.1.tgz#d295644fed4505fc9cde952c37ee12b477a83d82" + integrity sha512-2DcsyfABl+gVHEfCOaTrWgyt+tb6MSEGmKq+kI5HwLbIYgjgmMcV8KQ41uaKz1xxUcn9tJtgFbQUEVcEbd0FYw== + dependencies: + function-bind "^1.1.1" + has "^1.0.3" + has-proto "^1.0.1" + has-symbols "^1.0.3" + +get-stream@^2.2.0: + version "2.3.1" + resolved "https://registry.yarnpkg.com/get-stream/-/get-stream-2.3.1.tgz#5f38f93f346009666ee0150a054167f91bdd95de" + integrity sha512-AUGhbbemXxrZJRD5cDvKtQxLuYaIbNtDTK8YqupCI393Q2KSTreEsLUN3ZxAWFGiKTzL6nKuzfcIvieflUX9qA== + dependencies: + object-assign "^4.0.1" + pinkie-promise "^2.0.0" + +get-stream@^5.1.0: + version "5.2.0" + resolved "https://registry.yarnpkg.com/get-stream/-/get-stream-5.2.0.tgz#4966a1795ee5ace65e706c4b7beb71257d6e22d3" + integrity sha512-nBF+F1rAZVCu/p7rjzgA+Yb4lfYXrpl7a6VmJrU8wF9I1CKvP/QwPNZHnOlwbTkY6dvtFIzFMSyQXbLoTQPRpA== + dependencies: + pump "^3.0.0" + +get-uri@^6.0.1: + version "6.0.2" + resolved "https://registry.yarnpkg.com/get-uri/-/get-uri-6.0.2.tgz#e019521646f4a8ff6d291fbaea2c46da204bb75b" + integrity sha512-5KLucCJobh8vBY1K07EFV4+cPZH3mrV9YeAruUseCQKHB58SGjjT2l9/eA9LD082IiuMjSlFJEcdJ27TXvbZNw== + dependencies: + basic-ftp "^5.0.2" + data-uri-to-buffer "^6.0.0" + debug "^4.3.4" + fs-extra "^8.1.0" + +glob-parent@~5.1.2: + version "5.1.2" + resolved "https://registry.yarnpkg.com/glob-parent/-/glob-parent-5.1.2.tgz#869832c58034fe68a4093c17dc15e8340d8401c4" + integrity sha512-AOIgSQCepiJYwP3ARnGx+5VnTu2HBYdzbGP45eLw1vr3zB3vZLeyed1sC9hnbcOc9/SrMyM5RPQrkGz4aS9Zow== + dependencies: + is-glob "^4.0.1" + +glob@^7.1.3: + version "7.2.3" + resolved "https://registry.yarnpkg.com/glob/-/glob-7.2.3.tgz#b8df0fb802bbfa8e89bd1d938b4e16578ed44f2b" + integrity sha512-nFR0zLpU2YCaRxwoCJvL6UvCH2JFyFVIvwTLsIf21AuHlMskA1hhTdk+LlYJtOlYt9v6dvszD2BGRqBL+iQK9Q== + dependencies: + fs.realpath "^1.0.0" + inflight "^1.0.4" + inherits "2" + minimatch "^3.1.1" + once "^1.3.0" + path-is-absolute "^1.0.0" + +google-auth-library@^8.0.2: + version "8.9.0" + resolved "https://registry.yarnpkg.com/google-auth-library/-/google-auth-library-8.9.0.tgz#15a271eb2ec35d43b81deb72211bd61b1ef14dd0" + integrity sha512-f7aQCJODJFmYWN6PeNKzgvy9LI2tYmXnzpNDHEjG5sDNPgGb2FXQyTBnXeSH+PAtpKESFD+LmHw3Ox3mN7e1Fg== + dependencies: + arrify "^2.0.0" + base64-js "^1.3.0" + ecdsa-sig-formatter "^1.0.11" + fast-text-encoding "^1.0.0" + gaxios "^5.0.0" + gcp-metadata "^5.3.0" + gtoken "^6.1.0" + jws "^4.0.0" + lru-cache "^6.0.0" + +google-p12-pem@^4.0.0: + version "4.0.1" + resolved "https://registry.yarnpkg.com/google-p12-pem/-/google-p12-pem-4.0.1.tgz#82841798253c65b7dc2a4e5fe9df141db670172a" + integrity sha512-WPkN4yGtz05WZ5EhtlxNDWPhC4JIic6G8ePitwUWy4l+XPVYec+a0j0Ts47PDtW59y3RwAhUd9/h9ZZ63px6RQ== + dependencies: + node-forge "^1.3.1" + +googleapis-common@^6.0.3: + version "6.0.4" + resolved "https://registry.yarnpkg.com/googleapis-common/-/googleapis-common-6.0.4.tgz#bd968bef2a478bcd3db51b27655502a11eaf8bf4" + integrity sha512-m4ErxGE8unR1z0VajT6AYk3s6a9gIMM6EkDZfkPnES8joeOlEtFEJeF8IyZkb0tjPXkktUfYrE4b3Li1DNyOwA== + dependencies: + extend "^3.0.2" + gaxios "^5.0.1" + google-auth-library "^8.0.2" + qs "^6.7.0" + url-template "^2.0.8" + uuid "^9.0.0" + +graceful-fs@^4.1.10, graceful-fs@^4.1.6, graceful-fs@^4.2.0: + version "4.2.11" + resolved "https://registry.yarnpkg.com/graceful-fs/-/graceful-fs-4.2.11.tgz#4183e4e8bf08bb6e05bbb2f7d2e0c8f712ca40e3" + integrity sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ== + +gtoken@^6.1.0: + version "6.1.2" + resolved "https://registry.yarnpkg.com/gtoken/-/gtoken-6.1.2.tgz#aeb7bdb019ff4c3ba3ac100bbe7b6e74dce0e8bc" + integrity sha512-4ccGpzz7YAr7lxrT2neugmXQ3hP9ho2gcaityLVkiUecAiwiy60Ii8gRbZeOsXV19fYaRjgBSshs8kXw+NKCPQ== + dependencies: + gaxios "^5.0.1" + google-p12-pem "^4.0.0" + jws "^4.0.0" + +has-flag@^3.0.0: + version "3.0.0" + resolved "https://registry.yarnpkg.com/has-flag/-/has-flag-3.0.0.tgz#b5d454dc2199ae225699f3467e5a07f3b955bafd" + integrity sha512-sKJf1+ceQBr4SMkvQnBDNDtf4TXpVhVGateu0t918bl30FnbE2m4vNLX+VWe/dpjlb+HugGYzW7uQXH98HPEYw== + +has-proto@^1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/has-proto/-/has-proto-1.0.1.tgz#1885c1305538958aff469fef37937c22795408e0" + integrity sha512-7qE+iP+O+bgF9clE5+UoBFzE65mlBiVj3tKCrlNQ0Ogwm0BjpT/gK4SlLYDMybDh5I3TCTKnPPa0oMG7JDYrhg== + +has-symbols@^1.0.3: + version "1.0.3" + resolved "https://registry.yarnpkg.com/has-symbols/-/has-symbols-1.0.3.tgz#bb7b2c4349251dce87b125f7bdf874aa7c8b39f8" + integrity sha512-l3LCuF6MgDNwTDKkdYGEihYjt5pRPbEg46rtlmnSPlUbgmB8LOIrKJbYYFBSbnPaJexMKtiPO8hmeRjRz2Td+A== + +has-unicode@^2.0.1: + version "2.0.1" + resolved "https://registry.yarnpkg.com/has-unicode/-/has-unicode-2.0.1.tgz#e0e6fe6a28cf51138855e086d1691e771de2a8b9" + integrity sha512-8Rf9Y83NBReMnx0gFzA8JImQACstCYWUplepDa9xprwwtmgEZUF0h/i5xSA625zB/I37EtrswSST6OXxwaaIJQ== + +has@^1.0.3: + version "1.0.3" + resolved "https://registry.yarnpkg.com/has/-/has-1.0.3.tgz#722d7cbfc1f6aa8241f16dd814e011e1f41e8796" + integrity sha512-f2dvO0VU6Oej7RkWJGrehjbzMAjFp5/VKPp5tTpWIV4JHHZK1/BxbFRtf/siA2SWTe09caDmVtYYzWEIbBS4zw== + dependencies: + function-bind "^1.1.1" + +he@1.2.0: + version "1.2.0" + resolved "https://registry.yarnpkg.com/he/-/he-1.2.0.tgz#84ae65fa7eafb165fddb61566ae14baf05664f0f" + integrity sha512-F/1DnUGPopORZi0ni+CvrCgHQ5FyEAHRLSApuYWMmrbSwoN2Mn/7k+Gl38gJnR7yyDZk6WLXwiGod1JOWNDKGw== + +html-to-text@9.0.5: + version "9.0.5" + resolved "https://registry.yarnpkg.com/html-to-text/-/html-to-text-9.0.5.tgz#6149a0f618ae7a0db8085dca9bbf96d32bb8368d" + integrity sha512-qY60FjREgVZL03vJU6IfMV4GDjGBIoOyvuFdpBDIX9yTlDw0TjxVBQp+P8NvpdIXNJvfWBTNul7fsAQJq2FNpg== + dependencies: + "@selderee/plugin-htmlparser2" "^0.11.0" + deepmerge "^4.3.1" + dom-serializer "^2.0.0" + htmlparser2 "^8.0.2" + selderee "^0.11.0" + +htmlparser2@^8.0.2: + version "8.0.2" + resolved "https://registry.yarnpkg.com/htmlparser2/-/htmlparser2-8.0.2.tgz#f002151705b383e62433b5cf466f5b716edaec21" + integrity sha512-GYdjWKDkbRLkZ5geuHs5NY1puJ+PXwP7+fHPRz06Eirsb9ugf6d8kkXav6ADhcODhFFPMIXyxkxSuMf3D6NCFA== + dependencies: + domelementtype "^2.3.0" + domhandler "^5.0.3" + domutils "^3.0.1" + entities "^4.4.0" + +http-errors@2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/http-errors/-/http-errors-2.0.0.tgz#b7774a1486ef73cf7667ac9ae0858c012c57b9d3" + integrity sha512-FtwrG/euBzaEjYeRqOgly7G0qviiXoJWnvEH2Z1plBdXgbyjv34pHTSb9zoeHMyDy33+DWy5Wt9Wo+TURtOYSQ== + dependencies: + depd "2.0.0" + inherits "2.0.4" + setprototypeof "1.2.0" + statuses "2.0.1" + toidentifier "1.0.1" + +http-proxy-agent@^7.0.0: + version "7.0.0" + resolved "https://registry.yarnpkg.com/http-proxy-agent/-/http-proxy-agent-7.0.0.tgz#e9096c5afd071a3fce56e6252bb321583c124673" + integrity sha512-+ZT+iBxVUQ1asugqnD6oWoRiS25AkjNfG085dKJGtGxkdwLQrMKU5wJr2bOOFAXzKcTuqq+7fZlTMgG3SRfIYQ== + dependencies: + agent-base "^7.1.0" + debug "^4.3.4" + +https-proxy-agent@^5.0.0: + version "5.0.1" + resolved "https://registry.yarnpkg.com/https-proxy-agent/-/https-proxy-agent-5.0.1.tgz#c59ef224a04fe8b754f3db0063a25ea30d0005d6" + integrity sha512-dFcAjpTQFgoLMzC2VwU+C/CbS7uRL0lWmxDITmqm7C+7F0Odmj6s9l6alZc6AELXhrnggM2CeWSXHGOdX2YtwA== + dependencies: + agent-base "6" + debug "4" + +https-proxy-agent@^7.0.2: + version "7.0.2" + resolved "https://registry.yarnpkg.com/https-proxy-agent/-/https-proxy-agent-7.0.2.tgz#e2645b846b90e96c6e6f347fb5b2e41f1590b09b" + integrity sha512-NmLNjm6ucYwtcUmL7JQC1ZQ57LmHP4lT15FQ8D61nak1rO6DH+fz5qNK2Ap5UN4ZapYICE3/0KodcLYSPsPbaA== + dependencies: + agent-base "^7.0.2" + debug "4" + +humanize-duration@^3.25.1: + version "3.31.0" + resolved "https://registry.yarnpkg.com/humanize-duration/-/humanize-duration-3.31.0.tgz#a0384d22555024cd17e6e9f8561540d37756bf4c" + integrity sha512-fRrehgBG26NNZysRlTq1S+HPtDpp3u+Jzdc/d5A4cEzOD86YLAkDaJyJg8krSdCi7CJ+s7ht3fwRj8Dl+Btd0w== + +humanize-ms@^1.2.1: + version "1.2.1" + resolved "https://registry.yarnpkg.com/humanize-ms/-/humanize-ms-1.2.1.tgz#c46e3159a293f6b896da29316d8b6fe8bb79bbed" + integrity sha512-Fl70vYtsAFb/C06PTS9dZBo7ihau+Tu/DNCk/OyHhea07S+aeMWpFFkUaXRa8fI+ScZbEI8dfSxwY7gxZ9SAVQ== + dependencies: + ms "^2.0.0" + +iconv-lite@0.4.24: + version "0.4.24" + resolved "https://registry.yarnpkg.com/iconv-lite/-/iconv-lite-0.4.24.tgz#2022b4b25fbddc21d2f524974a474aafe733908b" + integrity sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA== + dependencies: + safer-buffer ">= 2.1.2 < 3" + +iconv-lite@0.6.3, iconv-lite@^0.6.3: + version "0.6.3" + resolved "https://registry.yarnpkg.com/iconv-lite/-/iconv-lite-0.6.3.tgz#a52f80bf38da1952eb5c681790719871a1a72501" + integrity sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw== + dependencies: + safer-buffer ">= 2.1.2 < 3.0.0" + +ieee754@^1.1.13, ieee754@^1.2.1: + version "1.2.1" + resolved "https://registry.yarnpkg.com/ieee754/-/ieee754-1.2.1.tgz#8eb7a10a63fff25d15a57b001586d177d1b0d352" + integrity sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA== + +ignore-by-default@^1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/ignore-by-default/-/ignore-by-default-1.0.1.tgz#48ca6d72f6c6a3af00a9ad4ae6876be3889e2b09" + integrity sha512-Ius2VYcGNk7T90CppJqcIkS5ooHUZyIQK+ClZfMfMNFEF9VSE73Fq+906u/CWu92x4gzZMWOwfFYckPObzdEbA== + +immediate@~3.0.5: + version "3.0.6" + resolved "https://registry.yarnpkg.com/immediate/-/immediate-3.0.6.tgz#9db1dbd0faf8de6fbe0f5dd5e56bb606280de69b" + integrity sha512-XXOFtyqDjNDAQxVfYxuF7g9Il/IbWmmlQg2MYKOH8ExIT1qg6xc4zyS3HaEEATgs1btfzxq15ciUiY7gjSXRGQ== + +import-fresh@^3.3.0: + version "3.3.0" + resolved "https://registry.yarnpkg.com/import-fresh/-/import-fresh-3.3.0.tgz#37162c25fcb9ebaa2e6e53d5b4d88ce17d9e0c2b" + integrity sha512-veYYhQa+D1QBKznvhUHxb8faxlrwUnxseDAbAp457E0wLNio2bOSKnjYDhMj+YiAq61xrMGhQk9iXVk5FzgQMw== + dependencies: + parent-module "^1.0.0" + resolve-from "^4.0.0" + +inflight@^1.0.4: + version "1.0.6" + resolved "https://registry.yarnpkg.com/inflight/-/inflight-1.0.6.tgz#49bd6331d7d02d0c09bc910a1075ba8165b56df9" + integrity sha512-k92I/b08q4wvFscXCLvqfsHCrjrF7yiXsQuIVvVE7N82W3+aqpzuUdBbfhWcy/FZR3/4IgflMgKLOsvPDrGCJA== + dependencies: + once "^1.3.0" + wrappy "1" + +inherits@2, inherits@2.0.4, inherits@^2.0.3, inherits@~2.0.3: + version "2.0.4" + resolved "https://registry.yarnpkg.com/inherits/-/inherits-2.0.4.tgz#0fa2c64f932917c3433a0ded55363aae37416b7c" + integrity sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ== + +ip@^1.1.8: + version "1.1.8" + resolved "https://registry.yarnpkg.com/ip/-/ip-1.1.8.tgz#ae05948f6b075435ed3307acce04629da8cdbf48" + integrity sha512-PuExPYUiu6qMBQb4l06ecm6T6ujzhmh+MeJcW9wa89PoAz5pvd4zPgN5WJV104mb6S2T1AwNIAaB70JNrLQWhg== + +ip@^2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/ip/-/ip-2.0.0.tgz#4cf4ab182fee2314c75ede1276f8c80b479936da" + integrity sha512-WKa+XuLG1A1R0UWhl2+1XQSi+fZWMsYKffMZTTYsiZaUD8k2yDAj5atimTUD2TZkyCkNEeYE5NhFZmupOGtjYQ== + +ipaddr.js@1.9.1: + version "1.9.1" + resolved "https://registry.yarnpkg.com/ipaddr.js/-/ipaddr.js-1.9.1.tgz#bff38543eeb8984825079ff3a2a8e6cbd46781b3" + integrity sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g== + +is-any-array@^2.0.0: + version "2.0.1" + resolved "https://registry.yarnpkg.com/is-any-array/-/is-any-array-2.0.1.tgz#9233242a9c098220290aa2ec28f82ca7fa79899e" + integrity sha512-UtilS7hLRu++wb/WBAw9bNuP1Eg04Ivn1vERJck8zJthEvXCBEBpGR/33u/xLKWEQf95803oalHrVDptcAvFdQ== + +is-arrayish@^0.2.1: + version "0.2.1" + resolved "https://registry.yarnpkg.com/is-arrayish/-/is-arrayish-0.2.1.tgz#77c99840527aa8ecb1a8ba697b80645a7a926a9d" + integrity sha512-zz06S8t0ozoDXMG+ube26zeCTNXcKIPJZJi8hBrF4idCLms4CG9QtK7qBl1boi5ODzFpjswb5JPmHCbMpjaYzg== + +is-binary-path@~2.1.0: + version "2.1.0" + resolved "https://registry.yarnpkg.com/is-binary-path/-/is-binary-path-2.1.0.tgz#ea1f7f3b80f064236e83470f86c09c254fb45b09" + integrity sha512-ZMERYes6pDydyuGidse7OsHxtbI7WVeUEozgR/g7rd0xUimYNlvZRE/K2MgZTjWy725IfelLeVcEM97mmtRGXw== + dependencies: + binary-extensions "^2.0.0" + +is-buffer@~1.1.6: + version "1.1.6" + resolved "https://registry.yarnpkg.com/is-buffer/-/is-buffer-1.1.6.tgz#efaa2ea9daa0d7ab2ea13a97b2b8ad51fefbe8be" + integrity sha512-NcdALwpXkTm5Zvvbk7owOUSvVvBKDgKP5/ewfXEznmQFfs4ZRmanOeKBTjRVjka3QFoN6XJ+9F3USqfHqTaU5w== + +is-extglob@^2.1.1: + version "2.1.1" + resolved "https://registry.yarnpkg.com/is-extglob/-/is-extglob-2.1.1.tgz#a88c02535791f02ed37c76a1b9ea9773c833f8c2" + integrity sha512-SbKbANkN603Vi4jEZv49LeVJMn4yGwsbzZworEoyEiutsN3nJYdbO36zfhGJ6QEDpOZIFkDtnq5JRxmvl3jsoQ== + +is-fullwidth-code-point@^3.0.0: + version "3.0.0" + resolved "https://registry.yarnpkg.com/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz#f116f8064fe90b3f7844a38997c0b75051269f1d" + integrity sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg== + +is-glob@^4.0.1, is-glob@~4.0.1: + version "4.0.3" + resolved "https://registry.yarnpkg.com/is-glob/-/is-glob-4.0.3.tgz#64f61e42cbbb2eec2071a9dac0b28ba1e65d5084" + integrity sha512-xelSayHH36ZgE7ZWhli7pW34hNbNl8Ojv5KVmkJD4hBdD3th8Tfk9vYasLM+mXWOZhFkgZfxhLSnrwRr4elSSg== + dependencies: + is-extglob "^2.1.1" + +is-natural-number@^4.0.1: + version "4.0.1" + resolved "https://registry.yarnpkg.com/is-natural-number/-/is-natural-number-4.0.1.tgz#ab9d76e1db4ced51e35de0c72ebecf09f734cde8" + integrity sha512-Y4LTamMe0DDQIIAlaer9eKebAlDSV6huy+TWhJVPlzZh2o4tRP5SQWFlLn5N0To4mDD22/qdOq+veo1cSISLgQ== + +is-number@^7.0.0: + version "7.0.0" + resolved "https://registry.yarnpkg.com/is-number/-/is-number-7.0.0.tgz#7535345b896734d5f80c4d06c50955527a14f12b" + integrity sha512-41Cifkg6e8TylSpdtTpeLVMqvSBEVzTttHvERD741+pnZ8ANv0004MRL43QKPDlK9cGvNp6NZWZUBlbGXYxxng== + +is-stream@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/is-stream/-/is-stream-1.1.0.tgz#12d4a3dd4e68e0b79ceb8dbc84173ae80d91ca44" + integrity sha512-uQPm8kcs47jx38atAcWTVxyltQYoPT68y9aWYdV6yWXSyW8mzSat0TL6CiWdZeCdF3KrAvpVtnHbTv4RN+rqdQ== + +is-stream@^2.0.0: + version "2.0.1" + resolved "https://registry.yarnpkg.com/is-stream/-/is-stream-2.0.1.tgz#fac1e3d53b97ad5a9d0ae9cef2389f5810a5c077" + integrity sha512-hFoiJiTl63nn+kstHGBtewWSKnQLpyb155KHheA1l39uvtO9nWIop1p3udqPcUd/xbF1VLMO4n7OI6p7RbngDg== + +isarray@~1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/isarray/-/isarray-1.0.0.tgz#bb935d48582cba168c06834957a54a3e07124f11" + integrity sha512-VLghIWNM6ELQzo7zwmcg0NmTVyWKYjvIeM83yjp0wRDTmUnrM678fQbcKBo6n2CJEF0szoG//ytg+TKla89ALQ== + +js-tiktoken@^1.0.7: + version "1.0.7" + resolved "https://registry.yarnpkg.com/js-tiktoken/-/js-tiktoken-1.0.7.tgz#56933fcd2093e8304060dfde3071bda91812e6f5" + integrity sha512-biba8u/clw7iesNEWLOLwrNGoBP2lA+hTaBLs/D45pJdUPFXyxD6nhcDVtADChghv4GgyAiMKYMiRx7x6h7Biw== + dependencies: + base64-js "^1.5.1" + +js-tiktoken@^1.0.8: + version "1.0.8" + resolved "https://registry.yarnpkg.com/js-tiktoken/-/js-tiktoken-1.0.8.tgz#21ab8ae222e71226b2ef0d2f4b507fb10d66a114" + integrity sha512-r7XK3E9/I+SOrbAGqb39pyO/rHAS1diAOSRAvaaLfHgXjkUSK9AiSd+r84Vn2f/GvXJYRAxKj8NHrUvqlaH5qg== + dependencies: + base64-js "^1.5.1" + +js-tokens@^4.0.0: + version "4.0.0" + resolved "https://registry.yarnpkg.com/js-tokens/-/js-tokens-4.0.0.tgz#19203fb59991df98e3a287050d4647cdeaf32499" + integrity sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ== + +js-yaml@^4.1.0: + version "4.1.0" + resolved "https://registry.yarnpkg.com/js-yaml/-/js-yaml-4.1.0.tgz#c1fb65f8f5017901cdd2c951864ba18458a10602" + integrity sha512-wpxZs9NoxZaJESJGIZTyDEaYpl0FKSA+FB9aJiyemKhMwkxQg63h4T1KJgUGHpTqPDNRcmmYLugrRjJlBtWvRA== + dependencies: + argparse "^2.0.1" + +json-bigint@^1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/json-bigint/-/json-bigint-1.0.0.tgz#ae547823ac0cad8398667f8cd9ef4730f5b01ff1" + integrity sha512-SiPv/8VpZuWbvLSMtTDU8hEfrZWg/mH/nV/b4o0CYbSxu1UIQPLdwKOCIyLQX+VIPO5vrLX3i8qtqFyhdPSUSQ== + dependencies: + bignumber.js "^9.0.0" + +json-parse-even-better-errors@^2.3.0: + version "2.3.1" + resolved "https://registry.yarnpkg.com/json-parse-even-better-errors/-/json-parse-even-better-errors-2.3.1.tgz#7c47805a94319928e05777405dc12e1f7a4ee02d" + integrity sha512-xyFwyhro/JEof6Ghe2iz2NcXoj2sloNsWr/XsERDK/oiPCfaNhl5ONfp+jQdAZRQQ0IJWNzH9zIZF7li91kh2w== + +jsonfile@^4.0.0: + version "4.0.0" + resolved "https://registry.yarnpkg.com/jsonfile/-/jsonfile-4.0.0.tgz#8771aae0799b64076b76640fca058f9c10e33ecb" + integrity sha512-m6F1R3z8jjlf2imQHS2Qez5sjKWQzbuuhuJ/FKYFRZvPE3PuHcSMVZzfsLhGVOkfd20obL5SWEBew5ShlquNxg== + optionalDependencies: + graceful-fs "^4.1.6" + +jsonpointer@^5.0.1: + version "5.0.1" + resolved "https://registry.yarnpkg.com/jsonpointer/-/jsonpointer-5.0.1.tgz#2110e0af0900fd37467b5907ecd13a7884a1b559" + integrity sha512-p/nXbhSEcu3pZRdkW1OfJhpsVtW1gd4Wa1fnQc9YLiTfAjn0312eMKimbdIQzuZl9aa9xUGaRlP9T/CJE/ditQ== + +jszip@^3.7.1: + version "3.10.1" + resolved "https://registry.yarnpkg.com/jszip/-/jszip-3.10.1.tgz#34aee70eb18ea1faec2f589208a157d1feb091c2" + integrity sha512-xXDvecyTpGLrqFrvkrUSoxxfJI5AH7U8zxxtVclpsUtMCq4JQ290LY8AW5c7Ggnr/Y/oK+bQMbqK2qmtk3pN4g== + dependencies: + lie "~3.3.0" + pako "~1.0.2" + readable-stream "~2.3.6" + setimmediate "^1.0.5" + +jwa@^2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/jwa/-/jwa-2.0.0.tgz#a7e9c3f29dae94027ebcaf49975c9345593410fc" + integrity sha512-jrZ2Qx916EA+fq9cEAeCROWPTfCwi1IVHqT2tapuqLEVVDKFDENFw1oL+MwrTvH6msKxsd1YTDVw6uKEcsrLEA== + dependencies: + buffer-equal-constant-time "1.0.1" + ecdsa-sig-formatter "1.0.11" + safe-buffer "^5.0.1" + +jws@^4.0.0: + version "4.0.0" + resolved "https://registry.yarnpkg.com/jws/-/jws-4.0.0.tgz#2d4e8cf6a318ffaa12615e9dec7e86e6c97310f4" + integrity sha512-KDncfTmOZoOMTFG4mBlG0qUIOlc03fmzH+ru6RgYVZhPkyiy/92Owlt/8UEN+a4TXR1FQetfIpJE8ApdvdVxTg== + dependencies: + jwa "^2.0.0" + safe-buffer "^5.0.1" + +langchain@0.0.201: + version "0.0.201" + resolved "https://registry.yarnpkg.com/langchain/-/langchain-0.0.201.tgz#dd4aa2d3a57cf40f3a7d122cc8e42c4d8dce27c5" + integrity sha512-zbmKpH3zbCgzbBxBq/YBXKS9xTLqUzQoBqW2j7ctiVYwNq+wIGHef8YuVnDXlli6Wx0+3T2b4kTI8506xkriMg== + dependencies: + "@anthropic-ai/sdk" "^0.9.1" + "@langchain/core" "~0.0.8" + binary-extensions "^2.2.0" + expr-eval "^2.0.2" + flat "^5.0.2" + js-tiktoken "^1.0.7" + js-yaml "^4.1.0" + jsonpointer "^5.0.1" + langchainhub "~0.0.6" + langsmith "~0.0.48" + ml-distance "^4.0.0" + openai "^4.19.0" + openapi-types "^12.1.3" + p-retry "4" + uuid "^9.0.0" + yaml "^2.2.1" + zod "^3.22.3" + zod-to-json-schema "3.20.3" + +langchainhub@~0.0.6: + version "0.0.6" + resolved "https://registry.yarnpkg.com/langchainhub/-/langchainhub-0.0.6.tgz#9d2d06e4ce0807b4e8a31e19611f57aef990b54d" + integrity sha512-SW6105T+YP1cTe0yMf//7kyshCgvCTyFBMTgH2H3s9rTAR4e+78DA/BBrUL/Mt4Q5eMWui7iGuAYb3pgGsdQ9w== + +langsmith@^0.0.48: + version "0.0.48" + resolved "https://registry.yarnpkg.com/langsmith/-/langsmith-0.0.48.tgz#3a9a8ce257271ddb43d01ebf585c4370a3a3ba79" + integrity sha512-s0hW8iZ90Q9XLTnDK0Pgee245URV3b1cXQjPDj5OKm1+KN7iSK1pKx+4CO7RcFLz58Ixe7Mt+mVcomYqUuryxQ== + dependencies: + "@types/uuid" "^9.0.1" + commander "^10.0.1" + p-queue "^6.6.2" + p-retry "4" + uuid "^9.0.0" + +langsmith@~0.0.48: + version "0.0.49" + resolved "https://registry.yarnpkg.com/langsmith/-/langsmith-0.0.49.tgz#6b19298db6ca37a583ca05b25b5271674e9258b7" + integrity sha512-TC2RkjDizxtubClLNjLkB0m53NHtom9LHIhbdRYJ5O6gqUpcXkN/pefb8w734lBp9VYfnYBk/JJ8AiCq+ooeDQ== + dependencies: + "@types/uuid" "^9.0.1" + commander "^10.0.1" + p-queue "^6.6.2" + p-retry "4" + uuid "^9.0.0" + +leac@^0.6.0: + version "0.6.0" + resolved "https://registry.yarnpkg.com/leac/-/leac-0.6.0.tgz#dcf136e382e666bd2475f44a1096061b70dc0912" + integrity sha512-y+SqErxb8h7nE/fiEX07jsbuhrpO9lL8eca7/Y1nuWV2moNlXhyd59iDGcRf6moVyDMbmTNzL40SUyrFU/yDpg== + +libbase64@1.2.1: + version "1.2.1" + resolved "https://registry.yarnpkg.com/libbase64/-/libbase64-1.2.1.tgz#fb93bf4cb6d730f29b92155b6408d1bd2176a8c8" + integrity sha512-l+nePcPbIG1fNlqMzrh68MLkX/gTxk/+vdvAb388Ssi7UuUN31MI44w4Yf33mM3Cm4xDfw48mdf3rkdHszLNew== + +libmime@5.2.0: + version "5.2.0" + resolved "https://registry.yarnpkg.com/libmime/-/libmime-5.2.0.tgz#c4ed5cbd2d9fdd27534543a68bb8d17c658d51d8" + integrity sha512-X2U5Wx0YmK0rXFbk67ASMeqYIkZ6E5vY7pNWRKtnNzqjvdYYG8xtPDpCnuUEnPU9vlgNev+JoSrcaKSUaNvfsw== + dependencies: + encoding-japanese "2.0.0" + iconv-lite "0.6.3" + libbase64 "1.2.1" + libqp "2.0.1" + +libmime@5.2.1: + version "5.2.1" + resolved "https://registry.yarnpkg.com/libmime/-/libmime-5.2.1.tgz#a1075eaf702fa597161948dcae3afd03be383ac4" + integrity sha512-A0z9O4+5q+ZTj7QwNe/Juy1KARNb4WaviO4mYeFC4b8dBT2EEqK2pkM+GC8MVnkOjqhl5nYQxRgnPYRRTNmuSQ== + dependencies: + encoding-japanese "2.0.0" + iconv-lite "0.6.3" + libbase64 "1.2.1" + libqp "2.0.1" + +libqp@2.0.1: + version "2.0.1" + resolved "https://registry.yarnpkg.com/libqp/-/libqp-2.0.1.tgz#b8fed76cc1ea6c9ceff8888169e4e0de70cd5cf2" + integrity sha512-Ka0eC5LkF3IPNQHJmYBWljJsw0UvM6j+QdKRbWyCdTmYwvIDE6a7bCm0UkTAL/K+3KXK5qXT/ClcInU01OpdLg== + +lie@~3.3.0: + version "3.3.0" + resolved "https://registry.yarnpkg.com/lie/-/lie-3.3.0.tgz#dcf82dee545f46074daf200c7c1c5a08e0f40f6a" + integrity sha512-UaiMJzeWRlEujzAuw5LokY1L5ecNQYZKfmyZ9L7wDHb/p5etKaxXhohBcrw0EYby+G/NA52vRSN4N39dxHAIwQ== + dependencies: + immediate "~3.0.5" + +lines-and-columns@^1.1.6: + version "1.2.4" + resolved "https://registry.yarnpkg.com/lines-and-columns/-/lines-and-columns-1.2.4.tgz#eca284f75d2965079309dc0ad9255abb2ebc1632" + integrity sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg== + +linkify-it@4.0.1: + version "4.0.1" + resolved "https://registry.yarnpkg.com/linkify-it/-/linkify-it-4.0.1.tgz#01f1d5e508190d06669982ba31a7d9f56a5751ec" + integrity sha512-C7bfi1UZmoj8+PQx22XyeXCuBlokoyWQL5pWSP+EI6nzRylyThouddufc2c1NDIcP9k5agmN9fLpA7VNJfIiqw== + dependencies: + uc.micro "^1.0.1" + +lop@^0.4.1: + version "0.4.1" + resolved "https://registry.yarnpkg.com/lop/-/lop-0.4.1.tgz#744f1696ef480e68ce1947fe557b09db5af2a738" + integrity sha512-9xyho9why2A2tzm5aIcMWKvzqKsnxrf9B5I+8O30olh6lQU8PH978LqZoI4++37RBgS1Em5i54v1TFs/3wnmXQ== + dependencies: + duck "^0.1.12" + option "~0.2.1" + underscore "^1.13.1" + +lru-cache@^6.0.0: + version "6.0.0" + resolved "https://registry.yarnpkg.com/lru-cache/-/lru-cache-6.0.0.tgz#6d6fe6570ebd96aaf90fcad1dafa3b2566db3a94" + integrity sha512-Jo6dJ04CmSjuznwJSS3pUeWmd/H0ffTlkXXgwZi+eq1UCmqQwCh+eLsYOYCwY991i2Fah4h1BEMCx4qThGbsiA== + dependencies: + yallist "^4.0.0" + +lru-cache@^7.14.1: + version "7.18.3" + resolved "https://registry.yarnpkg.com/lru-cache/-/lru-cache-7.18.3.tgz#f793896e0fd0e954a59dfdd82f0773808df6aa89" + integrity sha512-jumlc0BIUrS3qJGgIkWZsyfAM7NCWiBcCDhnd+3NNM5KbBmLTgHVfWBcg6W+rLUsIpzpERPsvwUP7CckAQSOoA== + +mailparser@^3.0.1: + version "3.6.5" + resolved "https://registry.yarnpkg.com/mailparser/-/mailparser-3.6.5.tgz#c82d312de32a6fa3d67254e044f8c4eb8f533c31" + integrity sha512-nteTpF0Khm5JLOnt4sigmzNdUH/6mO7PZ4KEnvxf4mckyXYFFhrtAWZzbq/V5aQMH+049gA7ZjfLdh+QiX2Uqg== + dependencies: + encoding-japanese "2.0.0" + he "1.2.0" + html-to-text "9.0.5" + iconv-lite "0.6.3" + libmime "5.2.1" + linkify-it "4.0.1" + mailsplit "5.4.0" + nodemailer "6.9.3" + tlds "1.240.0" + +mailsplit@5.4.0: + version "5.4.0" + resolved "https://registry.yarnpkg.com/mailsplit/-/mailsplit-5.4.0.tgz#9f4692fadd9013e9ce632147d996931d2abac6ba" + integrity sha512-wnYxX5D5qymGIPYLwnp6h8n1+6P6vz/MJn5AzGjZ8pwICWssL+CCQjWBIToOVHASmATot4ktvlLo6CyLfOXWYA== + dependencies: + libbase64 "1.2.1" + libmime "5.2.0" + libqp "2.0.1" + +make-dir@^1.0.0: + version "1.3.0" + resolved "https://registry.yarnpkg.com/make-dir/-/make-dir-1.3.0.tgz#79c1033b80515bd6d24ec9933e860ca75ee27f0c" + integrity sha512-2w31R7SJtieJJnQtGc7RVL2StM2vGYVfqUOvUDxH6bC6aJTxPxTF0GnIgCyu7tjockiUWAYQRbxa7vKn34s5sQ== + dependencies: + pify "^3.0.0" + +make-dir@^3.1.0: + version "3.1.0" + resolved "https://registry.yarnpkg.com/make-dir/-/make-dir-3.1.0.tgz#415e967046b3a7f1d185277d84aa58203726a13f" + integrity sha512-g3FeP20LNwhALb/6Cz6Dd4F2ngze0jz7tbzrD2wAV+o9FeNHe4rL+yK2md0J/fiSf1sa1ADhXqi5+oVwOM/eGw== + dependencies: + semver "^6.0.0" + +mammoth@^1.6.0: + version "1.6.0" + resolved "https://registry.yarnpkg.com/mammoth/-/mammoth-1.6.0.tgz#3a80e016b3d1374c9404eb841048c70295276532" + integrity sha512-jOwbj6BwJzxCf6jr2l1zmSemniIkLnchvELXnDJCANlJawhzyIKObIq48B8kWEPLgUUh57k7FtEO3DHFQMnjMg== + dependencies: + "@xmldom/xmldom" "^0.8.6" + argparse "~1.0.3" + base64-js "^1.5.1" + bluebird "~3.4.0" + dingbat-to-unicode "^1.0.1" + jszip "^3.7.1" + lop "^0.4.1" + path-is-absolute "^1.0.0" + underscore "^1.13.1" + xmlbuilder "^10.0.0" + +mbox-parser@^1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/mbox-parser/-/mbox-parser-1.0.1.tgz#cba6bb372a1d3c27aff9da356b4519aeee458e1f" + integrity sha512-9PNE026G/SodrYYxZiDZDJ5YQ0piN8QCqO662MupIGwgsvBtJHFPxYc/uUsQyhUysl4kMG0NLYvfELNkM5+HzQ== + dependencies: + "@types/mailparser" "^3.0.0" + humanize-duration "^3.25.1" + mailparser "^3.0.1" + +md5@^2.3.0: + version "2.3.0" + resolved "https://registry.yarnpkg.com/md5/-/md5-2.3.0.tgz#c3da9a6aae3a30b46b7b0c349b87b110dc3bda4f" + integrity sha512-T1GITYmFaKuO91vxyoQMFETst+O71VUPEU3ze5GNzDm0OWdP8v1ziTaAEPUr/3kLsY3Sftgz242A1SetQiDL7g== + dependencies: + charenc "0.0.2" + crypt "0.0.2" + is-buffer "~1.1.6" + +media-typer@0.3.0: + version "0.3.0" + resolved "https://registry.yarnpkg.com/media-typer/-/media-typer-0.3.0.tgz#8710d7af0aa626f8fffa1ce00168545263255748" + integrity sha512-dq+qelQ9akHpcOl/gUVRTxVIOkAJ1wR3QAvb4RsVjS8oVoFjDGTc679wJYmUmknUF5HwMLOgb5O+a3KxfWapPQ== + +merge-descriptors@1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/merge-descriptors/-/merge-descriptors-1.0.1.tgz#b00aaa556dd8b44568150ec9d1b953f3f90cbb61" + integrity sha512-cCi6g3/Zr1iqQi6ySbseM1Xvooa98N0w31jzUYrXPX2xqObmFGHJ0tQ5u74H3mVh7wLouTseZyYIq39g8cNp1w== + +methods@~1.1.2: + version "1.1.2" + resolved "https://registry.yarnpkg.com/methods/-/methods-1.1.2.tgz#5529a4d67654134edcc5266656835b0f851afcee" + integrity sha512-iclAHeNqNm68zFtnZ0e+1L2yUIdvzNoauKU4WBA3VvH/vPFieF7qfRlwUZU+DA9P9bPXIS90ulxoUoCH23sV2w== + +mime-db@1.52.0: + version "1.52.0" + resolved "https://registry.yarnpkg.com/mime-db/-/mime-db-1.52.0.tgz#bbabcdc02859f4987301c856e3387ce5ec43bf70" + integrity sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg== + +mime-types@^2.1.12, mime-types@~2.1.24, mime-types@~2.1.34: + version "2.1.35" + resolved "https://registry.yarnpkg.com/mime-types/-/mime-types-2.1.35.tgz#381a871b62a734450660ae3deee44813f70d959a" + integrity sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw== + dependencies: + mime-db "1.52.0" + +mime@1.6.0: + version "1.6.0" + resolved "https://registry.yarnpkg.com/mime/-/mime-1.6.0.tgz#32cd9e5c64553bd58d19a568af452acff04981b1" + integrity sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg== + +mime@^3.0.0: + version "3.0.0" + resolved "https://registry.yarnpkg.com/mime/-/mime-3.0.0.tgz#b374550dca3a0c18443b0c950a6a58f1931cf7a7" + integrity sha512-jSCU7/VB1loIWBZe14aEYHU/+1UMEHoaO7qxCOVJOw9GgH72VAWppxNcjU+x9a2k3GSIBXNKxXQFqRvvZ7vr3A== + +minimatch@^3.1.1, minimatch@^3.1.2: + version "3.1.2" + resolved "https://registry.yarnpkg.com/minimatch/-/minimatch-3.1.2.tgz#19cd194bfd3e428f049a70817c038d89ab4be35b" + integrity sha512-J7p63hRiAjw1NDEww1W7i37+ByIrOWO5XQQAzZ3VOcL0PNybwpfmV/N05zFAzwQ9USyEcX6t3UO+K5aqBQOIHw== + dependencies: + brace-expansion "^1.1.7" + +minimist@^1.2.6: + version "1.2.8" + resolved "https://registry.yarnpkg.com/minimist/-/minimist-1.2.8.tgz#c1a464e7693302e082a075cee0c057741ac4772c" + integrity sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA== + +minipass@^3.0.0: + version "3.3.6" + resolved "https://registry.yarnpkg.com/minipass/-/minipass-3.3.6.tgz#7bba384db3a1520d18c9c0e5251c3444e95dd94a" + integrity sha512-DxiNidxSEK+tHG6zOIklvNOwm3hvCrbUrdtzY74U6HKTJxvIDfOUL5W5P2Ghd3DTkhhKPYGqeNUIh5qcM4YBfw== + dependencies: + yallist "^4.0.0" + +minipass@^5.0.0: + version "5.0.0" + resolved "https://registry.yarnpkg.com/minipass/-/minipass-5.0.0.tgz#3e9788ffb90b694a5d0ec94479a45b5d8738133d" + integrity sha512-3FnjYuehv9k6ovOEbyOswadCDPX1piCfhV8ncmYtHOjuPwylVWsghTLo7rabjC3Rx5xD4HDx8Wm1xnMF7S5qFQ== + +minizlib@^2.1.1: + version "2.1.2" + resolved "https://registry.yarnpkg.com/minizlib/-/minizlib-2.1.2.tgz#e90d3466ba209b932451508a11ce3d3632145931" + integrity sha512-bAxsR8BVfj60DWXHE3u30oHzfl4G7khkSuPW+qvpd7jFRHm7dLxOjUk1EHACJ/hxLY8phGJ0YhYHZo7jil7Qdg== + dependencies: + minipass "^3.0.0" + yallist "^4.0.0" + +mitt@3.0.1: + version "3.0.1" + resolved "https://registry.yarnpkg.com/mitt/-/mitt-3.0.1.tgz#ea36cf0cc30403601ae074c8f77b7092cdab36d1" + integrity sha512-vKivATfr97l2/QBCYAkXYDbrIWPM2IIKEl7YPhjCvKlG3kE2gm+uBo6nEXK3M5/Ffh/FLpKExzOQ3JJoJGFKBw== + +mkdirp-classic@^0.5.2: + version "0.5.3" + resolved "https://registry.yarnpkg.com/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz#fa10c9115cc6d8865be221ba47ee9bed78601113" + integrity sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A== + +mkdirp@^0.5.4: + version "0.5.6" + resolved "https://registry.yarnpkg.com/mkdirp/-/mkdirp-0.5.6.tgz#7def03d2432dcae4ba1d611445c48396062255f6" + integrity sha512-FP+p8RB8OWpF3YZBCrP5gtADmtXApB5AMLn+vdyA+PyxCjrCs00mjyUozssO33cwDeT3wNGdLxJ5M//YqtHAJw== + dependencies: + minimist "^1.2.6" + +mkdirp@^1.0.3: + version "1.0.4" + resolved "https://registry.yarnpkg.com/mkdirp/-/mkdirp-1.0.4.tgz#3eb5ed62622756d79a5f0e2a221dfebad75c2f7e" + integrity sha512-vVqVZQyf3WLx2Shd0qJ9xuvqgAyKPLAiqITEtqW0oIUjzo3PePDd6fW9iFz30ef7Ysp/oiWqbhszeGWW2T6Gzw== + +ml-array-mean@^1.1.6: + version "1.1.6" + resolved "https://registry.yarnpkg.com/ml-array-mean/-/ml-array-mean-1.1.6.tgz#d951a700dc8e3a17b3e0a583c2c64abd0c619c56" + integrity sha512-MIdf7Zc8HznwIisyiJGRH9tRigg3Yf4FldW8DxKxpCCv/g5CafTw0RRu51nojVEOXuCQC7DRVVu5c7XXO/5joQ== + dependencies: + ml-array-sum "^1.1.6" + +ml-array-sum@^1.1.6: + version "1.1.6" + resolved "https://registry.yarnpkg.com/ml-array-sum/-/ml-array-sum-1.1.6.tgz#d1d89c20793cd29c37b09d40e85681aa4515a955" + integrity sha512-29mAh2GwH7ZmiRnup4UyibQZB9+ZLyMShvt4cH4eTK+cL2oEMIZFnSyB3SS8MlsTh6q/w/yh48KmqLxmovN4Dw== + dependencies: + is-any-array "^2.0.0" + +ml-distance-euclidean@^2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/ml-distance-euclidean/-/ml-distance-euclidean-2.0.0.tgz#3a668d236649d1b8fec96380b9435c6f42c9a817" + integrity sha512-yC9/2o8QF0A3m/0IXqCTXCzz2pNEzvmcE/9HFKOZGnTjatvBbsn4lWYJkxENkA4Ug2fnYl7PXQxnPi21sgMy/Q== + +ml-distance@^4.0.0: + version "4.0.1" + resolved "https://registry.yarnpkg.com/ml-distance/-/ml-distance-4.0.1.tgz#4741d17a1735888c5388823762271dfe604bd019" + integrity sha512-feZ5ziXs01zhyFUUUeZV5hwc0f5JW0Sh0ckU1koZe/wdVkJdGxcP06KNQuF0WBTj8FttQUzcvQcpcrOp/XrlEw== + dependencies: + ml-array-mean "^1.1.6" + ml-distance-euclidean "^2.0.0" + ml-tree-similarity "^1.0.0" + +ml-tree-similarity@^1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/ml-tree-similarity/-/ml-tree-similarity-1.0.0.tgz#24705a107e32829e24d945e87219e892159c53f0" + integrity sha512-XJUyYqjSuUQkNQHMscr6tcjldsOoAekxADTplt40QKfwW6nd++1wHWV9AArl0Zvw/TIHgNaZZNvr8QGvE8wLRg== + dependencies: + binary-search "^1.3.5" + num-sort "^2.0.0" + +moment@^2.29.4: + version "2.29.4" + resolved "https://registry.yarnpkg.com/moment/-/moment-2.29.4.tgz#3dbe052889fe7c1b2ed966fcb3a77328964ef108" + integrity sha512-5LC9SOxjSc2HF6vO2CyuTDNivEdoz2IvyJJGj6X8DJ0eFyfszE0QiEd+iXmBvUP3WHxSjFH/vIsA0EN00cgr8w== + +ms@2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/ms/-/ms-2.0.0.tgz#5608aeadfc00be6c2901df5f9861788de0d597c8" + integrity sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A== + +ms@2.1.2: + version "2.1.2" + resolved "https://registry.yarnpkg.com/ms/-/ms-2.1.2.tgz#d09d1f357b443f493382a8eb3ccd183872ae6009" + integrity sha512-sGkPx+VjMtmA6MX27oA4FBFELFCZZ4S4XqeGOXCv68tT+jb3vk/RyaKWP0PTKyWtmLSM0b+adUTEvbs1PEaH2w== + +ms@2.1.3, ms@^2.0.0, ms@^2.1.1: + version "2.1.3" + resolved "https://registry.yarnpkg.com/ms/-/ms-2.1.3.tgz#574c8138ce1d2b5861f0b44579dbadd60c6615b2" + integrity sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA== + +multer@^1.4.5-lts.1: + version "1.4.5-lts.1" + resolved "https://registry.yarnpkg.com/multer/-/multer-1.4.5-lts.1.tgz#803e24ad1984f58edffbc79f56e305aec5cfd1ac" + integrity sha512-ywPWvcDMeH+z9gQq5qYHCCy+ethsk4goepZ45GLD63fOu0YcNecQxi64nDs3qluZB+murG3/D4dJ7+dGctcCQQ== + dependencies: + append-field "^1.0.0" + busboy "^1.0.0" + concat-stream "^1.5.2" + mkdirp "^0.5.4" + object-assign "^4.1.1" + type-is "^1.6.4" + xtend "^4.0.0" + +negotiator@0.6.3: + version "0.6.3" + resolved "https://registry.yarnpkg.com/negotiator/-/negotiator-0.6.3.tgz#58e323a72fedc0d6f9cd4d31fe49f51479590ccd" + integrity sha512-+EUsqGPLsM+j/zdChZjsnX51g4XrHFOIXwfnCVPGlQk/k5giakcKsuxCObBRu6DSm9opw/O6slWbJdghQM4bBg== + +netmask@^2.0.2: + version "2.0.2" + resolved "https://registry.yarnpkg.com/netmask/-/netmask-2.0.2.tgz#8b01a07644065d536383835823bc52004ebac5e7" + integrity sha512-dBpDMdxv9Irdq66304OLfEmQ9tbNRFnFTuZiLo+bD+r332bBmMJ8GBLXklIXXgxd3+v9+KUnZaUR5PJMa75Gsg== + +node-addon-api@^5.0.0: + version "5.1.0" + resolved "https://registry.yarnpkg.com/node-addon-api/-/node-addon-api-5.1.0.tgz#49da1ca055e109a23d537e9de43c09cca21eb762" + integrity sha512-eh0GgfEkpnoWDq+VY8OyvYhFEzBk6jIYbRKdIlyTiAXIVJ8PyBaKb0rp7oDtoddbdoHWhq8wwr+XZ81F1rpNdA== + +node-domexception@1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/node-domexception/-/node-domexception-1.0.0.tgz#6888db46a1f71c0b76b3f7555016b63fe64766e5" + integrity sha512-/jKZoMpw0F8GRwl4/eLROPA3cfcXtLApP0QzLmUT/HuPCZWyB7IY9ZrMeKw2O/nFIqPQB3PVM9aYm0F312AXDQ== + +node-ensure@^0.0.0: + version "0.0.0" + resolved "https://registry.yarnpkg.com/node-ensure/-/node-ensure-0.0.0.tgz#ecae764150de99861ec5c810fd5d096b183932a7" + integrity sha512-DRI60hzo2oKN1ma0ckc6nQWlHU69RH6xN0sjQTjMpChPfTYvKZdcQFfdYK2RWbJcKyUizSIy/l8OTGxMAM1QDw== + +node-fetch@^2.6.12: + version "2.7.0" + resolved "https://registry.yarnpkg.com/node-fetch/-/node-fetch-2.7.0.tgz#d0f0fa6e3e2dc1d27efcd8ad99d550bda94d187d" + integrity sha512-c4FRfUm/dbcWZ7U+1Wq0AwCyFL+3nt2bEw05wfxSz+DWpWsitgmSgYmy2dQdWyKC1694ELPqMs/YzUSNozLt8A== + dependencies: + whatwg-url "^5.0.0" + +node-fetch@^2.6.7, node-fetch@^2.6.9: + version "2.6.12" + resolved "https://registry.yarnpkg.com/node-fetch/-/node-fetch-2.6.12.tgz#02eb8e22074018e3d5a83016649d04df0e348fba" + integrity sha512-C/fGU2E8ToujUivIO0H+tpQ6HWo4eEmchoPIoXtxCrVghxdKq+QOHqEZW7tuP3KlV3bC8FRMO5nMCC7Zm1VP6g== + dependencies: + whatwg-url "^5.0.0" + +node-forge@^1.3.1: + version "1.3.1" + resolved "https://registry.yarnpkg.com/node-forge/-/node-forge-1.3.1.tgz#be8da2af243b2417d5f646a770663a92b7e9ded3" + integrity sha512-dPEtOeMvF9VMcYV/1Wb8CPoVAXtp6MKMlcbAt4ddqmGqUJ6fQZFXkNZNkNlfevtNkGtaSoXf/vNNNSvgrdXwtA== + +nodemailer@6.9.3: + version "6.9.3" + resolved "https://registry.yarnpkg.com/nodemailer/-/nodemailer-6.9.3.tgz#e4425b85f05d83c43c5cd81bf84ab968f8ef5cbe" + integrity sha512-fy9v3NgTzBngrMFkDsKEj0r02U7jm6XfC3b52eoNV+GCrGj+s8pt5OqhiJdWKuw51zCTdiNR/IUD1z33LIIGpg== + +nodemon@^2.0.22: + version "2.0.22" + resolved "https://registry.yarnpkg.com/nodemon/-/nodemon-2.0.22.tgz#182c45c3a78da486f673d6c1702e00728daf5258" + integrity sha512-B8YqaKMmyuCO7BowF1Z1/mkPqLk6cs/l63Ojtd6otKjMx47Dq1utxfRxcavH1I7VSaL8n5BUaoutadnsX3AAVQ== + dependencies: + chokidar "^3.5.2" + debug "^3.2.7" + ignore-by-default "^1.0.1" + minimatch "^3.1.2" + pstree.remy "^1.1.8" + semver "^5.7.1" + simple-update-notifier "^1.0.7" + supports-color "^5.5.0" + touch "^3.1.0" + undefsafe "^2.0.5" + +nopt@^5.0.0: + version "5.0.0" + resolved "https://registry.yarnpkg.com/nopt/-/nopt-5.0.0.tgz#530942bb58a512fccafe53fe210f13a25355dc88" + integrity sha512-Tbj67rffqceeLpcRXrT7vKAN8CwfPeIBgM7E6iBkmKLV7bEMwpGgYLGv0jACUsECaa/vuxP0IjEont6umdMgtQ== + dependencies: + abbrev "1" + +nopt@~1.0.10: + version "1.0.10" + resolved "https://registry.yarnpkg.com/nopt/-/nopt-1.0.10.tgz#6ddd21bd2a31417b92727dd585f8a6f37608ebee" + integrity sha512-NWmpvLSqUrgrAC9HCuxEvb+PSloHpqVu+FqcO4eeF2h5qYRhA7ev6KvelyQAKtegUbC6RypJnlEOhd8vloNKYg== + dependencies: + abbrev "1" + +normalize-path@^3.0.0, normalize-path@~3.0.0: + version "3.0.0" + resolved "https://registry.yarnpkg.com/normalize-path/-/normalize-path-3.0.0.tgz#0dcd69ff23a1c9b11fd0978316644a0388216a65" + integrity sha512-6eZs5Ls3WtCisHWp9S2GUy8dqkpGi4BVSz3GaqiE6ezub0512ESztXUwUB6C6IKbQkY2Pnb/mD4WYojCRwcwLA== + +npmlog@^5.0.1: + version "5.0.1" + resolved "https://registry.yarnpkg.com/npmlog/-/npmlog-5.0.1.tgz#f06678e80e29419ad67ab964e0fa69959c1eb8b0" + integrity sha512-AqZtDUWOMKs1G/8lwylVjrdYgqA4d9nu8hc+0gzRxlDb1I10+FHBGMXs6aiQHFdCUUlqH99MUMuLfzWDNDtfxw== + dependencies: + are-we-there-yet "^2.0.0" + console-control-strings "^1.1.0" + gauge "^3.0.0" + set-blocking "^2.0.0" + +num-sort@^2.0.0: + version "2.1.0" + resolved "https://registry.yarnpkg.com/num-sort/-/num-sort-2.1.0.tgz#1cbb37aed071329fdf41151258bc011898577a9b" + integrity sha512-1MQz1Ed8z2yckoBeSfkQHHO9K1yDRxxtotKSJ9yvcTUUxSvfvzEq5GwBrjjHEpMlq/k5gvXdmJ1SbYxWtpNoVg== + +object-assign@^4, object-assign@^4.0.1, object-assign@^4.1.1: + version "4.1.1" + resolved "https://registry.yarnpkg.com/object-assign/-/object-assign-4.1.1.tgz#2109adc7965887cfc05cbbd442cac8bfbb360863" + integrity sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg== + +object-inspect@^1.9.0: + version "1.12.3" + resolved "https://registry.yarnpkg.com/object-inspect/-/object-inspect-1.12.3.tgz#ba62dffd67ee256c8c086dfae69e016cd1f198b9" + integrity sha512-geUvdk7c+eizMNUDkRpW1wJwgfOiOeHbxBR/hLXK1aT6zmVSO0jsQcs7fj6MGw89jC/cjGfLcNOrtMYtGqm81g== + +officeparser@^4.0.5: + version "4.0.5" + resolved "https://registry.yarnpkg.com/officeparser/-/officeparser-4.0.5.tgz#61ad3eb81a436170efa29dbe1ea5937f0e90e619" + integrity sha512-8uUuNCaafpHH5bF1uxLS0KxAzgN5KiX1K8GADqwCoebUJo8TlZR7EGEbJwdl5lG2OtDBJWNzujREkIM4fy5I3Q== + dependencies: + "@xmldom/xmldom" "^0.8.10" + decompress "^4.2.0" + file-type "^16.5.4" + pdf-parse "^1.1.1" + rimraf "^2.6.3" + +on-finished@2.4.1: + version "2.4.1" + resolved "https://registry.yarnpkg.com/on-finished/-/on-finished-2.4.1.tgz#58c8c44116e54845ad57f14ab10b03533184ac3f" + integrity sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg== + dependencies: + ee-first "1.1.1" + +once@^1.3.0, once@^1.3.1, once@^1.4.0: + version "1.4.0" + resolved "https://registry.yarnpkg.com/once/-/once-1.4.0.tgz#583b1aa775961d4b113ac17d9c50baef9dd76bd1" + integrity sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w== + dependencies: + wrappy "1" + +openai@^4.19.0: + version "4.20.1" + resolved "https://registry.yarnpkg.com/openai/-/openai-4.20.1.tgz#afa0d496d125b5a0f6cebcb4b9aeabf71e00214e" + integrity sha512-Dd3q8EvINfganZFtg6V36HjrMaihqRgIcKiHua4Nq9aw/PxOP48dhbsk8x5klrxajt5Lpnc1KTOG5i1S6BKAJA== + dependencies: + "@types/node" "^18.11.18" + "@types/node-fetch" "^2.6.4" + abort-controller "^3.0.0" + agentkeepalive "^4.2.1" + digest-fetch "^1.3.0" + form-data-encoder "1.7.2" + formdata-node "^4.3.2" + node-fetch "^2.6.7" + web-streams-polyfill "^3.2.1" + +openapi-types@^12.1.3: + version "12.1.3" + resolved "https://registry.yarnpkg.com/openapi-types/-/openapi-types-12.1.3.tgz#471995eb26c4b97b7bd356aacf7b91b73e777dd3" + integrity sha512-N4YtSYJqghVu4iek2ZUvcN/0aqH1kRDuNqzcycDxhOUpg7GdvLa2F3DgS6yBNhInhv2r/6I0Flkn7CqL8+nIcw== + +option@~0.2.1: + version "0.2.4" + resolved "https://registry.yarnpkg.com/option/-/option-0.2.4.tgz#fd475cdf98dcabb3cb397a3ba5284feb45edbfe4" + integrity sha512-pkEqbDyl8ou5cpq+VsnQbe/WlEy5qS7xPzMS1U55OCG9KPvwFD46zDbxQIj3egJSFc3D+XhYOPUzz49zQAVy7A== + +p-finally@^1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/p-finally/-/p-finally-1.0.0.tgz#3fbcfb15b899a44123b34b6dcc18b724336a2cae" + integrity sha512-LICb2p9CB7FS+0eR1oqWnHhp0FljGLZCWBE9aix0Uye9W8LTQPwMTYVGWQWIw9RdQiDg4+epXQODwIYJtSJaow== + +p-queue@^6.6.2: + version "6.6.2" + resolved "https://registry.yarnpkg.com/p-queue/-/p-queue-6.6.2.tgz#2068a9dcf8e67dd0ec3e7a2bcb76810faa85e426" + integrity sha512-RwFpb72c/BhQLEXIZ5K2e+AhgNVmIejGlTgiB9MzZ0e93GRvqZ7uSi0dvRF7/XIXDeNkra2fNHBxTyPDGySpjQ== + dependencies: + eventemitter3 "^4.0.4" + p-timeout "^3.2.0" + +p-retry@4: + version "4.6.2" + resolved "https://registry.yarnpkg.com/p-retry/-/p-retry-4.6.2.tgz#9baae7184057edd4e17231cee04264106e092a16" + integrity sha512-312Id396EbJdvRONlngUx0NydfrIQ5lsYu0znKVUzVvArzEIt08V1qhtyESbGVd1FGX7UKtiFp5uwKZdM8wIuQ== + dependencies: + "@types/retry" "0.12.0" + retry "^0.13.1" + +p-timeout@^3.2.0: + version "3.2.0" + resolved "https://registry.yarnpkg.com/p-timeout/-/p-timeout-3.2.0.tgz#c7e17abc971d2a7962ef83626b35d635acf23dfe" + integrity sha512-rhIwUycgwwKcP9yTOOFK/AKsAopjjCakVqLHePO3CC6Mir1Z99xT+R63jZxAT5lFZLa2inS5h+ZS2GvR99/FBg== + dependencies: + p-finally "^1.0.0" + +pac-proxy-agent@^7.0.1: + version "7.0.1" + resolved "https://registry.yarnpkg.com/pac-proxy-agent/-/pac-proxy-agent-7.0.1.tgz#6b9ddc002ec3ff0ba5fdf4a8a21d363bcc612d75" + integrity sha512-ASV8yU4LLKBAjqIPMbrgtaKIvxQri/yh2OpI+S6hVa9JRkUI3Y3NPFbfngDtY7oFtSMD3w31Xns89mDa3Feo5A== + dependencies: + "@tootallnate/quickjs-emscripten" "^0.23.0" + agent-base "^7.0.2" + debug "^4.3.4" + get-uri "^6.0.1" + http-proxy-agent "^7.0.0" + https-proxy-agent "^7.0.2" + pac-resolver "^7.0.0" + socks-proxy-agent "^8.0.2" + +pac-resolver@^7.0.0: + version "7.0.0" + resolved "https://registry.yarnpkg.com/pac-resolver/-/pac-resolver-7.0.0.tgz#79376f1ca26baf245b96b34c339d79bff25e900c" + integrity sha512-Fd9lT9vJbHYRACT8OhCbZBbxr6KRSawSovFpy8nDGshaK99S/EBhVIHp9+crhxrsZOuvLpgL1n23iyPg6Rl2hg== + dependencies: + degenerator "^5.0.0" + ip "^1.1.8" + netmask "^2.0.2" + +pako@~1.0.2: + version "1.0.11" + resolved "https://registry.yarnpkg.com/pako/-/pako-1.0.11.tgz#6c9599d340d54dfd3946380252a35705a6b992bf" + integrity sha512-4hLB8Py4zZce5s4yd9XzopqwVv/yGNhV1Bl8NTmCq1763HeK2+EwVTv+leGeL13Dnh2wfbqowVPXCIO0z4taYw== + +parent-module@^1.0.0: + version "1.0.1" + resolved "https://registry.yarnpkg.com/parent-module/-/parent-module-1.0.1.tgz#691d2709e78c79fae3a156622452d00762caaaa2" + integrity sha512-GQ2EWRpQV8/o+Aw8YqtfZZPfNRWZYkbidE9k5rpl/hC3vtHHBfGm2Ifi6qWV+coDGkrUKZAxE3Lot5kcsRlh+g== + dependencies: + callsites "^3.0.0" + +parse-json@^5.2.0: + version "5.2.0" + resolved "https://registry.yarnpkg.com/parse-json/-/parse-json-5.2.0.tgz#c76fc66dee54231c962b22bcc8a72cf2f99753cd" + integrity sha512-ayCKvm/phCGxOkYRSCM82iDwct8/EonSEgCSxWxD7ve6jHggsFl4fZVQBPRNgQoKiuV/odhFrGzQXZwbifC8Rg== + dependencies: + "@babel/code-frame" "^7.0.0" + error-ex "^1.3.1" + json-parse-even-better-errors "^2.3.0" + lines-and-columns "^1.1.6" + +parseley@^0.12.0: + version "0.12.1" + resolved "https://registry.yarnpkg.com/parseley/-/parseley-0.12.1.tgz#4afd561d50215ebe259e3e7a853e62f600683aef" + integrity sha512-e6qHKe3a9HWr0oMRVDTRhKce+bRO8VGQR3NyVwcjwrbhMmFCX9KszEV35+rn4AdilFAq9VPxP/Fe1wC9Qjd2lw== + dependencies: + leac "^0.6.0" + peberminta "^0.9.0" + +parseurl@~1.3.3: + version "1.3.3" + resolved "https://registry.yarnpkg.com/parseurl/-/parseurl-1.3.3.tgz#9da19e7bee8d12dff0513ed5b76957793bc2e8d4" + integrity sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ== + +path-is-absolute@^1.0.0: + version "1.0.1" + resolved "https://registry.yarnpkg.com/path-is-absolute/-/path-is-absolute-1.0.1.tgz#174b9268735534ffbc7ace6bf53a5a9e1b5c5f5f" + integrity sha512-AVbw3UJ2e9bq64vSaS9Am0fje1Pa8pbGqTTsmXfaIiMpnr5DlDhfJOuLj9Sf95ZPVDAUerDfEk88MPmPe7UCQg== + +path-to-regexp@0.1.7: + version "0.1.7" + resolved "https://registry.yarnpkg.com/path-to-regexp/-/path-to-regexp-0.1.7.tgz#df604178005f522f15eb4490e7247a1bfaa67f8c" + integrity sha512-5DFkuoqlv1uYQKxy8omFBeJPQcdoE07Kv2sferDCrAq1ohOU+MSDswDIbnx3YAM60qIOnYa53wBhXW0EbMonrQ== + +path-type@^4.0.0: + version "4.0.0" + resolved "https://registry.yarnpkg.com/path-type/-/path-type-4.0.0.tgz#84ed01c0a7ba380afe09d90a8c180dcd9d03043b" + integrity sha512-gDKb8aZMDeD/tZWs9P6+q0J9Mwkdl6xMV8TjnGP3qJVJ06bdMgkbBlLU8IdfOsIsFz2BW1rNVT3XuNEl8zPAvw== + +pdf-parse@^1.1.1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/pdf-parse/-/pdf-parse-1.1.1.tgz#745e07408679548b3995ff896fd38e96e19d14a7" + integrity sha512-v6ZJ/efsBpGrGGknjtq9J/oC8tZWq0KWL5vQrk2GlzLEQPUDB1ex+13Rmidl1neNN358Jn9EHZw5y07FFtaC7A== + dependencies: + debug "^3.1.0" + node-ensure "^0.0.0" + +peberminta@^0.9.0: + version "0.9.0" + resolved "https://registry.yarnpkg.com/peberminta/-/peberminta-0.9.0.tgz#8ec9bc0eb84b7d368126e71ce9033501dca2a352" + integrity sha512-XIxfHpEuSJbITd1H3EeQwpcZbTLHc+VVr8ANI9t5sit565tsI4/xK3KWTUFE2e6QiangUkh3B0jihzmGnNrRsQ== + +peek-readable@^4.1.0: + version "4.1.0" + resolved "https://registry.yarnpkg.com/peek-readable/-/peek-readable-4.1.0.tgz#4ece1111bf5c2ad8867c314c81356847e8a62e72" + integrity sha512-ZI3LnwUv5nOGbQzD9c2iDG6toheuXSZP5esSHBjopsXH4dg19soufvpUGA3uohi5anFtGb2lhAVdHzH6R/Evvg== + +pend@~1.2.0: + version "1.2.0" + resolved "https://registry.yarnpkg.com/pend/-/pend-1.2.0.tgz#7a57eb550a6783f9115331fcf4663d5c8e007a50" + integrity sha512-F3asv42UuXchdzt+xXqfW1OGlVBe+mxa2mqI0pg5yAHZPvFmY3Y6drSf/GQ1A86WgWEN9Kzh/WrgKa6iGcHXLg== + +picomatch@^2.0.4, picomatch@^2.2.1: + version "2.3.1" + resolved "https://registry.yarnpkg.com/picomatch/-/picomatch-2.3.1.tgz#3ba3833733646d9d3e4995946c1365a67fb07a42" + integrity sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA== + +pify@^2.3.0: + version "2.3.0" + resolved "https://registry.yarnpkg.com/pify/-/pify-2.3.0.tgz#ed141a6ac043a849ea588498e7dca8b15330e90c" + integrity sha512-udgsAY+fTnvv7kI7aaxbqwWNb0AHiB0qBO89PZKPkoTmGOgdbrHDKD+0B2X4uTfJ/FT1R09r9gTsjUjNJotuog== + +pify@^3.0.0: + version "3.0.0" + resolved "https://registry.yarnpkg.com/pify/-/pify-3.0.0.tgz#e5a4acd2c101fdf3d9a4d07f0dbc4db49dd28176" + integrity sha512-C3FsVNH1udSEX48gGX1xfvwTWfsYWj5U+8/uK15BGzIGrKoUpghX8hWZwa/OFnakBiiVNmBvemTJR5mcy7iPcg== + +pinkie-promise@^2.0.0: + version "2.0.1" + resolved "https://registry.yarnpkg.com/pinkie-promise/-/pinkie-promise-2.0.1.tgz#2135d6dfa7a358c069ac9b178776288228450ffa" + integrity sha512-0Gni6D4UcLTbv9c57DfxDGdr41XfgUjqWZu492f0cIGr16zDU06BWP/RAEvOuo7CQ0CNjHaLlM59YJJFm3NWlw== + dependencies: + pinkie "^2.0.0" + +pinkie@^2.0.0: + version "2.0.4" + resolved "https://registry.yarnpkg.com/pinkie/-/pinkie-2.0.4.tgz#72556b80cfa0d48a974e80e77248e80ed4f7f870" + integrity sha512-MnUuEycAemtSaeFSjXKW/aroV7akBbY+Sv+RkyqFjgAe73F+MR0TBWKBRDkmfWq/HiFmdavfZ1G7h4SPZXaCSg== + +prettier@^2.4.1: + version "2.8.8" + resolved "https://registry.yarnpkg.com/prettier/-/prettier-2.8.8.tgz#e8c5d7e98a4305ffe3de2e1fc4aca1a71c28b1da" + integrity sha512-tdN8qQGvNjw4CHbY+XXk0JgCXn9QiF21a55rBe5LJAU+kDyC4WQn4+awm2Xfk2lQMk5fKup9XgzTZtGkjBdP9Q== + +process-nextick-args@~2.0.0: + version "2.0.1" + resolved "https://registry.yarnpkg.com/process-nextick-args/-/process-nextick-args-2.0.1.tgz#7820d9b16120cc55ca9ae7792680ae7dba6d7fe2" + integrity sha512-3ouUOpQhtgrbOa17J7+uxOTpITYWaGP7/AhoR3+A+/1e9skrzelGi/dXzEYyvbxubEF6Wn2ypscTKiKJFFn1ag== + +progress@2.0.3: + version "2.0.3" + resolved "https://registry.yarnpkg.com/progress/-/progress-2.0.3.tgz#7e8cf8d8f5b8f239c1bc68beb4eb78567d572ef8" + integrity sha512-7PiHtLll5LdnKIMw100I+8xJXR5gW2QwWYkT6iJva0bXitZKa/XMrSbdmg3r2Xnaidz9Qumd0VPaMrZlF9V9sA== + +proxy-addr@~2.0.7: + version "2.0.7" + resolved "https://registry.yarnpkg.com/proxy-addr/-/proxy-addr-2.0.7.tgz#f19fe69ceab311eeb94b42e70e8c2070f9ba1025" + integrity sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg== + dependencies: + forwarded "0.2.0" + ipaddr.js "1.9.1" + +proxy-agent@6.3.1: + version "6.3.1" + resolved "https://registry.yarnpkg.com/proxy-agent/-/proxy-agent-6.3.1.tgz#40e7b230552cf44fd23ffaf7c59024b692612687" + integrity sha512-Rb5RVBy1iyqOtNl15Cw/llpeLH8bsb37gM1FUfKQ+Wck6xHlbAhWGUFiTRHtkjqGTA5pSHz6+0hrPW/oECihPQ== + dependencies: + agent-base "^7.0.2" + debug "^4.3.4" + http-proxy-agent "^7.0.0" + https-proxy-agent "^7.0.2" + lru-cache "^7.14.1" + pac-proxy-agent "^7.0.1" + proxy-from-env "^1.1.0" + socks-proxy-agent "^8.0.2" + +proxy-from-env@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/proxy-from-env/-/proxy-from-env-1.1.0.tgz#e102f16ca355424865755d2c9e8ea4f24d58c3e2" + integrity sha512-D+zkORCbA9f1tdWRK0RaCR3GPv50cMxcrz4X8k5LTSUD1Dkw47mKJEZQNunItRTkWwgtaUSo1RVFRIG9ZXiFYg== + +pstree.remy@^1.1.8: + version "1.1.8" + resolved "https://registry.yarnpkg.com/pstree.remy/-/pstree.remy-1.1.8.tgz#c242224f4a67c21f686839bbdb4ac282b8373d3a" + integrity sha512-77DZwxQmxKnu3aR542U+X8FypNzbfJ+C5XQDk3uWjWxn6151aIMGthWYRXTqT1E5oJvg+ljaa2OJi+VfvCOQ8w== + +pump@^3.0.0: + version "3.0.0" + resolved "https://registry.yarnpkg.com/pump/-/pump-3.0.0.tgz#b4a2116815bde2f4e1ea602354e8c75565107a64" + integrity sha512-LwZy+p3SFs1Pytd/jYct4wpv49HiYCqd9Rlc5ZVdk0V+8Yzv6jR5Blk3TRmPL1ft69TxP0IMZGJ+WPFU2BFhww== + dependencies: + end-of-stream "^1.1.0" + once "^1.3.1" + +puppeteer-core@21.6.1: + version "21.6.1" + resolved "https://registry.yarnpkg.com/puppeteer-core/-/puppeteer-core-21.6.1.tgz#10eccb4dc3167c8c26bc21122fabb45a9fda9ca7" + integrity sha512-0chaaK/RL9S1U3bsyR4fUeUfoj51vNnjWvXgG6DcsyMjwYNpLcAThv187i1rZCo7QhJP0wZN8plQkjNyrq2h+A== + dependencies: + "@puppeteer/browsers" "1.9.0" + chromium-bidi "0.5.1" + cross-fetch "4.0.0" + debug "4.3.4" + devtools-protocol "0.0.1203626" + ws "8.15.1" + +puppeteer@^21.6.1: + version "21.6.1" + resolved "https://registry.yarnpkg.com/puppeteer/-/puppeteer-21.6.1.tgz#2ec0878906ff90b3a424f19e5eb006592abe25b6" + integrity sha512-O+pbc61oj8ln6m8EJKncrsQFmytgRyFYERtk190PeLbJn5JKpmmynn2p1PiFrlhCitAQXLJ0MOy7F0TeyCRqBg== + dependencies: + "@puppeteer/browsers" "1.9.0" + cosmiconfig "8.3.6" + puppeteer-core "21.6.1" + +qs@6.11.0: + version "6.11.0" + resolved "https://registry.yarnpkg.com/qs/-/qs-6.11.0.tgz#fd0d963446f7a65e1367e01abd85429453f0c37a" + integrity sha512-MvjoMCJwEarSbUYk5O+nmoSzSutSsTwF85zcHPQ9OrlFoZOYIjaqBAJIqIXjptyD5vThxGq52Xu/MaJzRkIk4Q== + dependencies: + side-channel "^1.0.4" + +qs@^6.7.0: + version "6.11.2" + resolved "https://registry.yarnpkg.com/qs/-/qs-6.11.2.tgz#64bea51f12c1f5da1bc01496f48ffcff7c69d7d9" + integrity sha512-tDNIz22aBzCDxLtVH++VnTfzxlfeK5CbqohpSqpJgj1Wg/cQbStNAz3NuqCs5vV+pjBsK4x4pN9HlVh7rcYRiA== + dependencies: + side-channel "^1.0.4" + +queue-tick@^1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/queue-tick/-/queue-tick-1.0.1.tgz#f6f07ac82c1fd60f82e098b417a80e52f1f4c142" + integrity sha512-kJt5qhMxoszgU/62PLP1CJytzd2NKetjSRnyuj31fDd3Rlcz3fzlFdFLD1SItunPwyqEOkca6GbV612BWfaBag== + +range-parser@~1.2.1: + version "1.2.1" + resolved "https://registry.yarnpkg.com/range-parser/-/range-parser-1.2.1.tgz#3cf37023d199e1c24d1a55b84800c2f3e6468031" + integrity sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg== + +raw-body@2.5.1: + version "2.5.1" + resolved "https://registry.yarnpkg.com/raw-body/-/raw-body-2.5.1.tgz#fe1b1628b181b700215e5fd42389f98b71392857" + integrity sha512-qqJBtEyVgS0ZmPGdCFPWJ3FreoqvG4MVQln/kCgF7Olq95IbOp0/BWyMwbdtn4VTvkM8Y7khCQ2Xgk/tcrCXig== + dependencies: + bytes "3.1.2" + http-errors "2.0.0" + iconv-lite "0.4.24" + unpipe "1.0.0" + +raw-body@2.5.2: + version "2.5.2" + resolved "https://registry.yarnpkg.com/raw-body/-/raw-body-2.5.2.tgz#99febd83b90e08975087e8f1f9419a149366b68a" + integrity sha512-8zGqypfENjCIqGhgXToC8aB2r7YrBX+AQAfIPs/Mlk+BtPTztOvTS01NRW/3Eh60J+a48lt8qsCzirQ6loCVfA== + dependencies: + bytes "3.1.2" + http-errors "2.0.0" + iconv-lite "0.4.24" + unpipe "1.0.0" + +readable-stream@^2.2.2, readable-stream@^2.3.0, readable-stream@^2.3.5, readable-stream@~2.3.6: + version "2.3.8" + resolved "https://registry.yarnpkg.com/readable-stream/-/readable-stream-2.3.8.tgz#91125e8042bba1b9887f49345f6277027ce8be9b" + integrity sha512-8p0AUk4XODgIewSi0l8Epjs+EVnWiK7NoDIEGU0HhE7+ZyY8D1IMY7odu5lRrFXGg71L15KG8QrPmum45RTtdA== + dependencies: + core-util-is "~1.0.0" + inherits "~2.0.3" + isarray "~1.0.0" + process-nextick-args "~2.0.0" + safe-buffer "~5.1.1" + string_decoder "~1.1.1" + util-deprecate "~1.0.1" + +readable-stream@^3.6.0: + version "3.6.2" + resolved "https://registry.yarnpkg.com/readable-stream/-/readable-stream-3.6.2.tgz#56a9b36ea965c00c5a93ef31eb111a0f11056967" + integrity sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA== + dependencies: + inherits "^2.0.3" + string_decoder "^1.1.1" + util-deprecate "^1.0.1" + +readable-web-to-node-stream@^3.0.0: + version "3.0.2" + resolved "https://registry.yarnpkg.com/readable-web-to-node-stream/-/readable-web-to-node-stream-3.0.2.tgz#5d52bb5df7b54861fd48d015e93a2cb87b3ee0bb" + integrity sha512-ePeK6cc1EcKLEhJFt/AebMCLL+GgSKhuygrZ/GLaKZYEecIgIECf4UaUuaByiGtzckwR4ain9VzUh95T1exYGw== + dependencies: + readable-stream "^3.6.0" + +readdirp@~3.6.0: + version "3.6.0" + resolved "https://registry.yarnpkg.com/readdirp/-/readdirp-3.6.0.tgz#74a370bd857116e245b29cc97340cd431a02a6c7" + integrity sha512-hOS089on8RduqdbhvQ5Z37A0ESjsqz6qnRcffsMU3495FuTdqSm+7bhJ29JvIOsBDEEnan5DPu9t3To9VRlMzA== + dependencies: + picomatch "^2.2.1" + +require-directory@^2.1.1: + version "2.1.1" + resolved "https://registry.yarnpkg.com/require-directory/-/require-directory-2.1.1.tgz#8c64ad5fd30dab1c976e2344ffe7f792a6a6df42" + integrity sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q== + +resolve-from@^4.0.0: + version "4.0.0" + resolved "https://registry.yarnpkg.com/resolve-from/-/resolve-from-4.0.0.tgz#4abcd852ad32dd7baabfe9b40e00a36db5f392e6" + integrity sha512-pb/MYmXstAkysRFx8piNI1tGFNQIFA3vkE3Gq4EuA1dF6gHp/+vgZqsCGJapvy8N3Q+4o7FwvquPJcnZ7RYy4g== + +retry@^0.13.1: + version "0.13.1" + resolved "https://registry.yarnpkg.com/retry/-/retry-0.13.1.tgz#185b1587acf67919d63b357349e03537b2484658" + integrity sha512-XQBQ3I8W1Cge0Seh+6gjj03LbmRFWuoszgK9ooCpwYIrhhoO80pfq4cUkU5DkknwfOfFteRwlZ56PYOGYyFWdg== + +rimraf@^2.6.3: + version "2.7.1" + resolved "https://registry.yarnpkg.com/rimraf/-/rimraf-2.7.1.tgz#35797f13a7fdadc566142c29d4f07ccad483e3ec" + integrity sha512-uWjbaKIK3T1OSVptzX7Nl6PvQ3qAGtKEtVRjRuazjfL3Bx5eI409VZSqgND+4UNnmzLVdPj9FqFJNPqBZFve4w== + dependencies: + glob "^7.1.3" + +rimraf@^3.0.2: + version "3.0.2" + resolved "https://registry.yarnpkg.com/rimraf/-/rimraf-3.0.2.tgz#f1a5402ba6220ad52cc1282bac1ae3aa49fd061a" + integrity sha512-JZkJMZkAGFFPP2YqXZXPbMlMBgsxzE8ILs4lMIX/2o0L9UBw9O/Y3o6wFw/i9YLapcUJWwqbi3kdxIPdC62TIA== + dependencies: + glob "^7.1.3" + +safe-buffer@5.2.1, safe-buffer@^5.0.1, safe-buffer@^5.1.1, safe-buffer@~5.2.0: + version "5.2.1" + resolved "https://registry.yarnpkg.com/safe-buffer/-/safe-buffer-5.2.1.tgz#1eaf9fa9bdb1fdd4ec75f58f9cdb4e6b7827eec6" + integrity sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ== + +safe-buffer@~5.1.0, safe-buffer@~5.1.1: + version "5.1.2" + resolved "https://registry.yarnpkg.com/safe-buffer/-/safe-buffer-5.1.2.tgz#991ec69d296e0313747d59bdfd2b745c35f8828d" + integrity sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g== + +"safer-buffer@>= 2.1.2 < 3", "safer-buffer@>= 2.1.2 < 3.0.0": + version "2.1.2" + resolved "https://registry.yarnpkg.com/safer-buffer/-/safer-buffer-2.1.2.tgz#44fa161b0187b9549dd84bb91802f9bd8385cd6a" + integrity sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg== + +seek-bzip@^1.0.5: + version "1.0.6" + resolved "https://registry.yarnpkg.com/seek-bzip/-/seek-bzip-1.0.6.tgz#35c4171f55a680916b52a07859ecf3b5857f21c4" + integrity sha512-e1QtP3YL5tWww8uKaOCQ18UxIT2laNBXHjV/S2WYCiK4udiv8lkG89KRIoCjUagnAmCBurjF4zEVX2ByBbnCjQ== + dependencies: + commander "^2.8.1" + +selderee@^0.11.0: + version "0.11.0" + resolved "https://registry.yarnpkg.com/selderee/-/selderee-0.11.0.tgz#6af0c7983e073ad3e35787ffe20cefd9daf0ec8a" + integrity sha512-5TF+l7p4+OsnP8BCCvSyZiSPc4x4//p5uPwK8TCnVPJYRmU2aYKMpOXvw8zM5a5JvuuCGN1jmsMwuU2W02ukfA== + dependencies: + parseley "^0.12.0" + +semver@^5.7.1: + version "5.7.2" + resolved "https://registry.yarnpkg.com/semver/-/semver-5.7.2.tgz#48d55db737c3287cd4835e17fa13feace1c41ef8" + integrity sha512-cBznnQ9KjJqU67B52RMC65CMarK2600WFnbkcaiwWq3xy/5haFJlshgnpjovMVJ+Hff49d8GEn0b87C5pDQ10g== + +semver@^6.0.0: + version "6.3.1" + resolved "https://registry.yarnpkg.com/semver/-/semver-6.3.1.tgz#556d2ef8689146e46dcea4bfdd095f3434dffcb4" + integrity sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA== + +semver@^7.3.5: + version "7.5.4" + resolved "https://registry.yarnpkg.com/semver/-/semver-7.5.4.tgz#483986ec4ed38e1c6c48c34894a9182dbff68a6e" + integrity sha512-1bCSESV6Pv+i21Hvpxp3Dx+pSD8lIPt8uVjRrxAUt/nbswYc+tK6Y2btiULjd4+fnq15PX+nqQDC7Oft7WkwcA== + dependencies: + lru-cache "^6.0.0" + +semver@~7.0.0: + version "7.0.0" + resolved "https://registry.yarnpkg.com/semver/-/semver-7.0.0.tgz#5f3ca35761e47e05b206c6daff2cf814f0316b8e" + integrity sha512-+GB6zVA9LWh6zovYQLALHwv5rb2PHGlJi3lfiqIHxR0uuwCgefcOJc59v9fv1w8GbStwxuuqqAjI9NMAOOgq1A== + +send@0.18.0: + version "0.18.0" + resolved "https://registry.yarnpkg.com/send/-/send-0.18.0.tgz#670167cc654b05f5aa4a767f9113bb371bc706be" + integrity sha512-qqWzuOjSFOuqPjFe4NOsMLafToQQwBSOEpS+FwEt3A2V3vKubTquT3vmLTQpFgMXp8AlFWFuP1qKaJZOtPpVXg== + dependencies: + debug "2.6.9" + depd "2.0.0" + destroy "1.2.0" + encodeurl "~1.0.2" + escape-html "~1.0.3" + etag "~1.8.1" + fresh "0.5.2" + http-errors "2.0.0" + mime "1.6.0" + ms "2.1.3" + on-finished "2.4.1" + range-parser "~1.2.1" + statuses "2.0.1" + +serve-static@1.15.0: + version "1.15.0" + resolved "https://registry.yarnpkg.com/serve-static/-/serve-static-1.15.0.tgz#faaef08cffe0a1a62f60cad0c4e513cff0ac9540" + integrity sha512-XGuRDNjXUijsUL0vl6nSD7cwURuzEgglbOaFuZM9g3kwDXOWVTck0jLzjPzGD+TazWbboZYu52/9/XPdUgne9g== + dependencies: + encodeurl "~1.0.2" + escape-html "~1.0.3" + parseurl "~1.3.3" + send "0.18.0" + +set-blocking@^2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/set-blocking/-/set-blocking-2.0.0.tgz#045f9782d011ae9a6803ddd382b24392b3d890f7" + integrity sha512-KiKBS8AnWGEyLzofFfmvKwpdPzqiy16LvQfK3yv/fVH7Bj13/wl3JSR1J+rfgRE9q7xUJK4qvgS8raSOeLUehw== + +setimmediate@^1.0.5: + version "1.0.5" + resolved "https://registry.yarnpkg.com/setimmediate/-/setimmediate-1.0.5.tgz#290cbb232e306942d7d7ea9b83732ab7856f8285" + integrity sha512-MATJdZp8sLqDl/68LfQmbP8zKPLQNV6BIZoIgrscFDQ+RsvK/BxeDQOgyxKKoh0y/8h3BqVFnCqQ/gd+reiIXA== + +setprototypeof@1.2.0: + version "1.2.0" + resolved "https://registry.yarnpkg.com/setprototypeof/-/setprototypeof-1.2.0.tgz#66c9a24a73f9fc28cbe66b09fed3d33dcaf1b424" + integrity sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw== + +side-channel@^1.0.4: + version "1.0.4" + resolved "https://registry.yarnpkg.com/side-channel/-/side-channel-1.0.4.tgz#efce5c8fdc104ee751b25c58d4290011fa5ea2cf" + integrity sha512-q5XPytqFEIKHkGdiMIrY10mvLRvnQh42/+GoBlFW3b2LXLE2xxJpZFdm94we0BaoV3RwJyGqg5wS7epxTv0Zvw== + dependencies: + call-bind "^1.0.0" + get-intrinsic "^1.0.2" + object-inspect "^1.9.0" + +signal-exit@^3.0.0: + version "3.0.7" + resolved "https://registry.yarnpkg.com/signal-exit/-/signal-exit-3.0.7.tgz#a9a1767f8af84155114eaabd73f99273c8f59ad9" + integrity sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ== + +simple-update-notifier@^1.0.7: + version "1.1.0" + resolved "https://registry.yarnpkg.com/simple-update-notifier/-/simple-update-notifier-1.1.0.tgz#67694c121de354af592b347cdba798463ed49c82" + integrity sha512-VpsrsJSUcJEseSbMHkrsrAVSdvVS5I96Qo1QAQ4FxQ9wXFcB+pjj7FB7/us9+GcgfW4ziHtYMc1J0PLczb55mg== + dependencies: + semver "~7.0.0" + +slugify@^1.6.6: + version "1.6.6" + resolved "https://registry.yarnpkg.com/slugify/-/slugify-1.6.6.tgz#2d4ac0eacb47add6af9e04d3be79319cbcc7924b" + integrity sha512-h+z7HKHYXj6wJU+AnS/+IH8Uh9fdcX1Lrhg1/VMdf9PwoBQXFcXiAdsy2tSK0P6gKwJLXp02r90ahUCqHk9rrw== + +smart-buffer@^4.2.0: + version "4.2.0" + resolved "https://registry.yarnpkg.com/smart-buffer/-/smart-buffer-4.2.0.tgz#6e1d71fa4f18c05f7d0ff216dd16a481d0e8d9ae" + integrity sha512-94hK0Hh8rPqQl2xXc3HsaBoOXKV20MToPkcXvwbISWLEs+64sBq5kFgn2kJDHb1Pry9yrP0dxrCI9RRci7RXKg== + +socks-proxy-agent@^8.0.2: + version "8.0.2" + resolved "https://registry.yarnpkg.com/socks-proxy-agent/-/socks-proxy-agent-8.0.2.tgz#5acbd7be7baf18c46a3f293a840109a430a640ad" + integrity sha512-8zuqoLv1aP/66PHF5TqwJ7Czm3Yv32urJQHrVyhD7mmA6d61Zv8cIXQYPTWwmg6qlupnPvs/QKDmfa4P/qct2g== + dependencies: + agent-base "^7.0.2" + debug "^4.3.4" + socks "^2.7.1" + +socks@^2.7.1: + version "2.7.1" + resolved "https://registry.yarnpkg.com/socks/-/socks-2.7.1.tgz#d8e651247178fde79c0663043e07240196857d55" + integrity sha512-7maUZy1N7uo6+WVEX6psASxtNlKaNVMlGQKkG/63nEDdLOWNbiUMoLK7X4uYoLhQstau72mLgfEWcXcwsaHbYQ== + dependencies: + ip "^2.0.0" + smart-buffer "^4.2.0" + +source-map@~0.6.1: + version "0.6.1" + resolved "https://registry.yarnpkg.com/source-map/-/source-map-0.6.1.tgz#74722af32e9614e9c287a8d0bbde48b5e2f1a263" + integrity sha512-UjgapumWlbMhkBgzT7Ykc5YXUT46F0iKu8SGXq0bcwP5dz/h0Plj6enJqjz1Zbq2l5WaqYnrVbwWOWMyF3F47g== + +sprintf-js@~1.0.2: + version "1.0.3" + resolved "https://registry.yarnpkg.com/sprintf-js/-/sprintf-js-1.0.3.tgz#04e6926f662895354f3dd015203633b857297e2c" + integrity sha512-D9cPgkvLlV3t3IzL0D0YLvGA9Ahk4PcvVwUbN0dSGr1aP0Nrt4AEnTUbuGvquEC0mA64Gqt1fzirlRs5ibXx8g== + +statuses@2.0.1: + version "2.0.1" + resolved "https://registry.yarnpkg.com/statuses/-/statuses-2.0.1.tgz#55cb000ccf1d48728bd23c685a063998cf1a1b63" + integrity sha512-RwNA9Z/7PrK06rYLIzFMlaF+l73iwpzsqRIFgbMLbTcLD6cOao82TaWefPXQvB2fOC4AjuYSEndS7N/mTCbkdQ== + +streamsearch@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/streamsearch/-/streamsearch-1.1.0.tgz#404dd1e2247ca94af554e841a8ef0eaa238da764" + integrity sha512-Mcc5wHehp9aXz1ax6bZUyY5afg9u2rv5cqQI3mRrYkGC8rW2hM02jWuwjtL++LS5qinSyhj2QfLyNsuc+VsExg== + +streamx@^2.15.0: + version "2.15.6" + resolved "https://registry.yarnpkg.com/streamx/-/streamx-2.15.6.tgz#28bf36997ebc7bf6c08f9eba958735231b833887" + integrity sha512-q+vQL4AAz+FdfT137VF69Cc/APqUbxy+MDOImRrMvchJpigHj9GksgDU2LYbO9rx7RX6osWgxJB2WxhYv4SZAw== + dependencies: + fast-fifo "^1.1.0" + queue-tick "^1.0.1" + +"string-width@^1.0.2 || 2 || 3 || 4", string-width@^4.1.0, string-width@^4.2.0, string-width@^4.2.3: + version "4.2.3" + resolved "https://registry.yarnpkg.com/string-width/-/string-width-4.2.3.tgz#269c7117d27b05ad2e536830a8ec895ef9c6d010" + integrity sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g== + dependencies: + emoji-regex "^8.0.0" + is-fullwidth-code-point "^3.0.0" + strip-ansi "^6.0.1" + +string_decoder@^1.1.1: + version "1.3.0" + resolved "https://registry.yarnpkg.com/string_decoder/-/string_decoder-1.3.0.tgz#42f114594a46cf1a8e30b0a84f56c78c3edac21e" + integrity sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA== + dependencies: + safe-buffer "~5.2.0" + +string_decoder@~1.1.1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/string_decoder/-/string_decoder-1.1.1.tgz#9cf1611ba62685d7030ae9e4ba34149c3af03fc8" + integrity sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg== + dependencies: + safe-buffer "~5.1.0" + +strip-ansi@^6.0.0, strip-ansi@^6.0.1: + version "6.0.1" + resolved "https://registry.yarnpkg.com/strip-ansi/-/strip-ansi-6.0.1.tgz#9e26c63d30f53443e9489495b2105d37b67a85d9" + integrity sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A== + dependencies: + ansi-regex "^5.0.1" + +strip-dirs@^2.0.0: + version "2.1.0" + resolved "https://registry.yarnpkg.com/strip-dirs/-/strip-dirs-2.1.0.tgz#4987736264fc344cf20f6c34aca9d13d1d4ed6c5" + integrity sha512-JOCxOeKLm2CAS73y/U4ZeZPTkE+gNVCzKt7Eox84Iej1LT/2pTWYpZKJuxwQpvX1LiZb1xokNR7RLfuBAa7T3g== + dependencies: + is-natural-number "^4.0.1" + +strtok3@^6.2.4: + version "6.3.0" + resolved "https://registry.yarnpkg.com/strtok3/-/strtok3-6.3.0.tgz#358b80ffe6d5d5620e19a073aa78ce947a90f9a0" + integrity sha512-fZtbhtvI9I48xDSywd/somNqgUHl2L2cstmXCCif0itOf96jeW18MBSyrLuNicYQVkvpOxkZtkzujiTJ9LW5Jw== + dependencies: + "@tokenizer/token" "^0.3.0" + peek-readable "^4.1.0" + +supports-color@^5.3.0, supports-color@^5.5.0: + version "5.5.0" + resolved "https://registry.yarnpkg.com/supports-color/-/supports-color-5.5.0.tgz#e2e69a44ac8772f78a1ec0b35b689df6530efc8f" + integrity sha512-QjVjwdXIt408MIiAqCX4oUKsgU2EqAGzs2Ppkm4aQYbjm+ZEWEcW4SfFNTr4uMNZma0ey4f5lgLrkB0aX0QMow== + dependencies: + has-flag "^3.0.0" + +tar-fs@3.0.4: + version "3.0.4" + resolved "https://registry.yarnpkg.com/tar-fs/-/tar-fs-3.0.4.tgz#a21dc60a2d5d9f55e0089ccd78124f1d3771dbbf" + integrity sha512-5AFQU8b9qLfZCX9zp2duONhPmZv0hGYiBPJsyUdqMjzq/mqVpy/rEUSeHk1+YitmxugaptgBh5oDGU3VsAJq4w== + dependencies: + mkdirp-classic "^0.5.2" + pump "^3.0.0" + tar-stream "^3.1.5" + +tar-stream@^1.5.2: + version "1.6.2" + resolved "https://registry.yarnpkg.com/tar-stream/-/tar-stream-1.6.2.tgz#8ea55dab37972253d9a9af90fdcd559ae435c555" + integrity sha512-rzS0heiNf8Xn7/mpdSVVSMAWAoy9bfb1WOTYC78Z0UQKeKa/CWS8FOq0lKGNa8DWKAn9gxjCvMLYc5PGXYlK2A== + dependencies: + bl "^1.0.0" + buffer-alloc "^1.2.0" + end-of-stream "^1.0.0" + fs-constants "^1.0.0" + readable-stream "^2.3.0" + to-buffer "^1.1.1" + xtend "^4.0.0" + +tar-stream@^3.1.5: + version "3.1.6" + resolved "https://registry.yarnpkg.com/tar-stream/-/tar-stream-3.1.6.tgz#6520607b55a06f4a2e2e04db360ba7d338cc5bab" + integrity sha512-B/UyjYwPpMBv+PaFSWAmtYjwdrlEaZQEhMIBFNC5oEG8lpiW8XjcSdmEaClj28ArfKScKHs2nshz3k2le6crsg== + dependencies: + b4a "^1.6.4" + fast-fifo "^1.2.0" + streamx "^2.15.0" + +tar@^6.1.11: + version "6.1.15" + resolved "https://registry.yarnpkg.com/tar/-/tar-6.1.15.tgz#c9738b0b98845a3b344d334b8fa3041aaba53a69" + integrity sha512-/zKt9UyngnxIT/EAGYuxaMYgOIJiP81ab9ZfkILq4oNLPFX50qyYmu7jRj9qeXoxmJHjGlbH0+cm2uy1WCs10A== + dependencies: + chownr "^2.0.0" + fs-minipass "^2.0.0" + minipass "^5.0.0" + minizlib "^2.1.1" + mkdirp "^1.0.3" + yallist "^4.0.0" + +through@^2.3.8: + version "2.3.8" + resolved "https://registry.yarnpkg.com/through/-/through-2.3.8.tgz#0dd4c9ffaabc357960b1b724115d7e0e86a2e1f5" + integrity sha512-w89qg7PI8wAdvX60bMDP+bFoD5Dvhm9oLheFp5O4a2QF0cSBGsBX4qZmadPMvVqlLJBBci+WqGGOAPvcDeNSVg== + +tlds@1.240.0: + version "1.240.0" + resolved "https://registry.yarnpkg.com/tlds/-/tlds-1.240.0.tgz#3d3d776d97aa079e43ef4d2f9ef9845e55cff08e" + integrity sha512-1OYJQenswGZSOdRw7Bql5Qu7uf75b+F3HFBXbqnG/ifHa0fev1XcG+3pJf3pA/KC6RtHQzfKgIf1vkMlMG7mtQ== + +to-buffer@^1.1.1: + version "1.1.1" + resolved "https://registry.yarnpkg.com/to-buffer/-/to-buffer-1.1.1.tgz#493bd48f62d7c43fcded313a03dcadb2e1213a80" + integrity sha512-lx9B5iv7msuFYE3dytT+KE5tap+rNYw+K4jVkb9R/asAb+pbBSM17jtunHplhBe6RRJdZx3Pn2Jph24O32mOVg== + +to-regex-range@^5.0.1: + version "5.0.1" + resolved "https://registry.yarnpkg.com/to-regex-range/-/to-regex-range-5.0.1.tgz#1648c44aae7c8d988a326018ed72f5b4dd0392e4" + integrity sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ== + dependencies: + is-number "^7.0.0" + +toidentifier@1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/toidentifier/-/toidentifier-1.0.1.tgz#3be34321a88a820ed1bd80dfaa33e479fbb8dd35" + integrity sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA== + +token-types@^4.1.1: + version "4.2.1" + resolved "https://registry.yarnpkg.com/token-types/-/token-types-4.2.1.tgz#0f897f03665846982806e138977dbe72d44df753" + integrity sha512-6udB24Q737UD/SDsKAHI9FCRP7Bqc9D/MQUV02ORQg5iskjtLJlZJNdN4kKtcdtwCeWIwIHDGaUsTsCCAa8sFQ== + dependencies: + "@tokenizer/token" "^0.3.0" + ieee754 "^1.2.1" + +touch@^3.1.0: + version "3.1.0" + resolved "https://registry.yarnpkg.com/touch/-/touch-3.1.0.tgz#fe365f5f75ec9ed4e56825e0bb76d24ab74af83b" + integrity sha512-WBx8Uy5TLtOSRtIq+M03/sKDrXCLHxwDcquSP2c43Le03/9serjQBIztjRz6FkJez9D/hleyAXTBGLwwZUw9lA== + dependencies: + nopt "~1.0.10" + +tr46@~0.0.3: + version "0.0.3" + resolved "https://registry.yarnpkg.com/tr46/-/tr46-0.0.3.tgz#8184fd347dac9cdc185992f3a6622e14b9d9ab6a" + integrity sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw== + +tslib@^2.0.1: + version "2.6.2" + resolved "https://registry.yarnpkg.com/tslib/-/tslib-2.6.2.tgz#703ac29425e7b37cd6fd456e92404d46d1f3e4ae" + integrity sha512-AEYxH93jGFPn/a2iVAwW87VuUIkR1FVUKB77NwMF7nBTDkDrrT/Hpt/IrCJ0QXhW27jTBDcf5ZY7w6RiqTMw2Q== + +type-is@^1.6.4, type-is@~1.6.18: + version "1.6.18" + resolved "https://registry.yarnpkg.com/type-is/-/type-is-1.6.18.tgz#4e552cd05df09467dcbc4ef739de89f2cf37c131" + integrity sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g== + dependencies: + media-typer "0.3.0" + mime-types "~2.1.24" + +typedarray@^0.0.6: + version "0.0.6" + resolved "https://registry.yarnpkg.com/typedarray/-/typedarray-0.0.6.tgz#867ac74e3864187b1d3d47d996a78ec5c8830777" + integrity sha512-/aCDEGatGvZ2BIk+HmLf4ifCJFwvKFNb9/JeZPMulfgFracn9QFcAf5GO8B/mweUjSoblS5In0cWhqpfs/5PQA== + +uc.micro@^1.0.1: + version "1.0.6" + resolved "https://registry.yarnpkg.com/uc.micro/-/uc.micro-1.0.6.tgz#9c411a802a409a91fc6cf74081baba34b24499ac" + integrity sha512-8Y75pvTYkLJW2hWQHXxoqRgV7qb9B+9vFEtidML+7koHUFapnVJAZ6cKs+Qjz5Aw3aZWHMC6u0wJE3At+nSGwA== + +unbzip2-stream@1.4.3, unbzip2-stream@^1.0.9: + version "1.4.3" + resolved "https://registry.yarnpkg.com/unbzip2-stream/-/unbzip2-stream-1.4.3.tgz#b0da04c4371311df771cdc215e87f2130991ace7" + integrity sha512-mlExGW4w71ebDJviH16lQLtZS32VKqsSfk80GCfUlwT/4/hNRFsoscrF/c++9xinkMzECL1uL9DDwXqFWkruPg== + dependencies: + buffer "^5.2.1" + through "^2.3.8" + +undefsafe@^2.0.5: + version "2.0.5" + resolved "https://registry.yarnpkg.com/undefsafe/-/undefsafe-2.0.5.tgz#38733b9327bdcd226db889fb723a6efd162e6e2c" + integrity sha512-WxONCrssBM8TSPRqN5EmsjVrsv4A8X12J4ArBiiayv3DyyG3ZlIg6yysuuSYdZsVz3TKcTg2fd//Ujd4CHV1iA== + +underscore@^1.13.1: + version "1.13.6" + resolved "https://registry.yarnpkg.com/underscore/-/underscore-1.13.6.tgz#04786a1f589dc6c09f761fc5f45b89e935136441" + integrity sha512-+A5Sja4HP1M08MaXya7p5LvjuM7K6q/2EaC0+iovj/wOcMsTzMvDFbasi/oSapiwOlt252IqsKqPjCl7huKS0A== + +undici-types@~5.26.4: + version "5.26.5" + resolved "https://registry.yarnpkg.com/undici-types/-/undici-types-5.26.5.tgz#bcd539893d00b56e964fd2657a4866b221a65617" + integrity sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA== + +universalify@^0.1.0: + version "0.1.2" + resolved "https://registry.yarnpkg.com/universalify/-/universalify-0.1.2.tgz#b646f69be3942dabcecc9d6639c80dc105efaa66" + integrity sha512-rBJeI5CXAlmy1pV+617WB9J63U6XcazHHF2f2dbJix4XzpUF0RS3Zbj0FGIOCAva5P/d/GBOYaACQ1w+0azUkg== + +unpipe@1.0.0, unpipe@~1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/unpipe/-/unpipe-1.0.0.tgz#b2bf4ee8514aae6165b4817829d21b2ef49904ec" + integrity sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ== + +url-template@^2.0.8: + version "2.0.8" + resolved "https://registry.yarnpkg.com/url-template/-/url-template-2.0.8.tgz#fc565a3cccbff7730c775f5641f9555791439f21" + integrity sha512-XdVKMF4SJ0nP/O7XIPB0JwAEuT9lDIYnNsK8yGVe43y0AWoKeJNdv3ZNWh7ksJ6KqQFjOO6ox/VEitLnaVNufw== + +urlpattern-polyfill@9.0.0: + version "9.0.0" + resolved "https://registry.yarnpkg.com/urlpattern-polyfill/-/urlpattern-polyfill-9.0.0.tgz#bc7e386bb12fd7898b58d1509df21d3c29ab3460" + integrity sha512-WHN8KDQblxd32odxeIgo83rdVDE2bvdkb86it7bMhYZwWKJz0+O0RK/eZiHYnM+zgt/U7hAHOlCQGfjjvSkw2g== + +util-deprecate@^1.0.1, util-deprecate@~1.0.1: + version "1.0.2" + resolved "https://registry.yarnpkg.com/util-deprecate/-/util-deprecate-1.0.2.tgz#450d4dc9fa70de732762fbd2d4a28981419a0ccf" + integrity sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw== + +utils-merge@1.0.1: + version "1.0.1" + resolved "https://registry.yarnpkg.com/utils-merge/-/utils-merge-1.0.1.tgz#9f95710f50a267947b2ccc124741c1028427e713" + integrity sha512-pMZTvIkT1d+TFGvDOqodOclx0QWkkgi6Tdoa8gC8ffGAAqz9pzPTZWAybbsHHoED/ztMtkv/VoYTYyShUn81hA== + +uuid@^9.0.0: + version "9.0.0" + resolved "https://registry.yarnpkg.com/uuid/-/uuid-9.0.0.tgz#592f550650024a38ceb0c562f2f6aa435761efb5" + integrity sha512-MXcSTerfPa4uqyzStbRoTgt5XIe3x5+42+q1sDuy3R5MDk66URdLMOZe5aPX/SQd+kuYAh0FdP/pO28IkQyTeg== + +vary@^1, vary@~1.1.2: + version "1.1.2" + resolved "https://registry.yarnpkg.com/vary/-/vary-1.1.2.tgz#2299f02c6ded30d4a5961b0b9f74524a18f634fc" + integrity sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg== + +web-streams-polyfill@4.0.0-beta.3: + version "4.0.0-beta.3" + resolved "https://registry.yarnpkg.com/web-streams-polyfill/-/web-streams-polyfill-4.0.0-beta.3.tgz#2898486b74f5156095e473efe989dcf185047a38" + integrity sha512-QW95TCTaHmsYfHDybGMwO5IJIM93I/6vTRk+daHTWFPhwh+C8Cg7j7XyKrwrj8Ib6vYXe0ocYNrmzY4xAAN6ug== + +web-streams-polyfill@^3.2.1: + version "3.2.1" + resolved "https://registry.yarnpkg.com/web-streams-polyfill/-/web-streams-polyfill-3.2.1.tgz#71c2718c52b45fd49dbeee88634b3a60ceab42a6" + integrity sha512-e0MO3wdXWKrLbL0DgGnUV7WHVuw9OUvL4hjgnPkIeEvESk74gAITi5G606JtZPp39cd8HA9VQzCIvA49LpPN5Q== + +webidl-conversions@^3.0.0: + version "3.0.1" + resolved "https://registry.yarnpkg.com/webidl-conversions/-/webidl-conversions-3.0.1.tgz#24534275e2a7bc6be7bc86611cc16ae0a5654871" + integrity sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ== + +whatwg-url@^5.0.0: + version "5.0.0" + resolved "https://registry.yarnpkg.com/whatwg-url/-/whatwg-url-5.0.0.tgz#966454e8765462e37644d3626f6742ce8b70965d" + integrity sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw== + dependencies: + tr46 "~0.0.3" + webidl-conversions "^3.0.0" + +wide-align@^1.1.2: + version "1.1.5" + resolved "https://registry.yarnpkg.com/wide-align/-/wide-align-1.1.5.tgz#df1d4c206854369ecf3c9a4898f1b23fbd9d15d3" + integrity sha512-eDMORYaPNZ4sQIuuYPDHdQvf4gyCF9rEEV/yPxGfwPkRodwEgiMUUXTx/dex+Me0wxx53S+NgUHaP7y3MGlDmg== + dependencies: + string-width "^1.0.2 || 2 || 3 || 4" + +wrap-ansi@^7.0.0: + version "7.0.0" + resolved "https://registry.yarnpkg.com/wrap-ansi/-/wrap-ansi-7.0.0.tgz#67e145cff510a6a6984bdf1152911d69d2eb9e43" + integrity sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q== + dependencies: + ansi-styles "^4.0.0" + string-width "^4.1.0" + strip-ansi "^6.0.0" + +wrappy@1: + version "1.0.2" + resolved "https://registry.yarnpkg.com/wrappy/-/wrappy-1.0.2.tgz#b5243d8f3ec1aa35f1364605bc0d1036e30ab69f" + integrity sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ== + +ws@8.15.1: + version "8.15.1" + resolved "https://registry.yarnpkg.com/ws/-/ws-8.15.1.tgz#271ba33a45ca0cc477940f7f200cd7fba7ee1997" + integrity sha512-W5OZiCjXEmk0yZ66ZN82beM5Sz7l7coYxpRkzS+p9PP+ToQry8szKh+61eNktr7EA9DOwvFGhfC605jDHbP6QQ== + +xmlbuilder@^10.0.0: + version "10.1.1" + resolved "https://registry.yarnpkg.com/xmlbuilder/-/xmlbuilder-10.1.1.tgz#8cae6688cc9b38d850b7c8d3c0a4161dcaf475b0" + integrity sha512-OyzrcFLL/nb6fMGHbiRDuPup9ljBycsdCypwuyg5AAHvyWzGfChJpCXMG88AGTIMFhGZ9RccFN1e6lhg3hkwKg== + +xtend@^4.0.0: + version "4.0.2" + resolved "https://registry.yarnpkg.com/xtend/-/xtend-4.0.2.tgz#bb72779f5fa465186b1f438f674fa347fdb5db54" + integrity sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ== + +y18n@^5.0.5: + version "5.0.8" + resolved "https://registry.yarnpkg.com/y18n/-/y18n-5.0.8.tgz#7f4934d0f7ca8c56f95314939ddcd2dd91ce1d55" + integrity sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA== + +yallist@^4.0.0: + version "4.0.0" + resolved "https://registry.yarnpkg.com/yallist/-/yallist-4.0.0.tgz#9bb92790d9c0effec63be73519e11a35019a3a72" + integrity sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A== + +yaml@^2.2.1: + version "2.3.1" + resolved "https://registry.yarnpkg.com/yaml/-/yaml-2.3.1.tgz#02fe0975d23cd441242aa7204e09fc28ac2ac33b" + integrity sha512-2eHWfjaoXgTBC2jNM1LRef62VQa0umtvRiDSk6HSzW7RvS5YtkabJrwYLLEKWBc8a5U2PTSCs+dJjUTJdlHsWQ== + +yargs-parser@^21.1.1: + version "21.1.1" + resolved "https://registry.yarnpkg.com/yargs-parser/-/yargs-parser-21.1.1.tgz#9096bceebf990d21bb31fa9516e0ede294a77d35" + integrity sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw== + +yargs@17.7.2: + version "17.7.2" + resolved "https://registry.yarnpkg.com/yargs/-/yargs-17.7.2.tgz#991df39aca675a192b816e1e0363f9d75d2aa269" + integrity sha512-7dSzzRQ++CKnNI/krKnYRV7JKKPUXMEh61soaHKg9mrWEhzFWhFnxPxGl+69cD1Ou63C13NUPCnmIcrvqCuM6w== + dependencies: + cliui "^8.0.1" + escalade "^3.1.1" + get-caller-file "^2.0.5" + require-directory "^2.1.1" + string-width "^4.2.3" + y18n "^5.0.5" + yargs-parser "^21.1.1" + +yauzl@^2.10.0, yauzl@^2.4.2: + version "2.10.0" + resolved "https://registry.yarnpkg.com/yauzl/-/yauzl-2.10.0.tgz#c7eb17c93e112cb1086fa6d8e51fb0667b79a5f9" + integrity sha512-p4a9I6X6nu6IhoGmBqAcbJy1mlC4j27vEPZX9F4L4/vZT3Lyq1VkFHw/V/PUcB9Buo+DG3iHkT0x3Qya58zc3g== + dependencies: + buffer-crc32 "~0.2.3" + fd-slicer "~1.1.0" + +zod-to-json-schema@3.20.3: + version "3.20.3" + resolved "https://registry.yarnpkg.com/zod-to-json-schema/-/zod-to-json-schema-3.20.3.tgz#8c95d8c20f20455ffa0b4b526c29703f35f6d787" + integrity sha512-/Q3wnyxAfCt94ZcrGiXXoiAfRqasxl9CX64LZ9fj+4dKH68zulUtU0uk1WMxQPfAxQ0ZI70dKzcoW7hHj+DwSQ== + +zod@^3.22.3: + version "3.22.4" + resolved "https://registry.yarnpkg.com/zod/-/zod-3.22.4.tgz#f31c3a9386f61b1f228af56faa9255e845cf3fff" + integrity sha512-iC+8Io04lddc+mVqQ9AZ7OQ2MrUKGN+oIQyq1vemgt46jwCwLfhq7/pwnBnNXXXZb8VTVLKwp9EDkx+ryxIWmg== diff --git a/docker/Dockerfile b/docker/Dockerfile index 87914416..69dc4fe6 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -8,7 +8,7 @@ ARG ARG_GID=1000 # Install system dependencies RUN DEBIAN_FRONTEND=noninteractive apt-get update && \ DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \ - curl gnupg libgfortran5 python3 python3-pip tzdata netcat \ + curl gnupg libgfortran5 libgbm1 tzdata netcat \ libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 \ libgcc1 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libx11-6 libx11-xcb1 libxcb1 \ libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 \ @@ -21,13 +21,7 @@ RUN DEBIAN_FRONTEND=noninteractive apt-get update && \ apt-get install -yq --no-install-recommends nodejs && \ curl -LO https://github.com/yarnpkg/yarn/releases/download/v1.22.19/yarn_1.22.19_all.deb \ && dpkg -i yarn_1.22.19_all.deb \ - && rm yarn_1.22.19_all.deb && \ - curl -LO https://github.com/jgm/pandoc/releases/download/3.1.3/pandoc-3.1.3-1-amd64.deb \ - && dpkg -i pandoc-3.1.3-1-amd64.deb \ - && rm pandoc-3.1.3-1-amd64.deb && \ - rm -rf /var/lib/apt/lists/* /usr/share/icons && \ - dpkg-reconfigure -f noninteractive tzdata && \ - python3 -m pip install --no-cache-dir virtualenv + && rm yarn_1.22.19_all.deb # Create a group and user with specific UID and GID RUN groupadd -g $ARG_GID anythingllm && \ @@ -81,10 +75,7 @@ COPY --from=build-stage /app/frontend/dist ./server/public COPY --chown=anythingllm:anythingllm ./collector/ ./collector/ # Install collector dependencies -RUN cd /app/collector && \ - python3 -m virtualenv v-env && \ - . v-env/bin/activate && \ - pip install --no-cache-dir -r requirements.txt +RUN cd /app/collector && yarn install --production && yarn cache clean # Migrate and Run Prisma against known schema RUN cd ./server && npx prisma generate --schema=./prisma/schema.prisma @@ -92,7 +83,6 @@ RUN cd ./server && npx prisma migrate deploy --schema=./prisma/schema.prisma # Setup the environment ENV NODE_ENV=production -ENV PATH=/app/collector/v-env/bin:$PATH # Expose the server port EXPOSE 3001 diff --git a/docker/HOW_TO_USE_DOCKER.md b/docker/HOW_TO_USE_DOCKER.md index 6818a5f7..85434958 100644 --- a/docker/HOW_TO_USE_DOCKER.md +++ b/docker/HOW_TO_USE_DOCKER.md @@ -24,6 +24,7 @@ export STORAGE_LOCATION="/var/lib/anythingllm" && \ mkdir -p $STORAGE_LOCATION && \ touch "$STORAGE_LOCATION/.env" && \ docker run -d -p 3001:3001 \ +--cap-add SYS_ADMIN \ -v ${STORAGE_LOCATION}:/app/server/storage \ -v ${STORAGE_LOCATION}/.env:/app/server/.env \ -e STORAGE_DIR="/app/server/storage" \ @@ -45,16 +46,6 @@ Your docker host will show the image as online once the build process is complet ## How to use the user interface - To access the full application, visit `http://localhost:3001` in your browser. -## How to add files to my system using the standalone scripts -- Upload files from the UI in your Workspace settings - -- To run the collector scripts to grab external data (articles, URLs, etc.) - - `docker exec -it --workdir=/app/collector anything-llm python main.py` - -- To run the collector watch script to process files from the hotdir - - `docker exec -it --workdir=/app/collector anything-llm python watch.py` - - Upload [compliant files](../collector/hotdir/__HOTDIR__.md) to `./collector/hotdir` and they will be processed and made available in the UI. - ## About UID and GID in the ENV - The UID and GID are set to 1000 by default. This is the default user in the Docker container and on most host operating systems. If there is a mismatch between your host user UID and GID and what is set in the `.env` file, you may experience permission issues. diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml index 20d17dbb..5ec67cab 100644 --- a/docker/docker-compose.yml +++ b/docker/docker-compose.yml @@ -17,6 +17,8 @@ services: args: ARG_UID: ${UID:-1000} ARG_GID: ${GID:-1000} + cap_add: + - SYS_ADMIN volumes: - "./.env:/app/server/.env" - "../server/storage:/app/server/storage" diff --git a/docker/docker-entrypoint.sh b/docker/docker-entrypoint.sh index 4c264e88..3d890db1 100755 --- a/docker/docker-entrypoint.sh +++ b/docker/docker-entrypoint.sh @@ -4,6 +4,6 @@ npx prisma migrate deploy --schema=./prisma/schema.prisma &&\ node /app/server/index.js } & -{ FLASK_ENV=production FLASK_APP=wsgi.py cd collector && gunicorn --timeout 300 --workers 4 --bind 0.0.0.0:8888 wsgi:api; } & +{ node /app/collector/index.js; } & wait -n exit $? \ No newline at end of file diff --git a/package.json b/package.json index 880ab612..830a20df 100644 --- a/package.json +++ b/package.json @@ -9,10 +9,11 @@ "node": ">=18" }, "scripts": { - "lint": "cd server && yarn lint && cd .. && cd frontend && yarn lint", - "setup": "cd server && yarn && cd ../frontend && yarn && cd .. && yarn setup:envs && yarn prisma:setup && echo \"Please run yarn dev:server and yarn dev:frontend in separate terminal tabs.\"", + "lint": "cd server && yarn lint && cd ../frontend && yarn lint && cd ../collector && yarn lint", + "setup": "cd server && yarn && cd ../collector && yarn && cd ../frontend && yarn && cd .. && yarn setup:envs && yarn prisma:setup && echo \"Please run yarn dev:server, yarn dev:collector, and yarn dev:frontend in separate terminal tabs.\"", "setup:envs": "cp -n ./frontend/.env.example ./frontend/.env && cp -n ./server/.env.example ./server/.env.development && cp -n ./collector/.env.example ./collector/.env && cp -n ./docker/.env.example ./docker/.env && echo \"All ENV files copied!\n\"", "dev:server": "cd server && yarn dev", + "dev:collector": "cd collector && yarn dev", "dev:frontend": "cd frontend && yarn start", "prisma:generate": "cd server && npx prisma generate", "prisma:migrate": "cd server && npx prisma migrate dev --name init", diff --git a/server/endpoints/api/document/index.js b/server/endpoints/api/document/index.js index 28de98a7..a813e2df 100644 --- a/server/endpoints/api/document/index.js +++ b/server/endpoints/api/document/index.js @@ -2,7 +2,7 @@ const { Telemetry } = require("../../../models/telemetry"); const { validApiKey } = require("../../../utils/middleware/validApiKey"); const { setupMulter } = require("../../../utils/files/multer"); const { - checkPythonAppAlive, + checkProcessorAlive, acceptedFileTypes, processDocument, } = require("../../../utils/files/documentProcessor"); @@ -60,14 +60,14 @@ function apiDocumentEndpoints(app) { */ try { const { originalname } = request.file; - const processingOnline = await checkPythonAppAlive(); + const processingOnline = await checkProcessorAlive(); if (!processingOnline) { response .status(500) .json({ success: false, - error: `Python processing API is not online. Document ${originalname} will not be processed automatically.`, + error: `Document processing API is not online. Document ${originalname} will not be processed automatically.`, }) .end(); } diff --git a/server/endpoints/system.js b/server/endpoints/system.js index 024bdd99..790305cf 100644 --- a/server/endpoints/system.js +++ b/server/endpoints/system.js @@ -4,7 +4,7 @@ process.env.NODE_ENV === "development" const { viewLocalFiles } = require("../utils/files"); const { exportData, unpackAndOverwriteImport } = require("../utils/files/data"); const { - checkPythonAppAlive, + checkProcessorAlive, acceptedFileTypes, } = require("../utils/files/documentProcessor"); const { purgeDocument } = require("../utils/files/purgeDocument"); @@ -221,7 +221,7 @@ function systemEndpoints(app) { [validatedRequest], async (_, response) => { try { - const online = await checkPythonAppAlive(); + const online = await checkProcessorAlive(); response.sendStatus(online ? 200 : 503); } catch (e) { console.log(e.message, e); diff --git a/server/endpoints/workspaces.js b/server/endpoints/workspaces.js index 3170eb1a..7873ef76 100644 --- a/server/endpoints/workspaces.js +++ b/server/endpoints/workspaces.js @@ -7,7 +7,7 @@ const { convertToChatHistory } = require("../utils/chats"); const { getVectorDbClass } = require("../utils/helpers"); const { setupMulter } = require("../utils/files/multer"); const { - checkPythonAppAlive, + checkProcessorAlive, processDocument, processLink, } = require("../utils/files/documentProcessor"); @@ -82,14 +82,14 @@ function workspaceEndpoints(app) { handleUploads.single("file"), async function (request, response) { const { originalname } = request.file; - const processingOnline = await checkPythonAppAlive(); + const processingOnline = await checkProcessorAlive(); if (!processingOnline) { response .status(500) .json({ success: false, - error: `Python processing API is not online. Document ${originalname} will not be processed automatically.`, + error: `Document processing API is not online. Document ${originalname} will not be processed automatically.`, }) .end(); return; @@ -114,14 +114,14 @@ function workspaceEndpoints(app) { [validatedRequest], async (request, response) => { const { link = "" } = reqBody(request); - const processingOnline = await checkPythonAppAlive(); + const processingOnline = await checkProcessorAlive(); if (!processingOnline) { response .status(500) .json({ success: false, - error: `Python processing API is not online. Link ${link} will not be processed automatically.`, + error: `Document processing API is not online. Link ${link} will not be processed automatically.`, }) .end(); return; diff --git a/server/utils/files/documentProcessor.js b/server/utils/files/documentProcessor.js index c1a17238..ea47a158 100644 --- a/server/utils/files/documentProcessor.js +++ b/server/utils/files/documentProcessor.js @@ -2,15 +2,15 @@ // of docker this endpoint is not exposed so it is only on the Docker instances internal network // so no additional security is needed on the endpoint directly. Auth is done however by the express // middleware prior to leaving the node-side of the application so that is good enough >:) -const PYTHON_API = "http://0.0.0.0:8888"; -async function checkPythonAppAlive() { - return await fetch(`${PYTHON_API}`) +const PROCESSOR_API = "http://0.0.0.0:8888"; +async function checkProcessorAlive() { + return await fetch(`${PROCESSOR_API}`) .then((res) => res.ok) .catch((e) => false); } async function acceptedFileTypes() { - return await fetch(`${PYTHON_API}/accepts`) + return await fetch(`${PROCESSOR_API}/accepts`) .then((res) => { if (!res.ok) throw new Error("Could not reach"); return res.json(); @@ -21,7 +21,7 @@ async function acceptedFileTypes() { async function processDocument(filename = "") { if (!filename) return false; - return await fetch(`${PYTHON_API}/process`, { + return await fetch(`${PROCESSOR_API}/process`, { method: "POST", headers: { "Content-Type": "application/json", @@ -41,7 +41,7 @@ async function processDocument(filename = "") { async function processLink(link = "") { if (!link) return false; - return await fetch(`${PYTHON_API}/process-link`, { + return await fetch(`${PROCESSOR_API}/process-link`, { method: "POST", headers: { "Content-Type": "application/json", @@ -60,7 +60,7 @@ async function processLink(link = "") { } module.exports = { - checkPythonAppAlive, + checkProcessorAlive, processDocument, processLink, acceptedFileTypes,