chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,73 @@
|
||||
# AnythingLLM Development Container Setup
|
||||
|
||||
Welcome to the AnythingLLM development container configuration, designed to create a seamless and feature-rich development environment for this project.
|
||||
|
||||
<center><h1><b>PLEASE READ THIS</b></h1></center>
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [Docker](https://www.docker.com/get-started)
|
||||
- [Visual Studio Code](https://code.visualstudio.com/)
|
||||
- [Remote - Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) VS Code extension
|
||||
|
||||
## Features
|
||||
|
||||
- **Base Image**: Built on `mcr.microsoft.com/devcontainers/javascript-node:1-18-bookworm`, thus Node.JS LTS v18.
|
||||
- **Additional Tools**: Includes `hadolint`, and essential apt-packages such as `curl`, `gnupg`, and more.
|
||||
- **Ports**: Configured to auto-forward ports `3000` (Frontend) and `3001` (Backend).
|
||||
- **Environment Variables**: Sets `NODE_ENV` to `development` and `ESLINT_USE_FLAT_CONFIG` to `true`.
|
||||
- **VS Code Extensions**: A suite of extensions such as `Prettier`, `Docker`, `ESLint`, and more are automatically installed. Please revise if you do not agree with any of these extensions. AI-powered extensions and time trackers are (for now) not included to avoid any privacy concerns, but you can install them later in your own environment.
|
||||
|
||||
## Getting Started
|
||||
|
||||
1. Using GitHub Codespaces. Just select to create a new workspace, and the devcontainer will be created for you.
|
||||
|
||||
2. Using your Local VSCode (Release or Insiders). We suggest you first make a fork of the repo and then clone it to your local machine using VSCode tools. Then open the project folder in VSCode, which will prompt you to open the project in a devcontainer. Select yes, and the devcontainer will be created for you. If this does not happen, you can open the command palette and select "Remote-Containers: Reopen in Container".
|
||||
|
||||
## On Creation:
|
||||
|
||||
When the container is built for the first time, it will automatically run `yarn setup` to ensure everything is in place for the Collector, Server and Frontend. This command is expected to be automatically re-run if there is a content change on next reboot.
|
||||
|
||||
## Work in the Container:
|
||||
|
||||
Once the container is up, be patient. Some extensions may complain because dependencies are still being installed, and in the Extensions tab, some may ask you to "Reload" the project. Don't do that yet. First, wait until all settle down for the first time. We suggest you create a new VSCode profile for this devcontainer, so any configuration and extensions you change, won't affect your default profile.
|
||||
|
||||
Checklist:
|
||||
|
||||
- [ ] The usual message asking you to start the Server and Frontend in different windows are now "hidden" in the building process of the devcontainer. Don't forget to do as suggested.
|
||||
- [ ] Open a JavaScript file, for example "server/index.js" and check if `eslint` is working. It will complain that `'err' is defined but never used.`. This means it is working.
|
||||
- [ ] Open a React File, for example, "frontend/src/main.jsx," and check if `eslint` complains about `Fast refresh only works when a file has exports. Move your component(s) to a separate file.`. Again, it means `eslint` is working. Now check at the status bar if the `Prettier` has a double checkmark :heavy_check_mark: (double). It means Prettier is working. You will see a nice extension `Formatting:`:heavy_check_mark: that can be used to disable the `Format on Save` feature temporarily.
|
||||
- [ ] Check if, on the left pane, you have the NPM Scripts (this may be disabled; look at the "Explorer" tree-dots up-right). There will be scripts inside the `package.json` files. You will basically need to run the `dev:collector`, `dev:server` and the `dev:frontend` in this order. When the frontend finishes starting, a window browser will open **inside** the VSCode. Still, you can open it outside.
|
||||
|
||||
:warning: **Important for all developers** :warning:
|
||||
|
||||
- [ ] When you are using the `NODE_ENV=development` the server will not store the configurations you set for security reasons. Please set the proper config on file `.env.development`. The side-effect if you don't, everytime you restart the server, you will be sent to the "Onboarding" page again.
|
||||
|
||||
**Note when using GitHub Codespaces**
|
||||
|
||||
- [ ] When running the "Server" for the first time, it will automatically configure its port to be publicly accessible by default, as this is required for the front end to reach the server backend. To know more, read the content of the `.env` file on the frontend folder about this, and if any issues occur, make sure to manually set the port "Visibility" of the "Server" is set to "Public" if needed. Again, this is only needed for developing on GitHub Codespaces.
|
||||
|
||||
|
||||
**For the Collector:**
|
||||
|
||||
- [x] In the past, the Collector dwelled within the Python domain, but now it has journeyed to the splendid realm of Node.JS. Consequently, the configuration complexities of bygone versions are no longer a concern.
|
||||
|
||||
### Now it is ready to start
|
||||
|
||||
In the status bar you will see three shortcuts names `Collector`, `Server` and `Frontend`. Just click-and-wait on that order (don't forget to set the Server port 3001 to Public if you are using GH Codespaces **_before_** starting the Frontend).
|
||||
|
||||
Now you can enjoy your time developing instead of reconfiguring everything.
|
||||
|
||||
## Debugging with the devcontainers
|
||||
|
||||
### For debugging the collector, server and frontend
|
||||
|
||||
First, make sure the built-in extension (ms-vscode.js-debug) is active (I don't know why it would not be, but just in case). If you want, you can install the nightly version (ms-vscode.js-debug-nightly)
|
||||
|
||||
Then, in the "Run and Debug" tab (Ctrl+shift+D), you can select on the menu:
|
||||
|
||||
- Collector debug. This will start the collector in debug mode and attach the debugger. Works very well.
|
||||
- Server debug. This will start the server in debug mode and attach the debugger. Works very well.
|
||||
- Frontend debug. This will start the frontend in debug mode and attach the debugger. I am still struggling with this one. I don't know if VSCode can handle the .jsx files seamlessly as the pure .js on the server. Maybe there is a need for a particular configuration for Vite or React. Anyway, it starts. Another two configurations launch Chrome and Edge, and I think we could add breakpoints on .jsx files somehow. The best scenario would be always to use the embedded browser. WIP.
|
||||
|
||||
Please leave comments on the Issues tab or the []("https://discord.gg/6UyHPeGZAC")
|
||||
@@ -0,0 +1,214 @@
|
||||
// For format details, see https://aka.ms/devcontainer.json. For config options, see the
|
||||
// README at: https://github.com/devcontainers/templates/tree/main/src/javascript-node
|
||||
{
|
||||
"name": "Node.js",
|
||||
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
|
||||
// "build": {
|
||||
// "args": {
|
||||
// "ARG_UID": "1000",
|
||||
// "ARG_GID": "1000"
|
||||
// },
|
||||
// "dockerfile": "Dockerfile"
|
||||
// },
|
||||
// "containerUser": "anythingllm",
|
||||
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
|
||||
"image": "mcr.microsoft.com/devcontainers/javascript-node:1-18-bookworm",
|
||||
|
||||
"forwardPorts": [3001, 3000],
|
||||
|
||||
// Features to add to the dev container. More info: https://containers.dev/features.
|
||||
"features": {
|
||||
// Docker very useful linter
|
||||
"ghcr.io/dhoeric/features/hadolint:1": {
|
||||
"version": "latest"
|
||||
},
|
||||
// Terraform support
|
||||
"ghcr.io/devcontainers/features/terraform:1": {},
|
||||
// Just a wrap to install needed packages
|
||||
"ghcr.io/devcontainers-contrib/features/apt-packages:1": {
|
||||
// Dependencies copied from ../docker/Dockerfile plus some dev stuff
|
||||
"packages": [
|
||||
"build-essential",
|
||||
"ca-certificates",
|
||||
"curl",
|
||||
"ffmpeg",
|
||||
"fonts-liberation",
|
||||
"git",
|
||||
"gnupg",
|
||||
"htop",
|
||||
"less",
|
||||
"libappindicator1",
|
||||
"libasound2",
|
||||
"libatk-bridge2.0-0",
|
||||
"libatk1.0-0",
|
||||
"libc6",
|
||||
"libcairo2",
|
||||
"libcups2",
|
||||
"libdbus-1-3",
|
||||
"libexpat1",
|
||||
"libfontconfig1",
|
||||
"libgbm1",
|
||||
"libgcc1",
|
||||
"libgfortran5",
|
||||
"libglib2.0-0",
|
||||
"libgtk-3-0",
|
||||
"libnspr4",
|
||||
"libnss3",
|
||||
"libpango-1.0-0",
|
||||
"libpangocairo-1.0-0",
|
||||
"libstdc++6",
|
||||
"libx11-6",
|
||||
"libx11-xcb1",
|
||||
"libxcb1",
|
||||
"libxcomposite1",
|
||||
"libxcursor1",
|
||||
"libxdamage1",
|
||||
"libxext6",
|
||||
"libxfixes3",
|
||||
"libxi6",
|
||||
"libxrandr2",
|
||||
"libxrender1",
|
||||
"libxss1",
|
||||
"libxtst6",
|
||||
"locales",
|
||||
"lsb-release",
|
||||
"procps",
|
||||
"tzdata",
|
||||
"wget",
|
||||
"xdg-utils"
|
||||
]
|
||||
}
|
||||
},
|
||||
"updateContentCommand": "cd server && yarn && cd ../collector && PUPPETEER_DOWNLOAD_BASE_URL=https://storage.googleapis.com/chrome-for-testing-public yarn && cd ../frontend && yarn && cd .. && yarn setup:envs && yarn prisma:setup && echo \"Please run yarn dev:server, yarn dev:collector, and yarn dev:frontend in separate terminal tabs.\"",
|
||||
// Use 'postCreateCommand' to run commands after the container is created.
|
||||
// This configures VITE for github codespaces and installs gh cli
|
||||
"postCreateCommand": "if [ \"${CODESPACES}\" = \"true\" ]; then echo 'VITE_API_BASE=\"https://$CODESPACE_NAME-3001.$GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN/api\"' > ./frontend/.env && (type -p wget >/dev/null || (sudo apt update && sudo apt-get install wget -y)) && sudo mkdir -p -m 755 /etc/apt/keyrings && wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null && sudo chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg && echo \"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main\" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null && sudo apt update && sudo apt install gh -y; fi",
|
||||
"portsAttributes": {
|
||||
"3001": {
|
||||
"label": "Backend",
|
||||
"onAutoForward": "notify"
|
||||
},
|
||||
"3000": {
|
||||
"label": "Frontend",
|
||||
"onAutoForward": "openPreview"
|
||||
}
|
||||
},
|
||||
"capAdd": [
|
||||
"SYS_ADMIN" // needed for puppeteer using headless chrome in sandbox
|
||||
],
|
||||
"remoteEnv": {
|
||||
"NODE_ENV": "development",
|
||||
"ESLINT_USE_FLAT_CONFIG": "true",
|
||||
"ANYTHING_LLM_RUNTIME": "docker"
|
||||
},
|
||||
// "initializeCommand": "echo Initialize....",
|
||||
"shutdownAction": "stopContainer",
|
||||
// Configure tool-specific properties.
|
||||
"customizations": {
|
||||
"codespaces": {
|
||||
"openFiles": [
|
||||
"README.md",
|
||||
".devcontainer/README.md"
|
||||
]
|
||||
},
|
||||
"vscode": {
|
||||
"openFiles": [
|
||||
"README.md",
|
||||
".devcontainer/README.md"
|
||||
],
|
||||
"extensions": [
|
||||
"bierner.github-markdown-preview",
|
||||
"bradlc.vscode-tailwindcss",
|
||||
"dbaeumer.vscode-eslint",
|
||||
"editorconfig.editorconfig",
|
||||
"esbenp.prettier-vscode",
|
||||
"exiasr.hadolint",
|
||||
"flowtype.flow-for-vscode",
|
||||
"gamunu.vscode-yarn",
|
||||
"hashicorp.terraform",
|
||||
"mariusschulz.yarn-lock-syntax",
|
||||
"ms-azuretools.vscode-docker",
|
||||
"streetsidesoftware.code-spell-checker",
|
||||
"actboy168.tasks",
|
||||
"tombonnike.vscode-status-bar-format-toggle",
|
||||
"ms-vscode.js-debug"
|
||||
],
|
||||
"settings": {
|
||||
"[css]": {
|
||||
"editor.defaultFormatter": "esbenp.prettier-vscode"
|
||||
},
|
||||
"[dockercompose]": {
|
||||
"editor.defaultFormatter": "esbenp.prettier-vscode"
|
||||
},
|
||||
"[dockerfile]": {
|
||||
"editor.defaultFormatter": "ms-azuretools.vscode-docker"
|
||||
},
|
||||
"[html]": {
|
||||
"editor.defaultFormatter": "esbenp.prettier-vscode"
|
||||
},
|
||||
"[javascript]": {
|
||||
"editor.defaultFormatter": "esbenp.prettier-vscode"
|
||||
},
|
||||
"[javascriptreact]": {
|
||||
"editor.defaultFormatter": "esbenp.prettier-vscode"
|
||||
},
|
||||
"[json]": {
|
||||
"editor.defaultFormatter": "esbenp.prettier-vscode"
|
||||
},
|
||||
"[jsonc]": {
|
||||
"editor.defaultFormatter": "esbenp.prettier-vscode"
|
||||
},
|
||||
"[markdown]": {
|
||||
"editor.defaultFormatter": "esbenp.prettier-vscode"
|
||||
},
|
||||
"[postcss]": {
|
||||
"editor.defaultFormatter": "esbenp.prettier-vscode"
|
||||
},
|
||||
"[toml]": {
|
||||
"editor.defaultFormatter": "tamasfe.even-better-toml"
|
||||
},
|
||||
"eslint.debug": true,
|
||||
"eslint.enable": true,
|
||||
"eslint.experimental.useFlatConfig": true,
|
||||
"eslint.run": "onSave",
|
||||
"files.associations": {
|
||||
".*ignore": "ignore",
|
||||
".editorconfig": "editorconfig",
|
||||
".env*": "properties",
|
||||
".flowconfig": "ini",
|
||||
".prettierrc": "json",
|
||||
"*.css": "tailwindcss",
|
||||
"*.md": "markdown",
|
||||
"*.sh": "shellscript",
|
||||
"docker-compose.*": "dockercompose",
|
||||
"Dockerfile*": "dockerfile",
|
||||
"yarn.lock": "yarnlock"
|
||||
},
|
||||
"javascript.format.enable": false,
|
||||
"javascript.inlayHints.enumMemberValues.enabled": true,
|
||||
"javascript.inlayHints.functionLikeReturnTypes.enabled": true,
|
||||
"javascript.inlayHints.parameterTypes.enabled": true,
|
||||
"javascript.inlayHints.variableTypes.enabled": true,
|
||||
"js/ts.implicitProjectConfig.module": "CommonJS",
|
||||
"json.format.enable": false,
|
||||
"json.schemaDownload.enable": true,
|
||||
"npm.autoDetect": "on",
|
||||
"npm.packageManager": "yarn",
|
||||
"prettier.useEditorConfig": false,
|
||||
"tailwindCSS.files.exclude": [
|
||||
"**/.git/**",
|
||||
"**/node_modules/**",
|
||||
"**/.hg/**",
|
||||
"**/.svn/**",
|
||||
"**/dist/**"
|
||||
],
|
||||
"typescript.validate.enable": false,
|
||||
"workbench.editorAssociations": {
|
||||
"*.md": "vscode.markdown.preview.editor"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
|
||||
// "remoteUser": "root"
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
**/server/utils/agents/aibitat/example/**
|
||||
**/server/storage/documents/**
|
||||
**/server/storage/vector-cache/**
|
||||
**/server/storage/*.db
|
||||
**/server/storage/lancedb
|
||||
**/collector/hotdir/**
|
||||
**/collector/outputs/**
|
||||
**/node_modules/
|
||||
**/dist/
|
||||
**/v-env/
|
||||
**/__pycache__/
|
||||
**/.env
|
||||
**/.env.*
|
||||
**/bundleinspector.html
|
||||
**/tmp/**
|
||||
**/.log
|
||||
!docker/.env.example
|
||||
!frontend/.env.production
|
||||
@@ -0,0 +1,17 @@
|
||||
# EditorConfig is awesome: https://EditorConfig.org
|
||||
|
||||
# top-most EditorConfig file
|
||||
root = true
|
||||
|
||||
[*]
|
||||
# Non-configurable Prettier behaviors
|
||||
charset = utf-8
|
||||
insert_final_newline = true
|
||||
trim_trailing_whitespace = true
|
||||
|
||||
# Configurable Prettier behaviors
|
||||
# (change these if your Prettier config differs)
|
||||
end_of_line = lf
|
||||
indent_style = space
|
||||
indent_size = 2
|
||||
max_line_length = 80
|
||||
@@ -0,0 +1 @@
|
||||
* text=auto eol=lf
|
||||
@@ -0,0 +1 @@
|
||||
github: Mintplex-Labs
|
||||
@@ -0,0 +1,60 @@
|
||||
name: 🐛 Bug Report
|
||||
description: File a bug report for AnythingLLM
|
||||
title: "[BUG]: "
|
||||
labels: [possible bug]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Use this template to file a bug report for AnythingLLM. Please be as descriptive as possible to allow everyone to replicate and solve your issue.
|
||||
- type: dropdown
|
||||
id: runtime
|
||||
attributes:
|
||||
label: How are you running AnythingLLM?
|
||||
description: AnythingLLM can be run in many environments, pick the one that best represents where you encounter the bug.
|
||||
options:
|
||||
- Docker (local)
|
||||
- Docker (remote machine)
|
||||
- Local development
|
||||
- AnythingLLM desktop app
|
||||
- All versions
|
||||
- Not listed
|
||||
default: 0
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: what-happened
|
||||
attributes:
|
||||
label: What happened?
|
||||
description: Also tell us, what did you expect to happen?
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: reproduction
|
||||
attributes:
|
||||
label: Are there known steps to reproduce?
|
||||
description: |
|
||||
Let us know how to reproduce the bug and we may be able to fix it more
|
||||
quickly. This is not required, but it is helpful.
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: llm-provider
|
||||
attributes:
|
||||
label: LLM Provider & Model (if applicable)
|
||||
description: What LLM provider and model are you using? (e.g., OpenAI GPT-4, Anthropic, Ollama, etc.)
|
||||
placeholder: e.g., Ollama / qwen2.5-coder-32b-instruct
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: embedder-provider
|
||||
attributes:
|
||||
label: Embedder Provider & Model (if applicable)
|
||||
description: What embedding provider and model are you using? (e.g., OpenAI text-embedding-ada-002, Lemonade, etc.)
|
||||
placeholder: e.g., Lemonade / nomic-embed-text-v1-GGUF
|
||||
validations:
|
||||
required: false
|
||||
@@ -0,0 +1,19 @@
|
||||
name: ✨ New Feature suggestion
|
||||
description: Suggest a new feature for AnythingLLM!
|
||||
title: "[FEAT]: "
|
||||
labels: [enhancement, feature request]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Share a new idea for a feature or improvement. Be sure to search existing
|
||||
issues first to avoid duplicates.
|
||||
|
||||
- type: textarea
|
||||
id: description
|
||||
attributes:
|
||||
label: What would you like to see?
|
||||
description: |
|
||||
Describe the feature and why it would be useful to your use-case as well as others.
|
||||
validations:
|
||||
required: true
|
||||
@@ -0,0 +1,13 @@
|
||||
name: 📚 Documentation improvement
|
||||
title: "[DOCS]: "
|
||||
description: Report an issue or problem with the documentation.
|
||||
labels: [documentation]
|
||||
|
||||
body:
|
||||
- type: textarea
|
||||
id: description
|
||||
attributes:
|
||||
label: Description
|
||||
description: Describe the issue with the documentation that is giving you trouble or causing confusion.
|
||||
validations:
|
||||
required: true
|
||||
@@ -0,0 +1,61 @@
|
||||
name: 🔌 New Integration Proposal
|
||||
description: Propose a new LLM, Embedding, or Vector DB provider.
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
Please review our Vetting Criteria in CONTRIBUTING.md before submitting.
|
||||
Low-effort requests or providers without an established user base or unique technical utility will be closed immediately.
|
||||
- type: input
|
||||
id: provider_name
|
||||
attributes:
|
||||
label: Provider Name
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: provider_website
|
||||
attributes:
|
||||
label: Official Website URL
|
||||
placeholder: "https://example.com"
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: provider_docs
|
||||
attributes:
|
||||
label: API Documentation URL
|
||||
placeholder: "https://docs.example.com"
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: provider_privacy
|
||||
attributes:
|
||||
label: Privacy Policy URL
|
||||
placeholder: "https://example.com/privacy"
|
||||
validations:
|
||||
required: true
|
||||
- type: input
|
||||
id: userbase_proof
|
||||
attributes:
|
||||
label: Proof of Active User Base & Demand
|
||||
description: Provide clear evidence of active community traction (e.g., Hugging Face download metrics, GitHub repo stats, or organic community discussions).
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: models_endpoint
|
||||
attributes:
|
||||
label: Native Models Discovery
|
||||
description: Does your API expose a standard GET `/v1/models` (or native equivalent) endpoint for dynamic model listing?
|
||||
options:
|
||||
- "Yes, it fully supports a model enumeration endpoint."
|
||||
- "No, users must manually type in the model strings."
|
||||
validations:
|
||||
required: true
|
||||
- type: dropdown
|
||||
id: technical_delta
|
||||
attributes:
|
||||
label: Technical Compatibility Layer
|
||||
options:
|
||||
- "This provider is NOT OpenAI-compatible and requires custom SDK/orchestration logic."
|
||||
- "This provider IS OpenAI-compatible (Stop here, use the Generic OpenAI Connector)."
|
||||
validations:
|
||||
required: true
|
||||
@@ -0,0 +1,5 @@
|
||||
blank_issues_enabled: true
|
||||
contact_links:
|
||||
- name: 🧑🤝🧑 Community Discord
|
||||
url: https://discord.gg/6UyHPeGZAC
|
||||
about: Interact with the Mintplex Labs community here by asking for help, discussing and more!
|
||||
@@ -0,0 +1,114 @@
|
||||
name: Publish AnythingLLM Docker image on Release (amd64 & arm64)
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
on:
|
||||
release:
|
||||
types: [published]
|
||||
|
||||
jobs:
|
||||
push_multi_platform_to_registries:
|
||||
name: Push Docker multi-platform image to multiple registries
|
||||
runs-on: ubuntu-22.04-arm
|
||||
permissions:
|
||||
packages: write
|
||||
contents: read
|
||||
steps:
|
||||
- name: Check out the repo
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Check if DockerHub build needed
|
||||
shell: bash
|
||||
run: |
|
||||
# Check if the secret for USERNAME is set (don't even check for the password)
|
||||
if [[ -z "${{ secrets.DOCKER_USERNAME }}" ]]; then
|
||||
echo "DockerHub build not needed"
|
||||
echo "enabled=false" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "DockerHub build needed"
|
||||
echo "enabled=true" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
id: dockerhub
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
with:
|
||||
version: v0.22.0
|
||||
|
||||
- name: Log in to Docker Hub
|
||||
uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
|
||||
# Only login to the Docker Hub if the repo is mintplex/anythingllm, to allow for forks to build on GHCR
|
||||
if: steps.dockerhub.outputs.enabled == 'true'
|
||||
with:
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
password: ${{ secrets.DOCKER_PASSWORD }}
|
||||
|
||||
- name: Log in to the Container registry
|
||||
uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Extract metadata (tags, labels) for Docker
|
||||
id: meta
|
||||
uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
|
||||
with:
|
||||
images: |
|
||||
${{ steps.dockerhub.outputs.enabled == 'true' && 'mintplexlabs/anythingllm' || '' }}
|
||||
ghcr.io/${{ github.repository }}
|
||||
tags: |
|
||||
type=semver,pattern={{version}}
|
||||
type=semver,pattern={{major}}.{{minor}}
|
||||
|
||||
- name: Build and push multi-platform Docker image
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: ./docker/Dockerfile
|
||||
push: true
|
||||
sbom: true
|
||||
provenance: mode=max
|
||||
platforms: linux/amd64,linux/arm64
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
# For Docker scout there are some intermediary reported CVEs which exists outside
|
||||
# of execution content or are unreachable by an attacker but exist in image.
|
||||
# We create VEX files for these so they don't show in scout summary.
|
||||
- name: Collect known and verified CVE exceptions
|
||||
id: cve-list
|
||||
run: |
|
||||
# Collect CVEs from filenames in vex folder
|
||||
CVE_NAMES=""
|
||||
for file in ./docker/vex/*.vex.json; do
|
||||
[ -e "$file" ] || continue
|
||||
filename=$(basename "$file")
|
||||
stripped_filename=${filename%.vex.json}
|
||||
CVE_NAMES+=" $stripped_filename"
|
||||
done
|
||||
echo "CVE_EXCEPTIONS=$CVE_NAMES" >> $GITHUB_OUTPUT
|
||||
shell: bash
|
||||
|
||||
# About VEX attestations https://docs.docker.com/scout/explore/exceptions/
|
||||
# Justifications https://github.com/openvex/spec/blob/main/OPENVEX-SPEC.md#status-justifications
|
||||
- name: Add VEX attestations
|
||||
env:
|
||||
CVE_EXCEPTIONS: ${{ steps.cve-list.outputs.CVE_EXCEPTIONS }}
|
||||
run: |
|
||||
echo $CVE_EXCEPTIONS
|
||||
curl -sSfL https://raw.githubusercontent.com/docker/scout-cli/main/install.sh | sh -s --
|
||||
for cve in $CVE_EXCEPTIONS; do
|
||||
for tag in "${{ join(fromJSON(steps.meta.outputs.json).tags, ' ') }}"; do
|
||||
echo "Attaching VEX exception $cve to $tag"
|
||||
docker scout attestation add \
|
||||
--file "./docker/vex/$cve.vex.json" \
|
||||
--predicate-type https://openvex.dev/ns/v0.2.0 \
|
||||
$tag
|
||||
done
|
||||
done
|
||||
shell: bash
|
||||
@@ -0,0 +1,136 @@
|
||||
# This GitHub action is for publishing of the primary image for AnythingLLM
|
||||
# It will publish a linux/amd64 and linux/arm64 image at the same time
|
||||
# This file should ONLY BY USED FOR `master` BRANCH.
|
||||
# TODO: GitHub now has an ubuntu-24.04-arm64 runner, but we still need
|
||||
# to use QEMU to build the arm64 image because Chromium is not available for Linux arm64
|
||||
# so builds will still fail, or fail much more often. Its inconsistent and frustrating.
|
||||
name: Publish AnythingLLM Primary Docker image (amd64/arm64)
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: ['master'] # master branch only. Do not modify.
|
||||
paths-ignore:
|
||||
- '**.md'
|
||||
- '.gitmodules'
|
||||
- 'cloud-deployments/**/*'
|
||||
- 'images/**/*'
|
||||
- '.vscode/**/*'
|
||||
- '**/.env.example'
|
||||
- '.github/ISSUE_TEMPLATE/**/*'
|
||||
- '.devcontainer/**/*'
|
||||
- 'embed/**/*' # Embed is submodule
|
||||
- 'browser-extension/**/*' # Chrome extension is submodule
|
||||
- 'server/utils/agents/aibitat/example/**/*' # Do not push new image for local dev testing of new aibitat images.
|
||||
- 'extras/**/*' # Extra is just for news and other local content.
|
||||
- 'open-computer/**/*'
|
||||
|
||||
jobs:
|
||||
push_multi_platform_to_registries:
|
||||
name: Push Docker multi-platform image to multiple registries
|
||||
runs-on: ubuntu-22.04-arm
|
||||
permissions:
|
||||
packages: write
|
||||
contents: read
|
||||
steps:
|
||||
- name: Check out the repo
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Check if DockerHub build needed
|
||||
shell: bash
|
||||
run: |
|
||||
# Check if the secret for USERNAME is set (don't even check for the password)
|
||||
if [[ -z "${{ secrets.DOCKER_USERNAME }}" ]]; then
|
||||
echo "DockerHub build not needed"
|
||||
echo "enabled=false" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "DockerHub build needed"
|
||||
echo "enabled=true" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
id: dockerhub
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
with:
|
||||
version: v0.22.0
|
||||
|
||||
- name: Log in to Docker Hub
|
||||
uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a
|
||||
# Only login to the Docker Hub if the repo is mintplex/anythingllm, to allow for forks to build on GHCR
|
||||
if: steps.dockerhub.outputs.enabled == 'true'
|
||||
with:
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
password: ${{ secrets.DOCKER_PASSWORD }}
|
||||
|
||||
- name: Log in to the Container registry
|
||||
uses: docker/login-action@65b78e6e13532edd9afa3aa52ac7964289d1a9c1
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Extract metadata (tags, labels) for Docker
|
||||
id: meta
|
||||
uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
|
||||
with:
|
||||
images: |
|
||||
${{ steps.dockerhub.outputs.enabled == 'true' && 'mintplexlabs/anythingllm' || '' }}
|
||||
ghcr.io/${{ github.repository }}
|
||||
tags: |
|
||||
type=raw,value=latest,enable={{is_default_branch}}
|
||||
type=ref,event=branch
|
||||
type=ref,event=tag
|
||||
type=ref,event=pr
|
||||
|
||||
- name: Build and push multi-platform Docker image
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: ./docker/Dockerfile
|
||||
push: true
|
||||
sbom: true
|
||||
provenance: mode=max
|
||||
platforms: linux/amd64,linux/arm64
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
|
||||
# For Docker scout there are some intermediary reported CVEs which exists outside
|
||||
# of execution content or are unreachable by an attacker but exist in image.
|
||||
# We create VEX files for these so they don't show in scout summary.
|
||||
- name: Collect known and verified CVE exceptions
|
||||
id: cve-list
|
||||
run: |
|
||||
# Collect CVEs from filenames in vex folder
|
||||
CVE_NAMES=""
|
||||
for file in ./docker/vex/*.vex.json; do
|
||||
[ -e "$file" ] || continue
|
||||
filename=$(basename "$file")
|
||||
stripped_filename=${filename%.vex.json}
|
||||
CVE_NAMES+=" $stripped_filename"
|
||||
done
|
||||
echo "CVE_EXCEPTIONS=$CVE_NAMES" >> $GITHUB_OUTPUT
|
||||
shell: bash
|
||||
|
||||
# About VEX attestations https://docs.docker.com/scout/explore/exceptions/
|
||||
# Justifications https://github.com/openvex/spec/blob/main/OPENVEX-SPEC.md#status-justifications
|
||||
- name: Add VEX attestations
|
||||
env:
|
||||
CVE_EXCEPTIONS: ${{ steps.cve-list.outputs.CVE_EXCEPTIONS }}
|
||||
run: |
|
||||
echo $CVE_EXCEPTIONS
|
||||
curl -sSfL https://raw.githubusercontent.com/docker/scout-cli/main/install.sh | sh -s --
|
||||
for cve in $CVE_EXCEPTIONS; do
|
||||
for tag in "${{ join(fromJSON(steps.meta.outputs.json).tags, ' ') }}"; do
|
||||
echo "Attaching VEX exception $cve to $tag"
|
||||
docker scout attestation add \
|
||||
--file "./docker/vex/$cve.vex.json" \
|
||||
--predicate-type https://openvex.dev/ns/v0.2.0 \
|
||||
$tag
|
||||
done
|
||||
done
|
||||
shell: bash
|
||||
@@ -0,0 +1,70 @@
|
||||
# Builds a QA GHCR image for a PR when the "PR: Ready for QA" label is present.
|
||||
# Triggers on:
|
||||
# - "PR: Ready for QA" label added to a PR
|
||||
# - New commits pushed to a PR that already has the label will trigger a new build
|
||||
name: Build QA GHCR Image
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [labeled, synchronize]
|
||||
paths-ignore:
|
||||
- "**.md"
|
||||
- ".gitmodules"
|
||||
- "cloud-deployments/**/*"
|
||||
- "images/**/*"
|
||||
- ".vscode/**/*"
|
||||
- "**/.env.example"
|
||||
- ".github/ISSUE_TEMPLATE/**/*"
|
||||
- ".devcontainer/**/*"
|
||||
- "embed/**/*"
|
||||
- "browser-extension/**/*"
|
||||
- "extras/**/*"
|
||||
- 'open-computer/**/*'
|
||||
|
||||
|
||||
concurrency:
|
||||
group: qa-build-pr-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
build:
|
||||
name: Build and push QA image for PR
|
||||
runs-on: ubuntu-22.04-arm
|
||||
# Run when labeled with "PR: Ready for QA"
|
||||
if: >-
|
||||
${{ contains(github.event.pull_request.labels.*.name, 'PR: Ready for QA') }}
|
||||
permissions:
|
||||
packages: write
|
||||
contents: read
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Check out the repo
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
with:
|
||||
version: v0.22.0
|
||||
|
||||
- name: Log in to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Set lowercase repository owner
|
||||
run: echo "REPO_OWNER_LC=${GITHUB_REPOSITORY_OWNER,,}" >> $GITHUB_ENV
|
||||
|
||||
- name: Build and push
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: ./docker/Dockerfile
|
||||
push: true
|
||||
sbom: true
|
||||
provenance: mode=max
|
||||
platforms: linux/arm64
|
||||
tags: ghcr.io/${{ env.REPO_OWNER_LC }}/${{ github.event.repository.name }}:pr-${{ github.event.pull_request.number }}
|
||||
cache-from: type=gha
|
||||
cache-to: type=gha,mode=max
|
||||
@@ -0,0 +1,37 @@
|
||||
# This GitHub action is for checking the versions of the packages in the project.
|
||||
# Any package that is present in both the `server` and `collector` package.json file
|
||||
# is checked to ensure that they are the same version.
|
||||
name: Check package versions
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened]
|
||||
paths:
|
||||
- "server/package.json"
|
||||
- "collector/package.json"
|
||||
|
||||
jobs:
|
||||
run-script:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v2
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: '18'
|
||||
|
||||
- name: Run verifyPackageVersions.mjs script
|
||||
run: |
|
||||
cd extras/scripts
|
||||
node verifyPackageVersions.mjs
|
||||
|
||||
- name: Fail job on error
|
||||
if: failure()
|
||||
run: exit 1
|
||||
@@ -0,0 +1,37 @@
|
||||
# This GitHub action is for validation of all languages which translations are offered for
|
||||
# in the locales folder in `frontend/src`. All languages are compared to the EN translation
|
||||
# schema since that is the fallback language setting. This workflow will run on all PRs that
|
||||
# modify any files in the translation directory
|
||||
name: Verify translations files
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened]
|
||||
paths:
|
||||
- "frontend/src/locales/**.js"
|
||||
|
||||
jobs:
|
||||
run-script:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v2
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: '18'
|
||||
|
||||
- name: Run verifyTranslations.mjs script
|
||||
run: |
|
||||
cd frontend/src/locales
|
||||
node verifyTranslations.mjs
|
||||
|
||||
- name: Fail job on error
|
||||
if: failure()
|
||||
run: exit 1
|
||||
@@ -0,0 +1,84 @@
|
||||
# Cleans up the GHCR image tag when the PR is closed or the "PR: Ready for QA" label is removed.
|
||||
name: Cleanup QA GHCR Image
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [closed, unlabeled]
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
pr_number:
|
||||
description: 'PR number to clean up (e.g., 123)'
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
cleanup-manual:
|
||||
name: Delete QA GHCR image tag (manual)
|
||||
runs-on: ubuntu-latest
|
||||
if: github.event_name == 'workflow_dispatch'
|
||||
permissions:
|
||||
packages: write
|
||||
steps:
|
||||
- name: Delete PR tag from GHCR
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.ALLM_RW_PACKAGES }}
|
||||
PR_NUMBER: ${{ inputs.pr_number }}
|
||||
run: |
|
||||
# Must use lowercase - packages are published with lowercase owner
|
||||
ORG_LC="${GITHUB_REPOSITORY_OWNER,,}"
|
||||
REPO_LC="${GITHUB_REPOSITORY#*/}"
|
||||
REPO_LC="${REPO_LC,,}"
|
||||
|
||||
echo "Looking for tag: pr-${PR_NUMBER}"
|
||||
echo "Package: /orgs/${ORG_LC}/packages/container/${REPO_LC}/versions"
|
||||
|
||||
VERSION_ID=$(gh api \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
--paginate \
|
||||
"/orgs/${ORG_LC}/packages/container/${REPO_LC}/versions" \
|
||||
--jq ".[] | select(.metadata.container.tags[] == \"pr-${PR_NUMBER}\") | .id")
|
||||
|
||||
if [ -n "$VERSION_ID" ]; then
|
||||
echo "Deleting package version $VERSION_ID (tag: pr-${PR_NUMBER})"
|
||||
gh api \
|
||||
--method DELETE \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
"/orgs/${ORG_LC}/packages/container/${REPO_LC}/versions/$VERSION_ID"
|
||||
else
|
||||
echo "No package found with tag pr-${PR_NUMBER}, skipping cleanup"
|
||||
fi
|
||||
|
||||
cleanup-auto:
|
||||
name: Delete QA GHCR image tag (auto)
|
||||
runs-on: ubuntu-latest
|
||||
if: >-
|
||||
(github.event.action == 'closed' && contains(github.event.pull_request.labels.*.name, 'PR: Ready for QA')) ||
|
||||
(github.event.action == 'unlabeled' && github.event.label.name == 'PR: Ready for QA')
|
||||
permissions:
|
||||
packages: write
|
||||
steps:
|
||||
- name: Delete PR tag from GHCR
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.ALLM_RW_PACKAGES }}
|
||||
PR_NUMBER: ${{ github.event.pull_request.number }}
|
||||
run: |
|
||||
# Must use lowercase - packages are published with lowercase owner
|
||||
ORG_LC="${GITHUB_REPOSITORY_OWNER,,}"
|
||||
REPO_LC="${GITHUB_REPOSITORY#*/}"
|
||||
REPO_LC="${REPO_LC,,}"
|
||||
|
||||
VERSION_ID=$(gh api \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
--paginate \
|
||||
"/orgs/${ORG_LC}/packages/container/${REPO_LC}/versions" \
|
||||
--jq ".[] | select(.metadata.container.tags[] == \"pr-${PR_NUMBER}\") | .id" \
|
||||
2>/dev/null || true)
|
||||
|
||||
if [ -n "$VERSION_ID" ]; then
|
||||
echo "Deleting package version $VERSION_ID (tag: pr-${PR_NUMBER})"
|
||||
gh api \
|
||||
--method DELETE \
|
||||
-H "Accept: application/vnd.github+json" \
|
||||
"/orgs/${ORG_LC}/packages/container/${REPO_LC}/versions/$VERSION_ID"
|
||||
else
|
||||
echo "No package found with tag pr-${PR_NUMBER}, skipping cleanup"
|
||||
fi
|
||||
@@ -0,0 +1,75 @@
|
||||
name: Lint
|
||||
|
||||
concurrency:
|
||||
group: lint-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened]
|
||||
paths:
|
||||
- "server/**/*.js"
|
||||
- "server/eslint.config.mjs"
|
||||
- "collector/**/*.js"
|
||||
- "collector/eslint.config.mjs"
|
||||
- "frontend/src/**/*.js"
|
||||
- "frontend/src/**/*.jsx"
|
||||
- "frontend/eslint.config.js"
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: "18"
|
||||
|
||||
- name: Cache server dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: server/node_modules
|
||||
key: ${{ runner.os }}-yarn-server-${{ hashFiles('server/yarn.lock') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-yarn-server-
|
||||
|
||||
- name: Cache frontend dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: frontend/node_modules
|
||||
key: ${{ runner.os }}-yarn-frontend-${{ hashFiles('frontend/yarn.lock') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-yarn-frontend-
|
||||
|
||||
- name: Cache collector dependencies
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: collector/node_modules
|
||||
key: ${{ runner.os }}-yarn-collector-${{ hashFiles('collector/yarn.lock') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-yarn-collector-
|
||||
|
||||
- name: Install server dependencies
|
||||
run: cd server && yarn install --frozen-lockfile
|
||||
|
||||
- name: Install frontend dependencies
|
||||
run: cd frontend && yarn install --frozen-lockfile
|
||||
|
||||
- name: Install collector dependencies
|
||||
run: cd collector && yarn install --frozen-lockfile
|
||||
env:
|
||||
PUPPETEER_SKIP_DOWNLOAD: "true"
|
||||
SHARP_IGNORE_GLOBAL_LIBVIPS: "true"
|
||||
|
||||
- name: Lint server
|
||||
run: cd server && yarn lint:check
|
||||
|
||||
- name: Lint frontend
|
||||
run: cd frontend && yarn lint:check
|
||||
|
||||
- name: Lint collector
|
||||
run: cd collector && yarn lint:check
|
||||
@@ -0,0 +1,80 @@
|
||||
name: Run backend tests
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, synchronize, reopened]
|
||||
paths:
|
||||
- "server/**.js"
|
||||
- "collector/**.js"
|
||||
|
||||
jobs:
|
||||
run-script:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v2
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: '18'
|
||||
|
||||
- name: Cache root dependencies
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
node_modules
|
||||
~/.cache/yarn
|
||||
key: ${{ runner.os }}-yarn-root-${{ hashFiles('**/yarn.lock') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-yarn-root-
|
||||
|
||||
- name: Cache server dependencies
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
server/node_modules
|
||||
~/.cache/yarn
|
||||
key: ${{ runner.os }}-yarn-server-${{ hashFiles('server/yarn.lock') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-yarn-server-
|
||||
|
||||
- name: Cache collector dependencies
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
collector/node_modules
|
||||
~/.cache/yarn
|
||||
key: ${{ runner.os }}-yarn-collector-${{ hashFiles('collector/yarn.lock') }}
|
||||
restore-keys: |
|
||||
${{ runner.os }}-yarn-collector-
|
||||
|
||||
- name: Install root dependencies
|
||||
if: steps.cache-root.outputs.cache-hit != 'true'
|
||||
run: yarn install --frozen-lockfile
|
||||
|
||||
- name: Install server dependencies
|
||||
if: steps.cache-server.outputs.cache-hit != 'true'
|
||||
run: cd server && yarn install --frozen-lockfile
|
||||
|
||||
- name: Install collector dependencies
|
||||
if: steps.cache-collector.outputs.cache-hit != 'true'
|
||||
run: cd collector && yarn install --frozen-lockfile
|
||||
env:
|
||||
PUPPETEER_SKIP_DOWNLOAD: "true"
|
||||
SHARP_IGNORE_GLOBAL_LIBVIPS: "true"
|
||||
|
||||
- name: Setup environment and Prisma
|
||||
run: yarn setup:envs && yarn prisma:setup
|
||||
|
||||
- name: Run test suites
|
||||
run: yarn test
|
||||
|
||||
- name: Fail job on error
|
||||
if: failure()
|
||||
run: exit 1
|
||||
@@ -0,0 +1,44 @@
|
||||
name: Generate Sponsors README
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: "0 12 * * 3" # Run every Wednesday at 12:00 PM UTC
|
||||
|
||||
permissions:
|
||||
contents: write
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout 🛎️
|
||||
uses: actions/checkout@v2
|
||||
|
||||
- name: Generate All Sponsors README
|
||||
id: generate-all-sponsors
|
||||
uses: JamesIves/github-sponsors-readme-action@v1
|
||||
with:
|
||||
token: ${{ secrets.SPONSOR_PAT }}
|
||||
file: 'README.md'
|
||||
organization: true
|
||||
active-only: false
|
||||
marker: 'all-sponsors'
|
||||
|
||||
- name: Commit and Push 🚀
|
||||
uses: stefanzweifel/git-auto-commit-action@v5
|
||||
id: auto-commit-action
|
||||
with:
|
||||
commit_message: 'Update Sponsors README'
|
||||
file_pattern: 'README.md'
|
||||
|
||||
- name: Generate PR if changes detected
|
||||
uses: peter-evans/create-pull-request@v7
|
||||
if: steps.auto-commit-action.outputs.files_changed == 'true'
|
||||
with:
|
||||
token: ${{ secrets.GITHUB_TOKEN }}
|
||||
title: 'Update Sponsors README'
|
||||
branch: 'chore/update-sponsors'
|
||||
base: 'master'
|
||||
draft: false
|
||||
reviewers: 'timothycarambat'
|
||||
assignees: 'timothycarambat'
|
||||
maintainer-can-modify: true
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
v-env
|
||||
.env
|
||||
!.env.example
|
||||
|
||||
node_modules
|
||||
__pycache__
|
||||
v-env
|
||||
.DS_Store
|
||||
aws_cf_deploy_anything_llm.json
|
||||
yarn.lock
|
||||
*.bak
|
||||
.idea
|
||||
@@ -0,0 +1,7 @@
|
||||
[submodule "browser-extension"]
|
||||
path = browser-extension
|
||||
url = https://github.com/Mintplex-Labs/anythingllm-extension.git
|
||||
[submodule "embed"]
|
||||
path = embed
|
||||
url = https://github.com/Mintplex-Labs/anythingllm-embed.git
|
||||
branch = main
|
||||
@@ -0,0 +1,8 @@
|
||||
failure-threshold: warning
|
||||
ignored:
|
||||
- DL3008
|
||||
- DL3013
|
||||
format: tty
|
||||
trustedRegistries:
|
||||
- docker.io
|
||||
- gcr.io
|
||||
@@ -0,0 +1,17 @@
|
||||
# defaults
|
||||
**/.git
|
||||
**/.svn
|
||||
**/.hg
|
||||
**/node_modules
|
||||
|
||||
#frontend
|
||||
frontend/bundleinspector.html
|
||||
**/dist
|
||||
|
||||
#server
|
||||
server/swagger/openapi.json
|
||||
server/**/*.mjs
|
||||
|
||||
#embed
|
||||
**/static/**
|
||||
embed/src/utils/chat/hljs.js
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"tabWidth": 2,
|
||||
"useTabs": false,
|
||||
"endOfLine": "lf",
|
||||
"semi": true,
|
||||
"singleQuote": false,
|
||||
"printWidth": 80,
|
||||
"trailingComma": "es5",
|
||||
"bracketSpacing": true,
|
||||
"bracketSameLine": false,
|
||||
"overrides": [
|
||||
{
|
||||
"files": ["*.js", "*.mjs", "*.jsx"],
|
||||
"options": {
|
||||
"parser": "flow",
|
||||
"arrowParens": "always"
|
||||
}
|
||||
},
|
||||
{
|
||||
"files": ["*.config.js"],
|
||||
"options": {
|
||||
"semi": false,
|
||||
"parser": "flow",
|
||||
"trailingComma": "none"
|
||||
}
|
||||
},
|
||||
{
|
||||
"files": "*.html",
|
||||
"options": {
|
||||
"bracketSameLine": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"files": ".prettierrc",
|
||||
"options": { "parser": "json" }
|
||||
}
|
||||
]
|
||||
}
|
||||
Vendored
+74
@@ -0,0 +1,74 @@
|
||||
{
|
||||
// Use IntelliSense to learn about possible attributes.
|
||||
// Hover to view descriptions of existing attributes.
|
||||
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
|
||||
"version": "0.2.0",
|
||||
"configurations": [
|
||||
{
|
||||
"name": "Collector debug",
|
||||
"request": "launch",
|
||||
"cwd": "${workspaceFolder}/collector",
|
||||
"env": {
|
||||
"NODE_ENV": "development"
|
||||
},
|
||||
"runtimeArgs": [
|
||||
"index.js"
|
||||
],
|
||||
// not using yarn/nodemon because it doesn't work with breakpoints
|
||||
// "runtimeExecutable": "yarn",
|
||||
"skipFiles": [
|
||||
"<node_internals>/**"
|
||||
],
|
||||
"type": "node"
|
||||
},
|
||||
{
|
||||
"name": "Server debug",
|
||||
"request": "launch",
|
||||
"cwd": "${workspaceFolder}/server",
|
||||
"env": {
|
||||
"NODE_ENV": "development"
|
||||
},
|
||||
"runtimeArgs": [
|
||||
"index.js"
|
||||
],
|
||||
// not using yarn/nodemon because it doesn't work with breakpoints
|
||||
// "runtimeExecutable": "yarn",
|
||||
"skipFiles": [
|
||||
"<node_internals>/**"
|
||||
],
|
||||
"type": "node"
|
||||
},
|
||||
{
|
||||
"name": "Frontend debug",
|
||||
"request": "launch",
|
||||
"cwd": "${workspaceFolder}/frontend",
|
||||
"env": {
|
||||
"NODE_ENV": "development",
|
||||
},
|
||||
"runtimeExecutable": "${workspaceFolder}/frontend/node_modules/.bin/vite",
|
||||
"runtimeArgs": [
|
||||
"--debug",
|
||||
"--host=0.0.0.0"
|
||||
],
|
||||
// "runtimeExecutable": "yarn",
|
||||
"skipFiles": [
|
||||
"<node_internals>/**"
|
||||
],
|
||||
"type": "node"
|
||||
},
|
||||
{
|
||||
"name": "Launch Edge",
|
||||
"request": "launch",
|
||||
"type": "msedge",
|
||||
"url": "http://localhost:3000",
|
||||
"webRoot": "${workspaceFolder}"
|
||||
},
|
||||
{
|
||||
"type": "chrome",
|
||||
"request": "launch",
|
||||
"name": "Launch Chrome against localhost",
|
||||
"url": "http://localhost:3000",
|
||||
"webRoot": "${workspaceFolder}"
|
||||
}
|
||||
]
|
||||
}
|
||||
Vendored
+64
@@ -0,0 +1,64 @@
|
||||
{
|
||||
"cSpell.words": [
|
||||
"adoc",
|
||||
"aibitat",
|
||||
"AIbitat",
|
||||
"allm",
|
||||
"anythingllm",
|
||||
"Apipie",
|
||||
"Astra",
|
||||
"Chartable",
|
||||
"cleancss",
|
||||
"comkey",
|
||||
"cooldown",
|
||||
"cooldowns",
|
||||
"datafile",
|
||||
"Deduplicator",
|
||||
"Dockerized",
|
||||
"docpath",
|
||||
"elevenlabs",
|
||||
"Embeddable",
|
||||
"epub",
|
||||
"fireworksai",
|
||||
"GROQ",
|
||||
"hljs",
|
||||
"huggingface",
|
||||
"inferencing",
|
||||
"koboldcpp",
|
||||
"Langchain",
|
||||
"lmstudio",
|
||||
"localai",
|
||||
"mbox",
|
||||
"Milvus",
|
||||
"Mintplex",
|
||||
"mixtral",
|
||||
"moderations",
|
||||
"novita",
|
||||
"numpages",
|
||||
"Ollama",
|
||||
"Oobabooga",
|
||||
"openai",
|
||||
"opendocument",
|
||||
"openrouter",
|
||||
"pagerender",
|
||||
"ppio",
|
||||
"Qdrant",
|
||||
"royalblue",
|
||||
"SearchApi",
|
||||
"searxng",
|
||||
"SerpApi",
|
||||
"Serper",
|
||||
"Serply",
|
||||
"streamable",
|
||||
"textgenwebui",
|
||||
"togetherai",
|
||||
"Unembed",
|
||||
"uuidv",
|
||||
"vectordbs",
|
||||
"Weaviate",
|
||||
"XAILLM",
|
||||
"Zilliz"
|
||||
],
|
||||
"eslint.experimental.useFlatConfig": true,
|
||||
"docker.languageserver.formatter.ignoreMultilineInstructions": true
|
||||
}
|
||||
Vendored
+94
@@ -0,0 +1,94 @@
|
||||
{
|
||||
// See https://go.microsoft.com/fwlink/?LinkId=733558
|
||||
// for the documentation about the tasks.json format
|
||||
"version": "2.0.0",
|
||||
"tasks": [
|
||||
{
|
||||
"type": "shell",
|
||||
"options": {
|
||||
"cwd": "${workspaceFolder}/collector",
|
||||
"statusbar": {
|
||||
"color": "#ffea00",
|
||||
"detail": "Runs the collector",
|
||||
"label": "Collector: $(play) run",
|
||||
"running": {
|
||||
"color": "#ffea00",
|
||||
"label": "Collector: $(gear~spin) running"
|
||||
}
|
||||
}
|
||||
},
|
||||
"command": "cd ${workspaceFolder}/collector/ && yarn dev",
|
||||
"runOptions": {
|
||||
"instanceLimit": 1,
|
||||
"reevaluateOnRerun": true
|
||||
},
|
||||
"presentation": {
|
||||
"echo": true,
|
||||
"reveal": "always",
|
||||
"focus": false,
|
||||
"panel": "shared",
|
||||
"showReuseMessage": true,
|
||||
"clear": false
|
||||
},
|
||||
"label": "Collector: run"
|
||||
},
|
||||
{
|
||||
"type": "shell",
|
||||
"options": {
|
||||
"cwd": "${workspaceFolder}/server",
|
||||
"statusbar": {
|
||||
"color": "#ffea00",
|
||||
"detail": "Runs the server",
|
||||
"label": "Server: $(play) run",
|
||||
"running": {
|
||||
"color": "#ffea00",
|
||||
"label": "Server: $(gear~spin) running"
|
||||
}
|
||||
}
|
||||
},
|
||||
"command": "if [ \"${CODESPACES}\" = \"true\" ]; then while ! gh codespace ports -c $CODESPACE_NAME | grep 3001; do sleep 1; done; gh codespace ports visibility 3001:public -c $CODESPACE_NAME; fi & cd ${workspaceFolder}/server/ && yarn dev",
|
||||
"runOptions": {
|
||||
"instanceLimit": 1,
|
||||
"reevaluateOnRerun": true
|
||||
},
|
||||
"presentation": {
|
||||
"echo": true,
|
||||
"reveal": "always",
|
||||
"focus": false,
|
||||
"panel": "shared",
|
||||
"showReuseMessage": true,
|
||||
"clear": false
|
||||
},
|
||||
"label": "Server: run"
|
||||
},
|
||||
{
|
||||
"type": "shell",
|
||||
"options": {
|
||||
"cwd": "${workspaceFolder}/frontend",
|
||||
"statusbar": {
|
||||
"color": "#ffea00",
|
||||
"detail": "Runs the frontend",
|
||||
"label": "Frontend: $(play) run",
|
||||
"running": {
|
||||
"color": "#ffea00",
|
||||
"label": "Frontend: $(gear~spin) running"
|
||||
}
|
||||
}
|
||||
},
|
||||
"command": "cd ${workspaceFolder}/frontend/ && yarn dev",
|
||||
"runOptions": {
|
||||
"instanceLimit": 1,
|
||||
"reevaluateOnRerun": true
|
||||
},
|
||||
"presentation": {
|
||||
"echo": true,
|
||||
"reveal": "always",
|
||||
"focus": false,
|
||||
"panel": "shared",
|
||||
"showReuseMessage": true,
|
||||
"clear": false
|
||||
},
|
||||
"label": "Frontend: run"
|
||||
}
|
||||
]
|
||||
}
|
||||
+147
@@ -0,0 +1,147 @@
|
||||
# Run AnythingLLM in production without Docker
|
||||
|
||||
> [!WARNING]
|
||||
> This method of deployment is **not supported** by the core-team and is to be used as a reference for your deployment.
|
||||
> You are fully responsible for securing your deployment and data in this mode.
|
||||
> **Any issues** experienced from bare-metal or non-containerized deployments will be **not** answered or supported.
|
||||
|
||||
Here you can find the scripts and known working process to run AnythingLLM outside of a Docker container.
|
||||
|
||||
### Minimum Requirements
|
||||
> [!TIP]
|
||||
> You should aim for at least 2GB of RAM. Disk storage is proportional to however much data
|
||||
> you will be storing (documents, vectors, models, etc). Minimum 10GB recommended.
|
||||
|
||||
- NodeJS v18
|
||||
- Yarn
|
||||
|
||||
|
||||
## Getting started
|
||||
|
||||
1. Clone the repo into your server as the user who the application will run as.
|
||||
`git clone git@github.com:Mintplex-Labs/anything-llm.git`
|
||||
|
||||
2. `cd anything-llm` and run `yarn setup`. This will install all dependencies to run in production as well as debug the application.
|
||||
|
||||
3. `cp server/.env.example server/.env` to create the basic ENV file for where instance settings will be read from on service start.
|
||||
|
||||
4. Ensure that the `server/.env` file has _at least_ these keys to start. These values will persist and this file will be automatically written and managed after your first successful boot.
|
||||
```
|
||||
STORAGE_DIR="/your/absolute/path/to/server/storage"
|
||||
```
|
||||
|
||||
5. Edit the `frontend/.env` file for the `VITE_API_BASE` to now be set to `/api`. This is documented in the .env for which one you should use.
|
||||
```
|
||||
# VITE_API_BASE='http://localhost:3001/api' # Use this URL when developing locally
|
||||
# VITE_API_BASE="https://$CODESPACE_NAME-3001.$GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN/api" # for GitHub Codespaces
|
||||
VITE_API_BASE='/api' # Use this URL deploying on non-localhost address OR in docker.
|
||||
```
|
||||
|
||||
## To start the application
|
||||
|
||||
AnythingLLM is comprised of three main sections. The `frontend`, `server`, and `collector`. When running in production you will be running `server` and `collector` on two different processes, with a build step for compilation of the frontend.
|
||||
|
||||
1. Build the frontend application.
|
||||
`cd frontend && yarn build` - this will produce a `frontend/dist` folder that will be used later.
|
||||
|
||||
2. Copy `frontend/dist` to `server/public` - `cp -R frontend/dist server/public`.
|
||||
This should create a folder in `server` named `public` which contains a top level `index.html` file and various other files/folders.
|
||||
|
||||
3. Migrate and prepare your database file.
|
||||
```
|
||||
cd server && npx prisma generate --schema=./prisma/schema.prisma
|
||||
cd server && npx prisma migrate deploy --schema=./prisma/schema.prisma
|
||||
```
|
||||
|
||||
4. Boot the server in production
|
||||
`cd server && NODE_ENV=production node index.js &`
|
||||
|
||||
5. Boot the collection in another process
|
||||
`cd collector && NODE_ENV=production node index.js &`
|
||||
|
||||
AnythingLLM should now be running on `http://localhost:3001`!
|
||||
|
||||
## Updating AnythingLLM
|
||||
|
||||
To update AnythingLLM with future updates you can `git pull origin master` to pull in the latest code and then repeat steps 2 - 5 to deploy with all changes fully.
|
||||
|
||||
_note_ You should ensure that each folder runs `yarn` again to ensure packages are up to date in case any dependencies were added, changed, or removed.
|
||||
|
||||
_note_ You should `pkill node` before running an update so that you are not running multiple AnythingLLM processes on the same instance as this can cause conflicts.
|
||||
|
||||
|
||||
### Example update script
|
||||
|
||||
```shell
|
||||
#!/bin/bash
|
||||
|
||||
cd $HOME/anything-llm &&\
|
||||
git checkout . &&\
|
||||
git pull origin master &&\
|
||||
echo "HEAD pulled to commit $(git log -1 --pretty=format:"%h" | tail -n 1)"
|
||||
|
||||
echo "Freezing current ENVs"
|
||||
curl -I "http://localhost:3001/api/env-dump" | head -n 1|cut -d$' ' -f2
|
||||
|
||||
echo "Rebuilding Frontend"
|
||||
cd $HOME/anything-llm/frontend && yarn && yarn build && cd $HOME/anything-llm
|
||||
|
||||
echo "Copying to Server Public"
|
||||
rm -rf server/public
|
||||
cp -r frontend/dist server/public
|
||||
|
||||
echo "Killing node processes"
|
||||
pkill node
|
||||
|
||||
echo "Installing collector dependencies"
|
||||
cd $HOME/anything-llm/collector && yarn
|
||||
|
||||
echo "Installing server dependencies & running migrations"
|
||||
cd $HOME/anything-llm/server && yarn
|
||||
cd $HOME/anything-llm/server && npx prisma migrate deploy --schema=./prisma/schema.prisma
|
||||
cd $HOME/anything-llm/server && npx prisma generate
|
||||
|
||||
echo "Booting up services."
|
||||
truncate -s 0 /logs/server.log # Or any other log file location.
|
||||
truncate -s 0 /logs/collector.log
|
||||
|
||||
cd $HOME/anything-llm/server
|
||||
(NODE_ENV=production node index.js) &> /logs/server.log &
|
||||
|
||||
cd $HOME/anything-llm/collector
|
||||
(NODE_ENV=production node index.js) &> /logs/collector.log &
|
||||
```
|
||||
|
||||
## Using Nginx?
|
||||
|
||||
If you are using Nginx, you can use the following example configuration to proxy the requests to the server. Chats for streaming require **websocket** connections, so you need to ensure that the Nginx configuration is set up to support websockets. You can do this with a simple reverse proxy configuration.
|
||||
|
||||
```nginx
|
||||
server {
|
||||
# Enable websocket connections for agent protocol.
|
||||
location ~* ^/api/agent-invocation/(.*) {
|
||||
proxy_pass http://0.0.0.0:3001;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "Upgrade";
|
||||
}
|
||||
|
||||
listen 80;
|
||||
server_name [insert FQDN here];
|
||||
location / {
|
||||
# Prevent timeouts on long-running requests.
|
||||
proxy_connect_timeout 605;
|
||||
proxy_send_timeout 605;
|
||||
proxy_read_timeout 605;
|
||||
send_timeout 605;
|
||||
keepalive_timeout 605;
|
||||
|
||||
# Enable readable HTTP Streaming for LLM streamed responses
|
||||
proxy_buffering off;
|
||||
proxy_cache off;
|
||||
|
||||
# Proxy your locally running service
|
||||
proxy_pass http://0.0.0.0:3001;
|
||||
}
|
||||
}
|
||||
```
|
||||
+146
@@ -0,0 +1,146 @@
|
||||
# Contributing to AnythingLLM
|
||||
|
||||
AnythingLLM is an open-source project and we welcome contributions from the community.
|
||||
|
||||
## Reporting Issues
|
||||
|
||||
If you encounter a bug or have a feature request, please open an issue on the
|
||||
[GitHub issue tracker](https://github.com/mintplex-labs/anything-llm).
|
||||
|
||||
## Picking an issue
|
||||
|
||||
We track issues on the GitHub issue tracker. If you are looking for something to
|
||||
work on, check the [good first issue](https://github.com/mintplex-labs/anything-llm/contribute) label. These issues are typically the best described and have the smallest scope. There may be issues that are not labeled as good first issue, but are still a good starting point.
|
||||
|
||||
If there's an issue you are interested in working on, please leave a comment on the issue. This will help us avoid duplicate work. Additionally, if you have questions about the issue, please ask them in the issue comments. We are happy to provide guidance on how to approach the issue.
|
||||
|
||||
## Before you start
|
||||
|
||||
Keep in mind that we are a small team and have limited resources. We will do our best to review and merge your PRs, but please be patient. Ultimately, **we become the maintainer** of your changes. It is our responsibility to make sure that the changes are working as expected and are of high quality as well as being compatible with the rest of the project both for existing users and for future users & features.
|
||||
|
||||
Before you start working on an issue, please read the following so that you don't waste time on something that is not a good fit for the project or is more suitable for a personal fork. We would rather answer a comment on an issue than close a PR after you've spent time on it. Your time is valuable and we appreciate your time and effort to make AnythingLLM better.
|
||||
|
||||
0. (most important) If you are making a PR that does not have a corresponding issue, **it will not be merged.** _The only exception to this is language translations._
|
||||
|
||||
1. If you are modifying the permission system for a new role or something custom, you are likely better off forking the project and building your own version since this is a core part of the project and is only to be maintained by the AnythingLLM team.
|
||||
|
||||
2. Integrations (LLM, Vector DB, etc.) are reviewed at our discretion. We will eventually get to them. Do not expect us to merge your integration PR instantly since there are often many moving parts and we want to make sure we get it right. We will get to it!
|
||||
|
||||
3. It is our discretion to merge or not merge a PR. We value every contribution, but we also value the quality of the code and the user experience we envision for the project. It is a fine line to walk when running a project like this and please understand that merging or not merging a PR is not a reflection of the quality of the contribution and is not personal. We will do our best to provide feedback on the PR and help you make the changes necessary to get it merged.
|
||||
|
||||
4. **Security** is always important. If you have a security concern, please do not open an issue. Instead, please open a CVE on our designated reporting platform [Huntr](https://huntr.com) or contact us at [team@mintplexlabs.com](mailto:team@mintplexlabs.com).
|
||||
|
||||
## Configuring Git
|
||||
|
||||
First, fork the repository on GitHub, then clone your fork:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/<username>/anything-llm.git
|
||||
cd anything-llm
|
||||
```
|
||||
|
||||
Then add the main repository as a remote:
|
||||
|
||||
```bash
|
||||
git remote add upstream https://github.com/mintplex-labs/anything-llm.git
|
||||
git fetch upstream
|
||||
```
|
||||
|
||||
## Setting up your development environment
|
||||
|
||||
In the root of the repository, run:
|
||||
|
||||
```bash
|
||||
yarn setup
|
||||
```
|
||||
|
||||
This will install the dependencies, set up the proper and expected ENV files for the project, and run the prisma setup script.
|
||||
Next, run:
|
||||
|
||||
```bash
|
||||
yarn dev
|
||||
```
|
||||
This will start the server, frontend, and collector in development mode. Changes to the code will be hot reloaded.
|
||||
|
||||
## Best practices for pull requests
|
||||
|
||||
For the best chance of having your pull request accepted, please follow these guidelines:
|
||||
|
||||
1. Unit test all bug fixes and new features. Your code will not be merged if it
|
||||
doesn't have tests.
|
||||
1. If you change the public API, update the documentation in the `anythingllm-docs` repository.
|
||||
1. Aim to minimize the number of changes in each pull request. Keep to solving
|
||||
one problem at a time, when possible.
|
||||
1. Before marking a pull request ready-for-review, do a self review of your code.
|
||||
Is it clear why you are making the changes? Are the changes easy to understand?
|
||||
1. Use [conventional commit messages](https://www.conventionalcommits.org/en/) as pull request titles. Examples:
|
||||
* New feature: `feat: adding foo API`
|
||||
* Bug fix: `fix: issue with foo API`
|
||||
* Documentation change: `docs: adding foo API documentation`
|
||||
1. If your pull request is a work in progress, leave the pull request as a draft.
|
||||
We will assume the pull request is ready for review when it is opened.
|
||||
1. When writing tests, test the error cases. Make sure they have understandable
|
||||
error messages.
|
||||
|
||||
## Project structure
|
||||
|
||||
The core library is written in Node.js. There are additional sub-repositories for the embed widget and browser extension. These are not part of the core AnythingLLM project, but are maintained by the AnythingLLM team.
|
||||
|
||||
* `server`: Node.js server source code
|
||||
* `frontend`: React frontend source code
|
||||
* `collector`: Node.js collector source code
|
||||
|
||||
## Release process
|
||||
|
||||
Changes to the core AnythingLLM project are released through the `master` branch. When a PR is merged into `master`, a new version of the package is published to Docker and GitHub Container Registry under the `latest` tag.
|
||||
|
||||
When a new version is released, the following steps are taken a new image is built and pushed to Docker Hub and GitHub Container Registry under the associated version tag. Version tags are of the format `v<major>.<minor>.<patch>` and are pinned code, while `latest` is the latest version of the code at any point in time.
|
||||
|
||||
### Desktop propagation
|
||||
|
||||
Changes to the desktop app are downstream of the core AnythingLLM project. Releases of the desktop app are published at the same time as the core AnythingLLM project. Code from the core AnythingLLM project is copied into the desktop app into an Electron wrapper. The Electron wrapper that wraps around the core AnythingLLM project is **not** part of the core AnythingLLM project, but is maintained by the AnythingLLM team.
|
||||
|
||||
## 🔌 Criteria for New LLM Providers
|
||||
|
||||
To ensure the long-term maintainability of AnythingLLM and prevent repository bloat, we enforce a vetting process for adding new third-party LLM provider integrations.
|
||||
|
||||
With thousands of new wrapper API services launching daily, we do not accept dedicated integrations for services that lack an established user base or offer no unique technical utility over our existing generic connectors which should be sufficient for most use cases.
|
||||
|
||||
While we understand everyone has to start somewhere, we cannot maintain a repository with thousands of **bespoke** LLM integrations that functionally are no different from one another. We want to keep the repository as clean and maintainable as possible since 99% of contributors for this specific integration do their integration PR and never contribute again.
|
||||
|
||||
> 🤝 **Strategic Partnership Exception:**
|
||||
> These guidelines apply strictly to unsolicited community or third-party startup contributions. If you are an ecosystem, silicon, or cloud hardware partner engaging directly with the Mintplex Labs core team on a co-developed integration, proof-of-concept, or native optimization project, this vetting process is not applicable.
|
||||
|
||||
Before opening an unsolicited issue or submitting a Pull Request for a new provider, it **must** meet both the Technical and Market Viability thresholds below.
|
||||
|
||||
### 1. Technical Threshold
|
||||
We do not accept dedicated integration code for providers whose API architecture mimics existing standards.
|
||||
|
||||
* **The OpenAI-Compatibility Rule:** If your service utilizes the OpenAI SDK, standard OpenAI API schema (eg: `/v1/chat/completions`, `/models`) without requiring unique orchestration logic, **it will be rejected.** Users must connect to your service using our generic **OpenAI Compatible** or **Generic API** connectors.
|
||||
* **To qualify for a dedicated integration, the PR must prove:**
|
||||
* **Custom Authentication:** Requires a complex, multi-step auth flow or custom request signing (e.g., AWS SigV4) that standard bearer tokens/headers cannot support.
|
||||
* **Proprietary SDK/Payloads:** Relies on a distinct, widely adopted native SDK with a JSON schema that cannot be cleanly mapped to our generic layers.
|
||||
* **Unique Architectural Features:** Exposes critical, native platform capabilities (e.g., custom server-side routing nodes or proprietary hyper-parameters) that are completely lost when forced through a generic wrapper.
|
||||
|
||||
### 2. Market Viability Threshold
|
||||
We cannot act as a discovery or marketing engine for early-stage startups. To qualify for codebase inclusion, a provider must demonstrate an active, existing user base who would benefit from AnythingLLM's functionality. Any of the following criteria are acceptable:
|
||||
|
||||
* **Community Demand:** An integration issue request must accumulate a minimum of **20 organic upvotes (`+1` reactions)** from unique GitHub users before a PR will be reviewed.
|
||||
* **Footprint Metrics:** The provider or core underlying model organization must possess a verifiable footprint (e.g., `50,000` aggregate downloads on Hugging Face, or `1,000` stars on its core open-source repository).
|
||||
* **Operational Longevity:** The provider's production API must be publicly accessible and stable for a minimum of **90 days**. We do not accept integrations for services that launched less than 90 days ago.
|
||||
|
||||
## 🤖 AI Use in Contributions
|
||||
|
||||
We are an AI company — we obviously use AI tools and expect contributors do too. However, we believe in **AI-augmented engineers**, not AI-replaced engineers. There is a difference between using an LLM to help you write code and having an LLM write code for you.
|
||||
|
||||
### What will get your PR closed immediately
|
||||
|
||||
1. **LLM-generated tests that don't test your code.** If your tests are clearly auto-generated boilerplate that blindly asserts unrelated functionality (e.g., testing the Node.js `fs` module instead of the feature you changed), your PR will be closed. Tests should demonstrate that you understand the code you wrote and that it works. It is not our job to understand the code your LLM generated, but it becomes our responsibility to maintain it forever.
|
||||
|
||||
2. **`Claude Code` or similar agent signatures in your commit history.** In our view, a commit history full of autonomous agent commits signals low-effort work and a lack of craft or care in the functionality being contributed. We take pride in what we ship and expect the same from contributors. Use an LLM to help you think, draft, and iterate — but the work should be yours, reviewed by you, and committed by you.
|
||||
|
||||
We are not anti-AI. We are anti-low-effort. If it looks like you prompted an agent, accepted the output without scrutiny, and opened a PR — we will close it to preserve our time and resources.
|
||||
|
||||
## License
|
||||
|
||||
By contributing to AnythingLLM (this repository), you agree to license your contributions under the MIT license.
|
||||
@@ -0,0 +1,21 @@
|
||||
The MIT License
|
||||
|
||||
Copyright (c) Mintplex Labs Inc.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in
|
||||
all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||
THE SOFTWARE.
|
||||
@@ -0,0 +1,7 @@
|
||||
# WeHub 来源说明
|
||||
|
||||
- 原始项目：`Mintplex-Labs/anything-llm`
|
||||
- 原始仓库：https://github.com/Mintplex-Labs/anything-llm
|
||||
- 导入方式：上游默认分支的最新快照
|
||||
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
|
||||
- 本文件仅用于记录来源，不代表 WeHub 是原项目作者
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
# Security Policy
|
||||
|
||||
## Supported Versions
|
||||
|
||||
Use this section to tell people about which versions of your project are
|
||||
currently being supported with security updates.
|
||||
|
||||
| Version | Supported |
|
||||
| ------- | ------------------ |
|
||||
| 0.1.x | :white_check_mark: |
|
||||
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
If a security concern is found that you would like to disclose you can create a PR for it or if you would like to clear this issue before posting you can email [Core Mintplex Labs Team](mailto:team@mintplexlabs.com).
|
||||
|
||||
## Invalid Report Types
|
||||
|
||||
Below are some common types of invalid reports that we will not accept and should not be submitted as they will be closed immediately without action.
|
||||
|
||||
### SSRF Reports
|
||||
|
||||
If you are about to report a SSRF about being able to call web-scraping or document collector against an internal host, [this is not a valid report](https://github.com/Mintplex-Labs/anything-llm/blob/master/collector/utils/url/index.js#L2-L13). This is a feature of the system and is intended to be used in this way given that AnythingLLM is designed to be used in this way so that it can leverage internal services for scraping and collecting content when deployed inside a VPC.
|
||||
|
||||
### XXS Reports where user must right-click and paste in the URL
|
||||
|
||||
If you are about to report a XXS about being able to right-click on an image (like user profile picture) and paste in the URL, this is not a valid report. User profile pictures must be uploaded by the user and cannot be set by the administrator. In fact, nobody can even see them aside from the user themselves. The same goes for any other image that is uploaded by the user or even produced by the system. If the user must right-click and paste in the URL to their browser, this is not a valid report.
|
||||
|
||||
Valid XXS Reports must be zero-action - like on loading a page or a image instantly.
|
||||
|
||||
### Any "Unauthenticated" actions
|
||||
|
||||
If the basis of your report relies on the system not setting up a password or multi-user mode, this is not a valid report. AnythingLLM is designed to be used in this way so that it can be used in trusted and fully isolated environments for single user or internal user. There are three options for authentication:
|
||||
|
||||
1. No authentication - this would have no endpoint authentication and would be accessible to anyone who knows the URL.
|
||||
2. Password - this would require a password to access the system.
|
||||
3. Multi-user mode - this would require a user to be logged in to the system with username and password and be given explicit access to the system by administrator.
|
||||
|
||||
During onboarding, the system will prompt the user to set up a password or multi-user mode. If the user does not opt to set up a password or multi-user mode, the system will be accessible to anyone who knows the URL. This is an intentional design choice and is not a vulnerability.
|
||||
|
||||
If your report is about being able to access the system via _bypassing the authentication_ or lackthereof, that **is a valid report** and will be investigated and fixed.
|
||||
@@ -0,0 +1,37 @@
|
||||
# AnythingLLM Self-Hosted: Data Privacy & Terms of Service
|
||||
|
||||
This document outlines the privacy standards, data handling procedures, and licensing terms for the self-hosted version of AnythingLLM, developed by Mintplex Labs Inc.
|
||||
|
||||
## 1. Data Sovereignty & Local-First Architecture
|
||||
AnythingLLM is designed as a **local-first** application. When utilizing the self-hosted version (Docker, Desktop, or Source):
|
||||
* **No External Access:** Mintplex Labs Inc. does not host, store, or have access to any documents, chat histories, workspace settings, or embeddings created within your instance.
|
||||
* **On-Premise Storage:** All data resides strictly on the infrastructure provisioned and managed by the user or their organization.
|
||||
* **Air-Gap Capability:** AnythingLLM can be operated in a strictly air-gapped environment with no internet connectivity, provided local LLM and Vector database providers (e.g., Ollama, LocalAI, LanceDB) are utilized.
|
||||
|
||||
## 2. Telemetry and Analytics
|
||||
To improve software performance and stability, AnythingLLM includes an optional telemetry feature.
|
||||
* **Anonymity:** Collected data is strictly anonymous and contains no Personally Identifiable Information (PII), document content, chat logs, fingerprinting data, or any other sensitive information. Purely usage based data is collected.
|
||||
* **Opt-Out:** Users may disable telemetry at any time via the **Settings** menu within the application. Once disabled, no usage data is transmitted to Mintplex Labs.
|
||||
|
||||
## 3. Third-Party Integrations
|
||||
AnythingLLM allows users to connect to external services (e.g., OpenAI, Anthropic, Pinecone).
|
||||
* **Data Transmission:** When these services are enabled, data is transmitted directly from your instance to the third-party provider.
|
||||
* **Governing Terms:** Data handled by third-party providers is subject to their respective Terms of Service and Privacy Policies. Mintplex Labs is not responsible for the data practices of these external entities.
|
||||
|
||||
_by default, AnythingLLM does **everything on-device first** - so you would have to manually configure and enable these integrations to be subject to third party terms._
|
||||
|
||||
## 4. Security & Network
|
||||
* **No "Phone Home":** Aside from [optional telemetry](https://github.com/Mintplex-Labs/anything-llm?tab=readme-ov-file#telemetry--privacy), the software does not require an external connection to Mintplex Labs servers to function.
|
||||
* **Environment Security:** The user is responsible for securing the host environment, including network firewalls, SSL/TLS encryption, and access control for the AnythingLLM instance.
|
||||
* **CDN Assets:** As a convenience to international users, we use a hosted CDN to mirror some critical path models (eg: the default embedder and reranking ONNX models) which are not available in all regions. These models are downloaded from our CDN as a fallback, and for any air-gapped installations you can either download these models manually or use another provider. Assets of this nature are downloaded once and cached in your associated local storage.
|
||||
|
||||
## 5. Licensing and Liability
|
||||
* **License:** The AnythingLLM core is provided under the **MIT License**.
|
||||
* **No Warranty:** As per the license agreement, the software is provided "as is," without warranty of any kind, express or implied, including but not limited to the warranties of merchantability or fitness for a particular purpose.
|
||||
* **Liability:** In no event shall the authors or copyright holders be liable for any claim, damages, or other liability arising from the use of the software.
|
||||
|
||||
## 6. Support and Compatibility
|
||||
While Mintplex Labs prioritizes stability and backward compatibility, the self-hosted version is used at the user's discretion. Formal Service Level Agreements (SLAs) are not provided for the standard self-hosted version unless otherwise negotiated via a separate enterprise agreement.
|
||||
|
||||
---
|
||||
*Last Updated: March 2026*
|
||||
@@ -0,0 +1,49 @@
|
||||
# How to deploy a private AnythingLLM instance on AWS
|
||||
|
||||
With an AWS account you can easily deploy a private AnythingLLM instance on AWS. This will create a url that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys and they will not be exposed - however if you want your instance to be protected it is highly recommend that you set a password once setup is complete.
|
||||
|
||||
**Quick Launch (EASY)**
|
||||
1. Log in to your AWS account
|
||||
2. Open [CloudFormation](https://us-west-1.console.aws.amazon.com/cloudformation/home)
|
||||
3. Ensure you are deploying in a geographic zone that is nearest to your physical location to reduce latency.
|
||||
4. Click `Create Stack`
|
||||
|
||||
|
||||
|
||||
5. Use the file `cloudformation_create_anythingllm.json` as your JSON template.
|
||||
|
||||
|
||||
|
||||
6. Click Deploy.
|
||||
7. Wait for stack events to finish and be marked as `Completed`
|
||||
8. View `Outputs` tab.
|
||||
|
||||
|
||||
|
||||
9. Wait for all resources to be built. Now wait until instance is available on `[InstanceIP]:3001`.
|
||||
This process may take up to 10 minutes. See **Note** below on how to visualize this process.
|
||||
|
||||
The output of this cloudformation stack will be:
|
||||
- 1 EC2 Instance
|
||||
- 1 Security Group with 0.0.0.0/0 access on port 3001
|
||||
- 1 EC2 Instance Volume `gb2` of 10Gib minimum - customizable pre-deploy.
|
||||
|
||||
**Requirements**
|
||||
- An AWS account with billing information.
|
||||
|
||||
## Please read this notice before submitting issues about your deployment
|
||||
|
||||
**Note:**
|
||||
Your instance will not be available instantly. Depending on the instance size you launched with it can take 5-10 minutes to fully boot up.
|
||||
|
||||
If you want to check the instance's progress, navigate to [your deployed EC2 instances](https://us-west-1.console.aws.amazon.com/ec2/home) and connect to your instance via SSH in browser.
|
||||
|
||||
Once connected run `sudo tail -f /var/log/cloud-init-output.log` and wait for the file to conclude deployment of the docker image.
|
||||
You should see an output like this
|
||||
```
|
||||
[+] Running 2/2
|
||||
⠿ Network docker_anything-llm Created
|
||||
⠿ Container anything-llm Started
|
||||
```
|
||||
|
||||
Additionally, your use of this deployment process means you are responsible for any costs of these AWS resources fully.
|
||||
@@ -0,0 +1,118 @@
|
||||
# How to Configure HTTPS for Anything LLM AWS private deployment
|
||||
Instructions for manual https configuration after generating and running the aws cloudformation template (aws_build_from_source_no_credentials.json). Tested on following browsers: Firefox version 119, Chrome version 118, Edge 118.
|
||||
|
||||
**Requirements**
|
||||
- Successful deployment of Amazon Linux 2023 EC2 instance with Docker container running Anything LLM
|
||||
- Admin priv to configure Elastic IP for EC2 instance via AWS Management Console UI
|
||||
- Admin priv to configure DNS services (i.e. AWS Route 53) via AWS Management Console UI
|
||||
- Admin priv to configure EC2 Security Group rules via AWS Management Console UI
|
||||
|
||||
## Step 1: Allocate and assign Elastic IP Address to your deployed EC2 instance
|
||||
1. Follow AWS instructions on allocating EIP here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html#using-instance-addressing-eips-allocating
|
||||
2. Follow AWS instructions on assigning EIP to EC2 instance here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html#using-instance-addressing-eips-associating
|
||||
|
||||
## Step 2: Configure DNS A record to resolve to the previously assigned EC2 instance via EIP
|
||||
These instructions assume that you already have a top-level domain configured and are using a subdomain
|
||||
to access AnythingLLM.
|
||||
1. Follow AWS instructions on routing traffic to EC2 instance here: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-ec2-instance.html
|
||||
|
||||
## Step 3: Install and enable nginx
|
||||
These instructions are for CLI configuration and assume you are logged in to EC2 instance as the ec2-user.
|
||||
1. $sudo yum install nginx -y
|
||||
2. $sudo systemctl enable nginx && sudo systemctl start nginx
|
||||
|
||||
## Step 4: Install certbot
|
||||
These instructions are for CLI configuration and assume you are logged in to EC2 instance as the ec2-user.
|
||||
1. $sudo yum install -y augeas-libs
|
||||
2. $sudo python3 -m venv /opt/certbot/
|
||||
3. $sudo /opt/certbot/bin/pip install --upgrade pip
|
||||
4. $sudo /opt/certbot/bin/pip install certbot certbot-nginx
|
||||
5. $sudo ln -s /opt/certbot/bin/certbot /usr/bin/certbot
|
||||
|
||||
## Step 5: Configure temporary Inbound Traffic Rule for Security Group to certbot DNS verification
|
||||
1. Follow AWS instructions on creating inbound rule (http port 80 0.0.0.0/0) for EC2 security group here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-security-groups.html#adding-security-group-rule
|
||||
|
||||
## Step 6: Comment out default http NGINX proxy configuration
|
||||
These instructions are for CLI configuration and assume you are logged in to EC2 instance as the ec2-user.
|
||||
1. $sudo vi /etc/nginx/nginx.conf
|
||||
2. In the nginx.conf file, comment out the default server block configuration for http/port 80. It should look something like the following:
|
||||
```
|
||||
# server {
|
||||
# listen 80;
|
||||
# listen [::]:80;
|
||||
# server_name _;
|
||||
# root /usr/share/nginx/html;
|
||||
#
|
||||
# # Load configuration files for the default server block.
|
||||
# include /etc/nginx/default.d/*.conf;
|
||||
#
|
||||
# error_page 404 /404.html;
|
||||
# location = /404.html {
|
||||
# }
|
||||
#
|
||||
# error_page 500 502 503 504 /50x.html;
|
||||
# location = /50x.html {
|
||||
# }
|
||||
# }
|
||||
```
|
||||
3. Enter ':wq' to save the changes to the nginx default config
|
||||
|
||||
## Step 7: Create simple http proxy configuration for AnythingLLM
|
||||
These instructions are for CLI configuration and assume you are logged in to EC2 instance as the ec2-user.
|
||||
1. $sudo vi /etc/nginx/conf.d/anything.conf
|
||||
2. Add the following configuration ensuring that you add your FQDN:.
|
||||
|
||||
```
|
||||
server {
|
||||
# Enable websocket connections for agent protocol.
|
||||
location ~* ^/api/agent-invocation/(.*) {
|
||||
proxy_pass http://0.0.0.0:3001;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "Upgrade";
|
||||
}
|
||||
|
||||
listen 80;
|
||||
server_name [insert FQDN here];
|
||||
location / {
|
||||
# Prevent timeouts on long-running requests.
|
||||
proxy_connect_timeout 605;
|
||||
proxy_send_timeout 605;
|
||||
proxy_read_timeout 605;
|
||||
send_timeout 605;
|
||||
keepalive_timeout 605;
|
||||
|
||||
# Enable readable HTTP Streaming for LLM streamed responses
|
||||
proxy_buffering off;
|
||||
proxy_cache off;
|
||||
|
||||
# Proxy your locally running service
|
||||
proxy_pass http://0.0.0.0:3001;
|
||||
}
|
||||
}
|
||||
```
|
||||
3. Enter ':wq' to save the changes to the anything config file
|
||||
|
||||
## Step 8: Test nginx http proxy config and restart nginx service
|
||||
These instructions are for CLI configuration and assume you are logged in to EC2 instance as the ec2-user.
|
||||
1. $sudo nginx -t
|
||||
2. $sudo systemctl restart nginx
|
||||
3. Navigate to http://FQDN in a browser and you should be proxied to the AnythingLLM web UI.
|
||||
|
||||
## Step 9: Generate/install cert
|
||||
These instructions are for CLI configuration and assume you are logged in to EC2 instance as the ec2-user.
|
||||
1. $sudo certbot --nginx -d [Insert FQDN here]
|
||||
Example command: $sudo certbot --nginx -d anythingllm.exampleorganization.org
|
||||
This command will generate the appropriate certificate files, write the files to /etc/letsencrypt/live/yourFQDN, and make updates to the nginx
|
||||
configuration file for anythingllm located at /etc/nginx/conf.d/anything.llm
|
||||
3. Enter the email address you would like to use for updates.
|
||||
4. Accept the terms of service.
|
||||
5. Accept or decline to receive communication from LetsEncrypt.
|
||||
|
||||
## Step 10: Test Cert installation
|
||||
1. $sudo cat /etc/nginx/conf.d/anything.conf
|
||||
Your should see a completely updated configuration that includes https/443 and a redirect configuration for http/80.
|
||||
2. Navigate to https://FQDN in a browser and you should be proxied to the AnythingLLM web UI.
|
||||
|
||||
## Step 11: (Optional) Remove temporary Inbound Traffic Rule for Security Group to certbot DNS verification
|
||||
1. Follow AWS instructions on deleting inbound rule (http port 80 0.0.0.0/0) for EC2 security group here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/working-with-security-groups.html#deleting-security-group-rule
|
||||
@@ -0,0 +1,234 @@
|
||||
{
|
||||
"AWSTemplateFormatVersion": "2010-09-09",
|
||||
"Description": "Create a stack that runs AnythingLLM on a single instance",
|
||||
"Parameters": {
|
||||
"InstanceType": {
|
||||
"Description": "EC2 instance type",
|
||||
"Type": "String",
|
||||
"Default": "t3.small"
|
||||
},
|
||||
"InstanceVolume": {
|
||||
"Description": "Storage size of disk on Instance in GB",
|
||||
"Type": "Number",
|
||||
"Default": 10,
|
||||
"MinValue": 4
|
||||
}
|
||||
},
|
||||
"Resources": {
|
||||
"AnythingLLMInstance": {
|
||||
"Type": "AWS::EC2::Instance",
|
||||
"Properties": {
|
||||
"ImageId": {
|
||||
"Fn::FindInMap": [
|
||||
"Region2AMI",
|
||||
{
|
||||
"Ref": "AWS::Region"
|
||||
},
|
||||
"AMI"
|
||||
]
|
||||
},
|
||||
"InstanceType": {
|
||||
"Ref": "InstanceType"
|
||||
},
|
||||
"SecurityGroupIds": [
|
||||
{
|
||||
"Ref": "AnythingLLMInstanceSecurityGroup"
|
||||
}
|
||||
],
|
||||
"BlockDeviceMappings": [
|
||||
{
|
||||
"DeviceName": {
|
||||
"Fn::FindInMap": [
|
||||
"Region2AMI",
|
||||
{
|
||||
"Ref": "AWS::Region"
|
||||
},
|
||||
"RootDeviceName"
|
||||
]
|
||||
},
|
||||
"Ebs": {
|
||||
"VolumeSize": {
|
||||
"Ref": "InstanceVolume"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"UserData": {
|
||||
"Fn::Base64": {
|
||||
"Fn::Join": [
|
||||
"",
|
||||
[
|
||||
"Content-Type: multipart/mixed; boundary=\"//\"\n",
|
||||
"MIME-Version: 1.0\n",
|
||||
"\n",
|
||||
"--//\n",
|
||||
"Content-Type: text/cloud-config; charset=\"us-ascii\"\n",
|
||||
"MIME-Version: 1.0\n",
|
||||
"Content-Transfer-Encoding: 7bit\n",
|
||||
"Content-Disposition: attachment; filename=\"cloud-config.txt\"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"#cloud-config\n",
|
||||
"cloud_final_modules:\n",
|
||||
"- [scripts-user, once-per-instance]\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"--//\n",
|
||||
"Content-Type: text/x-shellscript; charset=\"us-ascii\"\n",
|
||||
"MIME-Version: 1.0\n",
|
||||
"Content-Transfer-Encoding: 7bit\n",
|
||||
"Content-Disposition: attachment; filename=\"userdata.txt\"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"#!/bin/bash\n",
|
||||
"# check output of userdata script with sudo tail -f /var/log/cloud-init-output.log\n",
|
||||
"sudo yum install docker iptables -y\n",
|
||||
"sudo iptables -A OUTPUT -m owner ! --uid-owner root -d 169.254.169.254 -j DROP\n",
|
||||
"sudo systemctl enable docker\n",
|
||||
"sudo systemctl start docker\n",
|
||||
"mkdir -p /home/ec2-user/anythingllm\n",
|
||||
"touch /home/ec2-user/anythingllm/.env\n",
|
||||
"sudo chown ec2-user:ec2-user -R /home/ec2-user/anythingllm\n",
|
||||
"docker pull mintplexlabs/anythingllm\n",
|
||||
"docker run -d -p 3001:3001 --cap-add SYS_ADMIN -v /home/ec2-user/anythingllm:/app/server/storage -v /home/ec2-user/anythingllm/.env:/app/server/.env -e STORAGE_DIR=\"/app/server/storage\" mintplexlabs/anythingllm\n",
|
||||
"echo \"Container ID: $(sudo docker ps --latest --quiet)\"\n",
|
||||
"export ONLINE=$(curl -Is http://localhost:3001/api/ping | head -n 1|cut -d$' ' -f2)\n",
|
||||
"echo \"Health check: $ONLINE\"\n",
|
||||
"echo \"Setup complete! AnythingLLM instance is now online!\"\n",
|
||||
"\n",
|
||||
"--//--\n"
|
||||
]
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"AnythingLLMInstanceSecurityGroup": {
|
||||
"Type": "AWS::EC2::SecurityGroup",
|
||||
"Properties": {
|
||||
"GroupDescription": "AnythingLLM Instance Security Group",
|
||||
"SecurityGroupIngress": [
|
||||
{
|
||||
"IpProtocol": "tcp",
|
||||
"FromPort": "22",
|
||||
"ToPort": "22",
|
||||
"CidrIp": "0.0.0.0/0"
|
||||
},
|
||||
{
|
||||
"IpProtocol": "tcp",
|
||||
"FromPort": "3001",
|
||||
"ToPort": "3001",
|
||||
"CidrIp": "0.0.0.0/0"
|
||||
},
|
||||
{
|
||||
"IpProtocol": "tcp",
|
||||
"FromPort": "3001",
|
||||
"ToPort": "3001",
|
||||
"CidrIpv6": "::/0"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"Outputs": {
|
||||
"ServerIp": {
|
||||
"Description": "IP address of the AnythingLLM instance",
|
||||
"Value": {
|
||||
"Fn::GetAtt": [
|
||||
"AnythingLLMInstance",
|
||||
"PublicIp"
|
||||
]
|
||||
}
|
||||
},
|
||||
"ServerURL": {
|
||||
"Description": "URL of the AnythingLLM server",
|
||||
"Value": {
|
||||
"Fn::Join": [
|
||||
"",
|
||||
[
|
||||
"http://",
|
||||
{
|
||||
"Fn::GetAtt": [
|
||||
"AnythingLLMInstance",
|
||||
"PublicIp"
|
||||
]
|
||||
},
|
||||
":3001"
|
||||
]
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"Mappings": {
|
||||
"Region2AMI": {
|
||||
"ap-south-1": {
|
||||
"AMI": "ami-0e6329e222e662a52",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"eu-north-1": {
|
||||
"AMI": "ami-08c308b1bb265e927",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"eu-west-3": {
|
||||
"AMI": "ami-069d1ea6bc64443f0",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"eu-west-2": {
|
||||
"AMI": "ami-06a566ca43e14780d",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"eu-west-1": {
|
||||
"AMI": "ami-0a8dc52684ee2fee2",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"ap-northeast-3": {
|
||||
"AMI": "ami-0c8a89b455fae8513",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"ap-northeast-2": {
|
||||
"AMI": "ami-0ff56409a6e8ea2a0",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"ap-northeast-1": {
|
||||
"AMI": "ami-0ab0bbbd329f565e6",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"ca-central-1": {
|
||||
"AMI": "ami-033c256a10931f206",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"sa-east-1": {
|
||||
"AMI": "ami-0dabf4dab6b183eef",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"ap-southeast-1": {
|
||||
"AMI": "ami-0dc5785603ad4ff54",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"ap-southeast-2": {
|
||||
"AMI": "ami-0c5d61202c3b9c33e",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"eu-central-1": {
|
||||
"AMI": "ami-004359656ecac6a95",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"us-east-1": {
|
||||
"AMI": "ami-0cff7528ff583bf9a",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"us-east-2": {
|
||||
"AMI": "ami-02238ac43d6385ab3",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"us-west-1": {
|
||||
"AMI": "ami-01163e76c844a2129",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
},
|
||||
"us-west-2": {
|
||||
"AMI": "ami-0ceecbb0f30a902a6",
|
||||
"RootDeviceName": "/dev/xvda"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
# How to deploy a private AnythingLLM instance on DigitalOcean using Terraform
|
||||
|
||||
With a DigitalOcean account, you can easily deploy a private AnythingLLM instance using Terraform. This will create a URL that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys, and they will not be exposed. However, if you want your instance to be protected, it is highly recommended that you set a password once setup is complete.
|
||||
|
||||
The output of this Terraform configuration will be:
|
||||
- 1 DigitalOcean Droplet
|
||||
- An IP address to access your application
|
||||
|
||||
**Requirements**
|
||||
- An DigitalOcean account with billing information
|
||||
- Terraform installed on your local machine
|
||||
- Follow the instructions in the [official Terraform documentation](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) for your operating system.
|
||||
|
||||
## How to deploy on DigitalOcean
|
||||
Open your terminal and navigate to the `docker` folder
|
||||
1. Create a `.env` file by cloning the `.env.example`.
|
||||
2. Navigate to `digitalocean/terraform` folder.
|
||||
3. Replace the token value in the provider "digitalocean" block in main.tf with your DigitalOcean API token.
|
||||
4. Run the following commands to initialize Terraform, review the infrastructure changes, and apply them:
|
||||
```
|
||||
terraform init
|
||||
terraform plan
|
||||
terraform apply
|
||||
```
|
||||
Confirm the changes by typing yes when prompted.
|
||||
5. Once the deployment is complete, Terraform will output the public IP address of your droplet. You can access your application using this IP address.
|
||||
|
||||
## How to deploy on DigitalOcean
|
||||
To delete the resources created by Terraform, run the following command in the terminal:
|
||||
`
|
||||
terraform destroy
|
||||
`
|
||||
|
||||
## Please read this notice before submitting issues about your deployment
|
||||
|
||||
**Note:**
|
||||
Your instance will not be available instantly. Depending on the instance size you launched with it can take anywhere from 5-10 minutes to fully boot up.
|
||||
|
||||
If you want to check the instances progress, navigate to [your deployed instances](https://cloud.digitalocean.com/droplets) and connect to your instance via SSH in browser.
|
||||
|
||||
Once connected run `sudo tail -f /var/log/cloud-init-output.log` and wait for the file to conclude deployment of the docker image.
|
||||
|
||||
|
||||
Additionally, your use of this deployment process means you are responsible for any costs of these Digital Ocean resources fully.
|
||||
@@ -0,0 +1,52 @@
|
||||
terraform {
|
||||
required_version = ">= 1.0.0"
|
||||
|
||||
required_providers {
|
||||
digitalocean = {
|
||||
source = "digitalocean/digitalocean"
|
||||
version = "~> 2.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
provider "digitalocean" {
|
||||
# Add your DigitalOcean API token here
|
||||
token = "DigitalOcean API token"
|
||||
}
|
||||
|
||||
|
||||
resource "digitalocean_droplet" "anything_llm_instance" {
|
||||
image = "ubuntu-24-04-x64"
|
||||
name = "anything-llm-instance"
|
||||
region = "nyc3"
|
||||
size = "s-2vcpu-2gb"
|
||||
|
||||
user_data = templatefile("user_data.tp1", {
|
||||
env_content = local.formatted_env_content
|
||||
})
|
||||
}
|
||||
|
||||
locals {
|
||||
env_content = file("../../../docker/.env")
|
||||
formatted_env_content = join("\n", [
|
||||
for line in split("\n", local.env_content) :
|
||||
line
|
||||
if !(
|
||||
(
|
||||
substr(line, 0, 1) == "#"
|
||||
) ||
|
||||
(
|
||||
substr(line, 0, 3) == "UID"
|
||||
) ||
|
||||
(
|
||||
substr(line, 0, 3) == "GID"
|
||||
) ||
|
||||
(
|
||||
substr(line, 0, 11) == "CLOUD_BUILD"
|
||||
) ||
|
||||
(
|
||||
line == ""
|
||||
)
|
||||
)
|
||||
])
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
output "ip_address" {
|
||||
value = digitalocean_droplet.anything_llm_instance.ipv4_address
|
||||
description = "The public IP address of your droplet application."
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
#!/bin/bash
|
||||
# check output of userdata script with sudo tail -f /var/log/cloud-init-output.log
|
||||
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y docker.io
|
||||
sudo usermod -a -G docker ubuntu
|
||||
|
||||
sudo systemctl enable docker
|
||||
sudo systemctl start docker
|
||||
|
||||
mkdir -p /home/anythingllm
|
||||
cat <<EOF >/home/anythingllm/.env
|
||||
${env_content}
|
||||
EOF
|
||||
|
||||
sudo docker pull mintplexlabs/anythingllm
|
||||
sudo docker run -d -p 3001:3001 --cap-add SYS_ADMIN -v /home/anythingllm:/app/server/storage -v /home/anythingllm/.env:/app/server/.env -e STORAGE_DIR="/app/server/storage" mintplexlabs/anythingllm
|
||||
echo "Container ID: $(sudo docker ps --latest --quiet)"
|
||||
|
||||
export ONLINE=$(curl -Is http://localhost:3001/api/ping | head -n 1|cut -d$' ' -f2)
|
||||
echo "Health check: $ONLINE"
|
||||
echo "Setup complete! AnythingLLM instance is now online!"
|
||||
@@ -0,0 +1,54 @@
|
||||
# How to deploy a private AnythingLLM instance on GCP
|
||||
|
||||
With a GCP account you can easily deploy a private AnythingLLM instance on GCP. This will create a url that you can access from any browser over HTTP (HTTPS not supported). This single instance will run on your own keys and they will not be exposed - however if you want your instance to be protected it is highly recommend that you set a password once setup is complete.
|
||||
|
||||
The output of this cloudformation stack will be:
|
||||
- 1 GCP VM
|
||||
- 1 Security Group with 0.0.0.0/0 access on Ports 22 & 3001
|
||||
- 1 GCP VM Volume `gb2` of 10Gib minimum
|
||||
|
||||
**Requirements**
|
||||
- An GCP account with billing information.
|
||||
|
||||
## How to deploy on GCP
|
||||
Open your terminal
|
||||
1. Log in to your GCP account using the following command:
|
||||
```
|
||||
gcloud auth login
|
||||
```
|
||||
|
||||
2. After successful login, Run the following command to create a deployment using the Deployment Manager CLI:
|
||||
|
||||
```
|
||||
|
||||
gcloud deployment-manager deployments create anything-llm-deployment --config gcp/deployment/gcp_deploy_anything_llm.yaml
|
||||
|
||||
```
|
||||
|
||||
Once you execute these steps, the CLI will initiate the deployment process on GCP based on your configuration file. You can monitor the deployment status and view the outputs using the Google Cloud Console or the Deployment Manager CLI commands.
|
||||
|
||||
```
|
||||
gcloud compute instances get-serial-port-output anything-llm-instance
|
||||
```
|
||||
|
||||
ssh into the instance
|
||||
|
||||
```
|
||||
gcloud compute ssh anything-llm-instance
|
||||
```
|
||||
|
||||
Delete the deployment
|
||||
```
|
||||
gcloud deployment-manager deployments delete anything-llm-deployment
|
||||
```
|
||||
|
||||
## Please read this notice before submitting issues about your deployment
|
||||
|
||||
**Note:**
|
||||
Your instance will not be available instantly. Depending on the instance size you launched with it can take anywhere from 5-10 minutes to fully boot up.
|
||||
|
||||
If you want to check the instances progress, navigate to [your deployed instances](https://console.cloud.google.com/compute/instances) and connect to your instance via SSH in browser.
|
||||
|
||||
Once connected run `sudo tail -f /var/log/cloud-init-output.log` and wait for the file to conclude deployment of the docker image.
|
||||
|
||||
Additionally, your use of this deployment process means you are responsible for any costs of these GCP resources fully.
|
||||
@@ -0,0 +1,45 @@
|
||||
resources:
|
||||
- name: anything-llm-instance
|
||||
type: compute.v1.instance
|
||||
properties:
|
||||
zone: us-central1-a
|
||||
machineType: zones/us-central1-a/machineTypes/n1-standard-1
|
||||
disks:
|
||||
- deviceName: boot
|
||||
type: PERSISTENT
|
||||
boot: true
|
||||
autoDelete: true
|
||||
initializeParams:
|
||||
sourceImage: projects/ubuntu-os-cloud/global/images/family/ubuntu-2004-lts
|
||||
diskSizeGb: 10
|
||||
networkInterfaces:
|
||||
- network: global/networks/default
|
||||
accessConfigs:
|
||||
- name: External NAT
|
||||
type: ONE_TO_ONE_NAT
|
||||
metadata:
|
||||
items:
|
||||
- key: startup-script
|
||||
value: |
|
||||
#!/bin/bash
|
||||
# check output of userdata script with sudo tail -f /var/log/cloud-init-output.log
|
||||
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y docker.io
|
||||
sudo usermod -a -G docker ubuntu
|
||||
sudo systemctl enable docker
|
||||
sudo systemctl start docker
|
||||
|
||||
mkdir -p /home/anythingllm
|
||||
touch /home/anythingllm/.env
|
||||
sudo chown -R ubuntu:ubuntu /home/anythingllm
|
||||
|
||||
sudo docker pull mintplexlabs/anythingllm
|
||||
sudo docker run -d -p 3001:3001 --cap-add SYS_ADMIN -v /home/anythingllm:/app/server/storage -v /home/anythingllm/.env:/app/server/.env -e STORAGE_DIR="/app/server/storage" mintplexlabs/anythingllm
|
||||
echo "Container ID: $(sudo docker ps --latest --quiet)"
|
||||
|
||||
export ONLINE=$(curl -Is http://localhost:3001/api/ping | head -n 1|cut -d$' ' -f2)
|
||||
echo "Health check: $ONLINE"
|
||||
|
||||
echo "Setup complete! AnythingLLM instance is now online!"
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
# Patterns to ignore when building packages.
|
||||
# This supports shell glob matching, relative path matching, and
|
||||
# negation (prefixed with !). Only one pattern per line.
|
||||
.DS_Store
|
||||
# Common VCS dirs
|
||||
.git/
|
||||
.gitignore
|
||||
.bzr/
|
||||
.bzrignore
|
||||
.hg/
|
||||
.hgignore
|
||||
.svn/
|
||||
# Common backup files
|
||||
*.swp
|
||||
*.bak
|
||||
*.tmp
|
||||
*.orig
|
||||
*~
|
||||
# Various IDEs
|
||||
.project
|
||||
.idea/
|
||||
*.tmproj
|
||||
.vscode/
|
||||
@@ -0,0 +1,7 @@
|
||||
apiVersion: v2
|
||||
name: anythingllm
|
||||
description: The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, MCP compatibility, and more.
|
||||
type: application
|
||||
version: 1.0.0
|
||||
appVersion: "1.85.0"
|
||||
icon: https://raw.githubusercontent.com/Mintplex-Labs/anything-llm/refs/heads/master/frontend/public/favicon.png
|
||||
@@ -0,0 +1,149 @@
|
||||
# anythingllm
|
||||
|
||||
  
|
||||
|
||||
|
||||
|
||||
[AnythingLLM](https://github.com/Mintplex-Labs/anything-llm)
|
||||
|
||||
The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, MCP compatibility, and more.
|
||||
|
||||
**Configuration & Usage**
|
||||
|
||||
- **Config vs Secrets:** This chart exposes application configuration via two mechanisms:
|
||||
- `config` (in `values.yaml`) — rendered into a `ConfigMap` and injected using `envFrom` in the pod. Do NOT place sensitive values (API keys, secrets) in `config` because `ConfigMap`s are not encrypted.
|
||||
- `env` / `envFrom` — the preferred way to inject secrets. Use Kubernetes `Secret` objects and reference them from `env` (with `valueFrom.secretKeyRef`) or `envFrom.secretRef`.
|
||||
|
||||
- **Storage & STORAGE_DIR mapping:** The chart creates (or mounts) a `PersistentVolumeClaim` using the `persistentVolume.*` settings. The container mount path is set from `persistentVolume.mountPath`. Ensure the container `STORAGE_DIR` config key matches that path (defaults are set in `values.yaml`).
|
||||
|
||||
**Providing API keys & secrets (recommended)**
|
||||
|
||||
Use Kubernetes Secrets. Below are example workflows and `values.yaml` snippets.
|
||||
|
||||
1) Create a Kubernetes Secret with API keys:
|
||||
|
||||
```
|
||||
kubectl create secret generic openai-secret --from-literal=OPENAI_KEY="sk-..."
|
||||
# or from a file
|
||||
# kubectl create secret generic openai-secret --from-file=OPENAI_KEY=/path/to/keyfile
|
||||
```
|
||||
|
||||
2) Reference the Secret from `values.yaml` using `envFrom` (recommended when your secret contains multiple env keys):
|
||||
|
||||
```yaml
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: openai-secret
|
||||
```
|
||||
|
||||
This will inject all key/value pairs from the `openai-secret` Secret as environment variables in the container.
|
||||
|
||||
3) Or reference a single secret key via `env` (explicit mapping):
|
||||
|
||||
```yaml
|
||||
env:
|
||||
- name: OPENAI_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: openai-secret
|
||||
key: OPENAI_KEY
|
||||
```
|
||||
|
||||
Notes:
|
||||
- Avoid placing secret values into `config:` (the chart's `ConfigMap`) — `ConfigMap`s are visible to anyone who can read the namespace. Use `Secret` objects for any credentials/tokens.
|
||||
- If you use a GitOps workflow, consider integrating an external secret operator (ExternalSecrets, SealedSecrets, etc.) so you don't store raw secrets in Git.
|
||||
|
||||
**Example `values-secret.yaml` to pass during `helm install`**
|
||||
|
||||
```yaml
|
||||
image:
|
||||
repository: mintplexlabs/anythingllm
|
||||
tag: "1.15.0"
|
||||
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 3001
|
||||
|
||||
# Reference secret containing API keys
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: openai-secret
|
||||
|
||||
# Optionally override other values
|
||||
persistentVolume:
|
||||
size: 16Gi
|
||||
mountPath: /storage
|
||||
```
|
||||
|
||||
Install with:
|
||||
|
||||
```
|
||||
helm install my-anythingllm ./anythingllm -f values-secret.yaml
|
||||
```
|
||||
|
||||
**Best practices & tips**
|
||||
|
||||
- Use `envFrom` for convenience when many environment variables are stored in a single `Secret` and use `env`/`valueFrom` for explicit single-key mappings.
|
||||
- Use `kubectl create secret generic` or your secrets management solution. If you need to reference multiple different provider keys (OpenAI, Anthropic, etc.), create a single `Secret` with multiple keys or multiple Secrets and add multiple `envFrom` entries.
|
||||
- Keep probe paths and `service.port` aligned. If your probes fail after deployment, check that the probe `port` matches the container port (or named port `http`) and that the `path` is valid.
|
||||
- For storage, if you have a pre-existing PVC set `persistentVolume.existingClaim` to the PVC name; the chart will mount that claim (and will not attempt to create a new PVC).
|
||||
- For production, provide resource `requests` and `limits` in `values.yaml` to prevent scheduler starvation and to control cost.
|
||||
|
||||
## Values
|
||||
|
||||
| Key | Type | Default | Description |
|
||||
|-----|------|---------|-------------|
|
||||
| affinity | object | `{}` | |
|
||||
| config.DISABLE_TELEMETRY | string | `"true"` | |
|
||||
| config.GID | string | `"1000"` | |
|
||||
| config.NODE_ENV | string | `"production"` | |
|
||||
| config.STORAGE_DIR | string | `"/storage"` | |
|
||||
| config.UID | string | `"1000"` | |
|
||||
| env | object | `{}` | |
|
||||
| envFrom | object | `{}` | |
|
||||
| fullnameOverride | string | `""` | |
|
||||
| image.pullPolicy | string | `"IfNotPresent"` | |
|
||||
| image.repository | string | `"mintplexlabs/anythingllm"` | |
|
||||
| image.tag | string | `"1.15.0"` | |
|
||||
| imagePullSecrets | list | `[]` | |
|
||||
| ingress.annotations | object | `{}` | |
|
||||
| ingress.className | string | `""` | |
|
||||
| ingress.enabled | bool | `false` | |
|
||||
| ingress.hosts[0].host | string | `"chart-example.local"` | |
|
||||
| ingress.hosts[0].paths[0].path | string | `"/"` | |
|
||||
| ingress.hosts[0].paths[0].pathType | string | `"ImplementationSpecific"` | |
|
||||
| ingress.tls | list | `[]` | |
|
||||
| initContainers | list | `[]` | |
|
||||
| livenessProbe.failureThreshold | int | `3` | |
|
||||
| livenessProbe.httpGet.path | string | `"/v1/api/health"` | |
|
||||
| livenessProbe.httpGet.port | int | `8888` | |
|
||||
| livenessProbe.initialDelaySeconds | int | `15` | |
|
||||
| livenessProbe.periodSeconds | int | `5` | |
|
||||
| nameOverride | string | `""` | |
|
||||
| nodeSelector | object | `{}` | |
|
||||
| persistentVolume.accessModes[0] | string | `"ReadWriteOnce"` | |
|
||||
| persistentVolume.annotations | object | `{}` | |
|
||||
| persistentVolume.existingClaim | string | `""` | |
|
||||
| persistentVolume.labels | object | `{}` | |
|
||||
| persistentVolume.mountPath | string | `"/storage"` | |
|
||||
| persistentVolume.size | string | `"8Gi"` | |
|
||||
| podAnnotations | object | `{}` | |
|
||||
| podLabels | object | `{}` | |
|
||||
| podSecurityContext.fsGroup | int | `1000` | |
|
||||
| readinessProbe.httpGet.path | string | `"/v1/api/health"` | |
|
||||
| readinessProbe.httpGet.port | int | `8888` | |
|
||||
| readinessProbe.initialDelaySeconds | int | `15` | |
|
||||
| readinessProbe.periodSeconds | int | `5` | |
|
||||
| readinessProbe.successThreshold | int | `2` | |
|
||||
| replicaCount | int | `1` | |
|
||||
| resources | object | `{}` | |
|
||||
| securityContext | object | `{}` | |
|
||||
| service.port | int | `3001` | |
|
||||
| service.type | string | `"ClusterIP"` | |
|
||||
| serviceAccount.annotations | object | `{}` | |
|
||||
| serviceAccount.automount | bool | `true` | |
|
||||
| serviceAccount.create | bool | `true` | |
|
||||
| serviceAccount.name | string | `""` | |
|
||||
| tolerations | list | `[]` | |
|
||||
| volumeMounts | list | `[]` | |
|
||||
| volumes | list | `[]` | |
|
||||
@@ -0,0 +1,103 @@
|
||||
{{ template "chart.header" . }}
|
||||
{{ template "chart.deprecationWarning" . }}
|
||||
|
||||
{{ template "chart.badgesSection" . }}
|
||||
|
||||
|
||||
|
||||
[AnythingLLM](https://github.com/Mintplex-Labs/anything-llm)
|
||||
|
||||
{{ template "chart.description" . }}
|
||||
|
||||
{{ template "chart.homepageLine" . }}
|
||||
|
||||
{{ template "chart.maintainersSection" . }}
|
||||
|
||||
{{ template "chart.sourcesSection" . }}
|
||||
|
||||
{{ template "chart.requirementsSection" . }}
|
||||
|
||||
**Configuration & Usage**
|
||||
|
||||
- **Config vs Secrets:** This chart exposes application configuration via two mechanisms:
|
||||
- `config` (in `values.yaml`) — rendered into a `ConfigMap` and injected using `envFrom` in the pod. Do NOT place sensitive values (API keys, secrets) in `config` because `ConfigMap`s are not encrypted.
|
||||
- `env` / `envFrom` — the preferred way to inject secrets. Use Kubernetes `Secret` objects and reference them from `env` (with `valueFrom.secretKeyRef`) or `envFrom.secretRef`.
|
||||
|
||||
- **Storage & STORAGE_DIR mapping:** The chart creates (or mounts) a `PersistentVolumeClaim` using the `persistentVolume.*` settings. The container mount path is set from `persistentVolume.mountPath`. Ensure the container `STORAGE_DIR` config key matches that path (defaults are set in `values.yaml`).
|
||||
|
||||
|
||||
**Providing API keys & secrets (recommended)**
|
||||
|
||||
Use Kubernetes Secrets. Below are example workflows and `values.yaml` snippets.
|
||||
|
||||
1) Create a Kubernetes Secret with API keys:
|
||||
|
||||
```
|
||||
kubectl create secret generic openai-secret --from-literal=OPENAI_KEY="sk-..."
|
||||
# or from a file
|
||||
# kubectl create secret generic openai-secret --from-file=OPENAI_KEY=/path/to/keyfile
|
||||
```
|
||||
|
||||
2) Reference the Secret from `values.yaml` using `envFrom` (recommended when your secret contains multiple env keys):
|
||||
|
||||
```yaml
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: openai-secret
|
||||
```
|
||||
|
||||
This will inject all key/value pairs from the `openai-secret` Secret as environment variables in the container.
|
||||
|
||||
3) Or reference a single secret key via `env` (explicit mapping):
|
||||
|
||||
```yaml
|
||||
env:
|
||||
- name: OPENAI_KEY
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: openai-secret
|
||||
key: OPENAI_KEY
|
||||
```
|
||||
|
||||
Notes:
|
||||
- Avoid placing secret values into `config:` (the chart's `ConfigMap`) — `ConfigMap`s are visible to anyone who can read the namespace. Use `Secret` objects for any credentials/tokens.
|
||||
- If you use a GitOps workflow, consider integrating an external secret operator (ExternalSecrets, SealedSecrets, etc.) so you don't store raw secrets in Git.
|
||||
|
||||
|
||||
**Example `values-secret.yaml` to pass during `helm install`**
|
||||
|
||||
```yaml
|
||||
image:
|
||||
repository: mintplexlabs/anythingllm
|
||||
tag: "1.15.0"
|
||||
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 3001
|
||||
|
||||
# Reference secret containing API keys
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: openai-secret
|
||||
|
||||
# Optionally override other values
|
||||
persistentVolume:
|
||||
size: 16Gi
|
||||
mountPath: /storage
|
||||
```
|
||||
|
||||
Install with:
|
||||
|
||||
```
|
||||
helm install my-anythingllm ./anythingllm -f values-secret.yaml
|
||||
```
|
||||
|
||||
**Best practices & tips**
|
||||
|
||||
- Use `envFrom` for convenience when many environment variables are stored in a single `Secret` and use `env`/`valueFrom` for explicit single-key mappings.
|
||||
- Use `kubectl create secret generic` or your secrets management solution. If you need to reference multiple different provider keys (OpenAI, Anthropic, etc.), create a single `Secret` with multiple keys or multiple Secrets and add multiple `envFrom` entries.
|
||||
- Keep probe paths and `service.port` aligned. If your probes fail after deployment, check that the probe `port` matches the container port (or named port `http`) and that the `path` is valid.
|
||||
- For storage, if you have a pre-existing PVC set `persistentVolume.existingClaim` to the PVC name; the chart will mount that claim (and will not attempt to create a new PVC).
|
||||
- For production, provide resource `requests` and `limits` in `values.yaml` to prevent scheduler starvation and to control cost.
|
||||
|
||||
{{ template "chart.valuesSection" . }}
|
||||
@@ -0,0 +1,28 @@
|
||||
1. Get the application URL by running these commands:
|
||||
|
||||
{{- if .Values.ingress.enabled }}
|
||||
{{- range $host := .Values.ingress.hosts }}
|
||||
{{- range .paths }}
|
||||
http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ .path }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
||||
{{- else if contains "NodePort" .Values.service.type }}
|
||||
export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "anythingllm.fullname" . }})
|
||||
export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
|
||||
echo http://$NODE_IP:$NODE_PORT
|
||||
|
||||
{{- else if contains "LoadBalancer" .Values.service.type }}
|
||||
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
|
||||
You can watch its status by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "anythingllm.fullname" . }}'
|
||||
export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "anythingllm.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}")
|
||||
echo http://$SERVICE_IP:{{ .Values.service.port }}
|
||||
|
||||
{{- else if contains "ClusterIP" .Values.service.type }}
|
||||
export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "anythingllm.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
|
||||
export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
|
||||
echo "To access locally, run:"
|
||||
echo " kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT"
|
||||
echo "Then visit http://127.0.0.1:8080"
|
||||
|
||||
{{- end }}
|
||||
@@ -0,0 +1,62 @@
|
||||
{{/*
|
||||
Expand the name of the chart.
|
||||
*/}}
|
||||
{{- define "anythingllm.name" -}}
|
||||
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create a default fully qualified app name.
|
||||
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
|
||||
If release name contains chart name it will be used as a full name.
|
||||
*/}}
|
||||
{{- define "anythingllm.fullname" -}}
|
||||
{{- if .Values.fullnameOverride }}
|
||||
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- $name := default .Chart.Name .Values.nameOverride }}
|
||||
{{- if contains $name .Release.Name }}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
|
||||
{{- else }}
|
||||
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create chart name and version as used by the chart label.
|
||||
*/}}
|
||||
{{- define "anythingllm.chart" -}}
|
||||
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Common labels
|
||||
*/}}
|
||||
{{- define "anythingllm.labels" -}}
|
||||
helm.sh/chart: {{ include "anythingllm.chart" . }}
|
||||
{{ include "anythingllm.selectorLabels" . }}
|
||||
{{- if .Chart.AppVersion }}
|
||||
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
|
||||
{{- end }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Selector labels
|
||||
*/}}
|
||||
{{- define "anythingllm.selectorLabels" -}}
|
||||
app.kubernetes.io/name: {{ include "anythingllm.name" . }}
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
{{- end }}
|
||||
|
||||
{{/*
|
||||
Create the name of the service account to use
|
||||
*/}}
|
||||
{{- define "anythingllm.serviceAccountName" -}}
|
||||
{{- if .Values.serviceAccount.create }}
|
||||
{{- default (include "anythingllm.fullname" .) .Values.serviceAccount.name }}
|
||||
{{- else }}
|
||||
{{- default "default" .Values.serviceAccount.name }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,10 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "anythingllm.labels" . | nindent 4 }}
|
||||
name: {{ include "anythingllm.fullname" . }}-config
|
||||
data:
|
||||
{{- range $key, $value := .Values.config }}
|
||||
{{ $key }}: "{{ $value }}"
|
||||
{{- end }}
|
||||
@@ -0,0 +1,83 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: {{ include "anythingllm.fullname" . }}
|
||||
labels:
|
||||
{{- include "anythingllm.labels" . | nindent 4 }}
|
||||
spec:
|
||||
replicas: {{ .Values.replicaCount }}
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "anythingllm.selectorLabels" . | nindent 6 }}
|
||||
{{- with .Values.strategy }}
|
||||
strategy:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
template:
|
||||
metadata:
|
||||
{{- with .Values.podAnnotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
labels:
|
||||
{{- include "anythingllm.labels" . | nindent 8 }}
|
||||
{{- with .Values.podLabels }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
{{- with .Values.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
serviceAccountName: {{ include "anythingllm.serviceAccountName" . }}
|
||||
securityContext:
|
||||
{{- toYaml .Values.podSecurityContext | nindent 8 }}
|
||||
{{- with .Values.initContainers }}
|
||||
initContainers:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
containers:
|
||||
- name: {{ .Chart.Name }}
|
||||
securityContext:
|
||||
{{- toYaml .Values.securityContext | nindent 12 }}
|
||||
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
|
||||
imagePullPolicy: {{ .Values.image.pullPolicy }}
|
||||
{{- with .Values.env }}
|
||||
env:
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
envFrom:
|
||||
- configMapRef:
|
||||
name: {{ include "anythingllm.fullname" . }}-config
|
||||
{{- with .Values.envFrom }}
|
||||
{{- toYaml . | nindent 12 }}
|
||||
{{- end }}
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: {{ .Values.service.port }}
|
||||
protocol: TCP
|
||||
livenessProbe:
|
||||
{{- toYaml .Values.livenessProbe | nindent 12 }}
|
||||
readinessProbe:
|
||||
{{- toYaml .Values.readinessProbe | nindent 12 }}
|
||||
resources:
|
||||
{{- toYaml .Values.resources | nindent 12 }}
|
||||
volumeMounts:
|
||||
- name: storage
|
||||
mountPath: {{ .Values.persistentVolume.mountPath }}
|
||||
volumes:
|
||||
- name: storage
|
||||
persistentVolumeClaim:
|
||||
claimName: {{ include "anythingllm.fullname" . }}-storage-claim
|
||||
{{- with .Values.nodeSelector }}
|
||||
nodeSelector:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- with .Values.affinity }}
|
||||
affinity:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- with .Values.tolerations }}
|
||||
tolerations:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,4 @@
|
||||
{{ range .Values.extraObjects }}
|
||||
---
|
||||
{{ tpl (toYaml .) $ }}
|
||||
{{ end }}
|
||||
@@ -0,0 +1,38 @@
|
||||
{{- if .Values.httpRoute.enabled -}}
|
||||
{{- $fullName := include "anythingllm.fullname" . -}}
|
||||
{{- $svcPort := .Values.service.port -}}
|
||||
apiVersion: gateway.networking.k8s.io/v1
|
||||
kind: HTTPRoute
|
||||
metadata:
|
||||
name: {{ $fullName }}
|
||||
labels:
|
||||
{{- include "anythingllm.labels" . | nindent 4 }}
|
||||
{{- with .Values.httpRoute.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
parentRefs:
|
||||
{{- with .Values.httpRoute.parentRefs }}
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
{{- with .Values.httpRoute.hostnames }}
|
||||
hostnames:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
rules:
|
||||
{{- range .Values.httpRoute.rules }}
|
||||
{{- with .matches }}
|
||||
- matches:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
{{- with .filters }}
|
||||
filters:
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
backendRefs:
|
||||
- name: {{ $fullName }}
|
||||
port: {{ $svcPort }}
|
||||
weight: 1
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,61 @@
|
||||
{{- if .Values.ingress.enabled -}}
|
||||
{{- $fullName := include "anythingllm.fullname" . -}}
|
||||
{{- $svcPort := .Values.service.port -}}
|
||||
{{- if and .Values.ingress.className (not (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion)) }}
|
||||
{{- if not (hasKey .Values.ingress.annotations "kubernetes.io/ingress.class") }}
|
||||
{{- $_ := set .Values.ingress.annotations "kubernetes.io/ingress.class" .Values.ingress.className}}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion -}}
|
||||
apiVersion: networking.k8s.io/v1
|
||||
{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
|
||||
apiVersion: networking.k8s.io/v1beta1
|
||||
{{- else -}}
|
||||
apiVersion: extensions/v1beta1
|
||||
{{- end }}
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: {{ $fullName }}
|
||||
labels:
|
||||
{{- include "anythingllm.labels" . | nindent 4 }}
|
||||
{{- with .Values.ingress.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
spec:
|
||||
{{- if and .Values.ingress.className (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion) }}
|
||||
ingressClassName: {{ .Values.ingress.className }}
|
||||
{{- end }}
|
||||
{{- if .Values.ingress.tls }}
|
||||
tls:
|
||||
{{- range .Values.ingress.tls }}
|
||||
- hosts:
|
||||
{{- range .hosts }}
|
||||
- {{ . | quote }}
|
||||
{{- end }}
|
||||
secretName: {{ .secretName }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
rules:
|
||||
{{- range .Values.ingress.hosts }}
|
||||
- host: {{ .host | quote }}
|
||||
http:
|
||||
paths:
|
||||
{{- range .paths }}
|
||||
- path: {{ .path }}
|
||||
{{- if and .pathType (semverCompare ">=1.18-0" $.Capabilities.KubeVersion.GitVersion) }}
|
||||
pathType: {{ .pathType }}
|
||||
{{- end }}
|
||||
backend:
|
||||
{{- if semverCompare ">=1.19-0" $.Capabilities.KubeVersion.GitVersion }}
|
||||
service:
|
||||
name: {{ $fullName }}
|
||||
port:
|
||||
number: {{ $svcPort }}
|
||||
{{- else }}
|
||||
serviceName: {{ $fullName }}
|
||||
servicePort: {{ $svcPort }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,33 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
{{- if .Values.persistentVolume.annotations }}
|
||||
annotations:
|
||||
{{ toYaml .Values.persistentVolume.annotations | indent 4 }}
|
||||
{{- end }}
|
||||
labels:
|
||||
{{- include "anythingllm.labels" . | nindent 4 }}
|
||||
{{- with .Values.persistentVolume.labels }}
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
name: {{ include "anythingllm.fullname" . }}-storage-claim
|
||||
spec:
|
||||
accessModes:
|
||||
{{- toYaml .Values.persistentVolume.accessModes | nindent 4 }}
|
||||
{{- if .Values.persistentVolume.storageClass }}
|
||||
{{- if (eq "-" .Values.persistentVolume.storageClass) }}
|
||||
storageClassName: ""
|
||||
{{- else }}
|
||||
storageClassName: "{{ .Values.persistentVolume.storageClass }}"
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
resources:
|
||||
requests:
|
||||
storage: {{ .Values.persistentVolume.size }}
|
||||
{{- if .Values.persistentVolume.volumeName }}
|
||||
volumeName: "{{ .Values.persistentVolume.volumeName }}"
|
||||
{{- end -}}
|
||||
{{- with .Values.persistentVolume.selector }}
|
||||
selector:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,15 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: {{ include "anythingllm.fullname" . }}
|
||||
labels:
|
||||
{{- include "anythingllm.labels" . | nindent 4 }}
|
||||
spec:
|
||||
type: {{ .Values.service.type }}
|
||||
ports:
|
||||
- port: {{ .Values.service.port }}
|
||||
targetPort: http
|
||||
protocol: TCP
|
||||
name: http
|
||||
selector:
|
||||
{{- include "anythingllm.selectorLabels" . | nindent 4 }}
|
||||
@@ -0,0 +1,13 @@
|
||||
{{- if .Values.serviceAccount.create -}}
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: {{ include "anythingllm.serviceAccountName" . }}
|
||||
labels:
|
||||
{{- include "anythingllm.labels" . | nindent 4 }}
|
||||
{{- with .Values.serviceAccount.annotations }}
|
||||
annotations:
|
||||
{{- toYaml . | nindent 4 }}
|
||||
{{- end }}
|
||||
automountServiceAccountToken: {{ .Values.serviceAccount.automount }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,16 @@
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: "{{ include "anythingllm.fullname" . }}-test-connection"
|
||||
labels:
|
||||
{{- include "anythingllm.labels" . | nindent 4 }}
|
||||
annotations:
|
||||
"helm.sh/hook": test
|
||||
spec:
|
||||
containers:
|
||||
- name: healthcheck
|
||||
image: curlimages/curl:8.1.2
|
||||
command: ["sh", "-c"]
|
||||
args:
|
||||
- "curl -fsS -o /dev/null http://{{ include "anythingllm.fullname" . }}:{{ .Values.service.port }}|| exit 1"
|
||||
restartPolicy: Never
|
||||
@@ -0,0 +1,269 @@
|
||||
replicaCount: 1
|
||||
|
||||
initContainers: []
|
||||
# - name: init-myservice
|
||||
# image: busybox
|
||||
# command: ['sh', '-c', 'chown -R 1000:1000 /storage']
|
||||
|
||||
image:
|
||||
repository: mintplexlabs/anythingllm
|
||||
pullPolicy: IfNotPresent
|
||||
tag: "1.15.0"
|
||||
|
||||
imagePullSecrets: []
|
||||
nameOverride: ""
|
||||
fullnameOverride: ""
|
||||
|
||||
persistentVolume:
|
||||
# AnythingLLM storage data Persistent Volume access modes
|
||||
# Must match those of existing PV or dynamic provisioner
|
||||
# Ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
|
||||
#
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
|
||||
# AnythingLLM storage data Persistent Volume labels
|
||||
#
|
||||
labels: {}
|
||||
|
||||
# AnythingLLM storage data Persistent Volume annotations
|
||||
#
|
||||
annotations: {}
|
||||
|
||||
# AnythingLLM storage data Persistent Volume existing claim name
|
||||
# If defined, PVC must be created manually before volume will be bound
|
||||
#
|
||||
existingClaim: ""
|
||||
|
||||
# AnythingLLM storage data Persistent Volume size
|
||||
#
|
||||
size: 8Gi
|
||||
|
||||
# AnythingLLM storage data Persistent Volume mount path
|
||||
# Must match the STORAGE_DIR config value
|
||||
#
|
||||
mountPath: /app/server/storage
|
||||
|
||||
# AnythingLLM storage data Persistent Volume Storage Class
|
||||
# If defined, storageClassName: <storageClass>
|
||||
# If set to "-", storageClassName: "", which disables dynamic provisioning
|
||||
# If undefined (the default) or set to null, no storageClassName spec is
|
||||
# set, choosing the default provisioner. (gp2 on AWS, standard on
|
||||
# GKE, AWS & OpenStack)
|
||||
#
|
||||
storageClass: ""
|
||||
|
||||
# AnythingLLM storage data Persistent Volume Claim Selector
|
||||
# Useful if Persistent Volumes have been provisioned in advance
|
||||
# Ref: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#selector
|
||||
#
|
||||
selector: {}
|
||||
# selector:
|
||||
# matchLabels:
|
||||
# release: "stable"
|
||||
# matchExpressions:
|
||||
# - { key: environment, operator: In, values: [ dev ] }
|
||||
|
||||
# AnythingLLM storage data Persistent Volume Name
|
||||
# Useful if Persistent Volumes have been provisioned in advance and you want to use a specific one
|
||||
#
|
||||
volumeName: ""
|
||||
|
||||
serviceAccount:
|
||||
# Specifies whether a service account should be created
|
||||
create: true
|
||||
# Automatically mount a ServiceAccount's API credentials?
|
||||
automount: true
|
||||
# Annotations to add to the service account
|
||||
annotations: {}
|
||||
# The name of the service account to use.
|
||||
# If not set and create is true, a name is generated using the fullname template
|
||||
name: ""
|
||||
|
||||
# The Anything LLM application deployment strategy
|
||||
# This is set to "Recreate" by default as AnythingLLM support is not yet
|
||||
# production ready. Once it is, this can be changed to "RollingUpdate"
|
||||
# Ref: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy
|
||||
#
|
||||
strategy:
|
||||
# Type of deployment. Can be "Recreate" or "RollingUpdate". Default is "Recreate"
|
||||
type: Recreate
|
||||
# If type is "RollingUpdate", the following values can be set:
|
||||
# rollingUpdate:
|
||||
# maxUnavailable: 1
|
||||
# maxSurge: 1
|
||||
|
||||
podAnnotations: {}
|
||||
podLabels: {}
|
||||
|
||||
podSecurityContext:
|
||||
# fsGroup needs to be set as the same as the uid/gid used to run the application
|
||||
# in order to have the right permissions on mounted volumes
|
||||
fsGroup: 1000
|
||||
|
||||
securityContext: {}
|
||||
# capabilities:
|
||||
# drop:
|
||||
# - ALL
|
||||
# readOnlyRootFilesystem: true
|
||||
# runAsNonRoot: true
|
||||
# runAsUser: 1000
|
||||
|
||||
# AnythingLLM configuration options, these are stored in a ConfigMap and passed
|
||||
# to the container as environment variables.
|
||||
# See https://github.com/Mintplex-Labs/anything-llm/blob/render/docker/.env.example
|
||||
# for all available environment variables to use as configuration options
|
||||
#
|
||||
config:
|
||||
DISABLE_TELEMETRY: "true"
|
||||
NODE_ENV: production
|
||||
STORAGE_DIR: /app/server/storage
|
||||
UID: "1000"
|
||||
GID: "1000"
|
||||
|
||||
# The preferred method for setting secret environment variables
|
||||
# Ref: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#define-a-container-environment-variable-with-data-from-a-single-secret
|
||||
#
|
||||
env: {}
|
||||
# - name: OPEN_AI_KEY
|
||||
# valueFrom:
|
||||
# secretKeyRef:
|
||||
# name: openai-secret
|
||||
# key: openai_key
|
||||
|
||||
# Typically used to reference a pre existing Secret containing multiple environment variables
|
||||
# Ref: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#define-a-container-environment-variable-with-data-from-a-single-secret
|
||||
#
|
||||
envFrom: {}
|
||||
# - secretRef:
|
||||
# name: mysecret
|
||||
|
||||
service:
|
||||
type: ClusterIP
|
||||
port: 3001
|
||||
|
||||
# -- Expose the service via gateway-api HTTPRoute
|
||||
# Requires Gateway API resources and suitable controller installed within the cluster
|
||||
# (see: https://gateway-api.sigs.k8s.io/guides/)
|
||||
httpRoute:
|
||||
# HTTPRoute enabled.
|
||||
enabled: false
|
||||
# HTTPRoute annotations.
|
||||
annotations: {}
|
||||
# Which Gateways this Route is attached to.
|
||||
parentRefs:
|
||||
- name: gateway
|
||||
sectionName: http
|
||||
# namespace: default
|
||||
# Hostnames matching HTTP header.
|
||||
hostnames:
|
||||
- chart-example.local
|
||||
# List of rules and filters applied.
|
||||
rules:
|
||||
- matches:
|
||||
- path:
|
||||
type: PathPrefix
|
||||
value: /headers
|
||||
# filters:
|
||||
# - type: RequestHeaderModifier
|
||||
# requestHeaderModifier:
|
||||
# set:
|
||||
# - name: My-Overwrite-Header
|
||||
# value: this-is-the-only-value
|
||||
# remove:
|
||||
# - User-Agent
|
||||
# - matches:
|
||||
# - path:
|
||||
# type: PathPrefix
|
||||
# value: /echo
|
||||
# headers:
|
||||
# - name: version
|
||||
# value: v2
|
||||
|
||||
ingress:
|
||||
enabled: false
|
||||
className: ""
|
||||
annotations: {}
|
||||
# kubernetes.io/ingress.class: nginx
|
||||
# kubernetes.io/tls-acme: "true"
|
||||
hosts:
|
||||
- host: chart-example.local
|
||||
paths:
|
||||
- path: /
|
||||
pathType: ImplementationSpecific
|
||||
tls: []
|
||||
# - secretName: chart-example-tls
|
||||
# hosts:
|
||||
# - chart-example.local
|
||||
|
||||
resources: {}
|
||||
# We usually recommend not to specify default resources and to leave this as a conscious
|
||||
# choice for the user. This also increases chances charts run on environments with little
|
||||
# resources, such as Minikube. If you do want to specify resources, uncomment the following
|
||||
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
|
||||
# limits:
|
||||
# cpu: 100m
|
||||
# memory: 128Mi
|
||||
# requests:
|
||||
# cpu: 100m
|
||||
# memory: 128Mi
|
||||
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /v1/api/health
|
||||
port: 8888
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 5
|
||||
successThreshold: 2
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /v1/api/health
|
||||
port: 8888
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 5
|
||||
failureThreshold: 3
|
||||
|
||||
# Additional volumes on the output Deployment definition.
|
||||
#
|
||||
volumes: []
|
||||
# - name: foo
|
||||
# secret:
|
||||
# secretName: mysecret
|
||||
# optional: false
|
||||
|
||||
# Additional volumeMounts on the output Deployment definition.
|
||||
#
|
||||
volumeMounts: []
|
||||
# - name: foo
|
||||
# mountPath: "/etc/foo"
|
||||
# readOnly: true
|
||||
|
||||
nodeSelector: {}
|
||||
|
||||
tolerations: []
|
||||
|
||||
affinity: {}
|
||||
|
||||
## Array of extra manifests/obhects to create
|
||||
#
|
||||
extraObjects: []
|
||||
# - apiVersion: external-secrets.io/v1beta1
|
||||
# kind: ExternalSecret
|
||||
# metadata:
|
||||
# name: open-ai-api-key-external-secret
|
||||
# namespace: default
|
||||
# spec:
|
||||
# refreshInterval: 1h
|
||||
# secretStoreRef:
|
||||
# name: vault
|
||||
# kind: ClusterSecretStore
|
||||
# target:
|
||||
# name: open-ai-api-key-secret
|
||||
# template:
|
||||
# type: Opaque
|
||||
# data:
|
||||
# - secretKey: open_ai_key
|
||||
# remoteRef:
|
||||
# key: secret/data/anything-llm
|
||||
# property: open_ai_key
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
# With this dockerfile in a Huggingface space you will get an entire AnythingLLM instance running
|
||||
# in your space with all features you would normally get from the docker based version of AnythingLLM.
|
||||
#
|
||||
# How to use
|
||||
# - Login to https://huggingface.co/spaces
|
||||
# - Click on "Create new Space"
|
||||
# - Name the space and select "Docker" as the SDK w/ a blank template
|
||||
# - The default 2vCPU/16GB machine is OK. The more the merrier.
|
||||
# - Decide if you want your AnythingLLM Space public or private.
|
||||
# **You might want to stay private until you at least set a password or enable multi-user mode**
|
||||
# - Click "Create Space"
|
||||
# - Click on "Settings" on top of page (https://huggingface.co/spaces/<username>/<space-name>/settings)
|
||||
# - Scroll to "Persistent Storage" and select the lowest tier of now - you can upgrade if you run out.
|
||||
# - Confirm and continue storage upgrade
|
||||
# - Go to "Files" Tab (https://huggingface.co/spaces/<username>/<space-name>/tree/main)
|
||||
# - Click "Add Files"
|
||||
# - Upload this file or create a file named `Dockerfile` and copy-paste this content into it. "Commit to main" and save.
|
||||
# - Your container will build and boot. You now have AnythingLLM on HuggingFace. Your data is stored in the persistent storage attached.
|
||||
# Have Fun 🤗
|
||||
# Have issues? Check the logs on HuggingFace for clues.
|
||||
FROM mintplexlabs/anythingllm:render
|
||||
|
||||
USER root
|
||||
RUN mkdir -p /data/storage
|
||||
RUN ln -s /data/storage /storage
|
||||
USER anythingllm
|
||||
|
||||
ENV STORAGE_DIR="/data/storage"
|
||||
ENV SERVER_PORT=7860
|
||||
|
||||
ENTRYPOINT ["/bin/bash", "/usr/local/bin/render-entrypoint.sh"]
|
||||
@@ -0,0 +1,214 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: anything-llm-volume
|
||||
annotations:
|
||||
pv.beta.kubernetes.io/uid: "1000"
|
||||
pv.beta.kubernetes.io/gid: "1000"
|
||||
spec:
|
||||
storageClassName: gp2
|
||||
capacity:
|
||||
storage: 5Gi
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
awsElasticBlockStore:
|
||||
# This is the volume UUID from AWS EC2 EBS Volumes list.
|
||||
volumeID: "{{ anythingllm_awsElasticBlockStore_volumeID }}"
|
||||
fsType: ext4
|
||||
nodeAffinity:
|
||||
required:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: topology.kubernetes.io/zone
|
||||
operator: In
|
||||
values:
|
||||
- us-east-1c
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: anything-llm-volume-claim
|
||||
namespace: "{{ namespace }}"
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 5Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: anything-llm
|
||||
namespace: "{{ namespace }}"
|
||||
labels:
|
||||
anything-llm: "true"
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
k8s-app: anything-llm
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxSurge: 0%
|
||||
maxUnavailable: 100%
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
anything-llm: "true"
|
||||
k8s-app: anything-llm
|
||||
app.kubernetes.io/name: anything-llm
|
||||
app.kubernetes.io/part-of: anything-llm
|
||||
annotations:
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/path: /metrics
|
||||
prometheus.io/port: "9090"
|
||||
spec:
|
||||
serviceAccountName: "default"
|
||||
terminationGracePeriodSeconds: 10
|
||||
securityContext:
|
||||
fsGroup: 1000
|
||||
runAsNonRoot: true
|
||||
runAsGroup: 1000
|
||||
runAsUser: 1000
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: topology.kubernetes.io/zone
|
||||
operator: In
|
||||
values:
|
||||
- us-east-1c
|
||||
containers:
|
||||
- name: anything-llm
|
||||
resources:
|
||||
limits:
|
||||
memory: "1Gi"
|
||||
cpu: "500m"
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "250m"
|
||||
imagePullPolicy: IfNotPresent
|
||||
image: "mintplexlabs/anythingllm:render"
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: true
|
||||
capabilities:
|
||||
add:
|
||||
- SYS_ADMIN
|
||||
runAsNonRoot: true
|
||||
runAsGroup: 1000
|
||||
runAsUser: 1000
|
||||
command:
|
||||
# Specify a command to override the Dockerfile's ENTRYPOINT.
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
set -x -e
|
||||
sleep 3
|
||||
echo "AWS_REGION: $AWS_REGION"
|
||||
echo "SERVER_PORT: $SERVER_PORT"
|
||||
echo "NODE_ENV: $NODE_ENV"
|
||||
echo "STORAGE_DIR: $STORAGE_DIR"
|
||||
{
|
||||
cd /app/server/ &&
|
||||
npx prisma generate --schema=./prisma/schema.prisma &&
|
||||
npx prisma migrate deploy --schema=./prisma/schema.prisma &&
|
||||
node /app/server/index.js
|
||||
echo "Server process exited with status $?"
|
||||
} &
|
||||
{
|
||||
node /app/collector/index.js
|
||||
echo "Collector process exited with status $?"
|
||||
} &
|
||||
wait -n
|
||||
exit $?
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /v1/api/health
|
||||
port: 8888
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 5
|
||||
successThreshold: 2
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /v1/api/health
|
||||
port: 8888
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 5
|
||||
failureThreshold: 3
|
||||
env:
|
||||
- name: AWS_REGION
|
||||
value: "{{ aws_region }}"
|
||||
- name: AWS_ACCESS_KEY_ID
|
||||
value: "{{ aws_access_id }}"
|
||||
- name: AWS_SECRET_ACCESS_KEY
|
||||
value: "{{ aws_access_secret }}"
|
||||
- name: SERVER_PORT
|
||||
value: "3001"
|
||||
- name: JWT_SECRET
|
||||
value: "my-random-string-for-seeding" # Please generate random string at least 12 chars long.
|
||||
- name: STORAGE_DIR
|
||||
value: "/storage"
|
||||
- name: NODE_ENV
|
||||
value: "production"
|
||||
- name: UID
|
||||
value: "1000"
|
||||
- name: GID
|
||||
value: "1000"
|
||||
volumeMounts:
|
||||
- name: anything-llm-server-storage-volume-mount
|
||||
mountPath: /storage
|
||||
volumes:
|
||||
- name: anything-llm-server-storage-volume-mount
|
||||
persistentVolumeClaim:
|
||||
claimName: anything-llm-volume-claim
|
||||
---
|
||||
# This serves the UI and the backend.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: anything-llm-ingress
|
||||
namespace: "{{ namespace }}"
|
||||
annotations:
|
||||
external-dns.alpha.kubernetes.io/hostname: "{{ namespace }}-chat.{{ base_domain }}"
|
||||
kubernetes.io/ingress.class: "internal-ingress"
|
||||
nginx.ingress.kubernetes.io/rewrite-target: /
|
||||
ingress.kubernetes.io/ssl-redirect: "false"
|
||||
spec:
|
||||
rules:
|
||||
- host: "{{ namespace }}-chat.{{ base_domain }}"
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: anything-llm-svc
|
||||
port:
|
||||
number: 3001
|
||||
tls: # < placing a host in the TLS config will indicate a cert should be created
|
||||
- hosts:
|
||||
- "{{ namespace }}-chat.{{ base_domain }}"
|
||||
secretName: letsencrypt-prod
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
labels:
|
||||
kubernetes.io/name: anything-llm
|
||||
name: anything-llm-svc
|
||||
namespace: "{{ namespace }}"
|
||||
spec:
|
||||
ports:
|
||||
# "port" is external port, and "targetPort" is internal.
|
||||
- port: 3301
|
||||
targetPort: 3001
|
||||
name: traffic
|
||||
- port: 9090
|
||||
targetPort: 9090
|
||||
name: metrics
|
||||
selector:
|
||||
k8s-app: anything-llm
|
||||
@@ -0,0 +1,225 @@
|
||||
# OpenShift-compatible Dockerfile for AnythingLLM
|
||||
#
|
||||
# This Dockerfile is specifically designed for OpenShift deployments which use
|
||||
# arbitrary UIDs with GID 0 (root group). Do NOT use this for standard Docker
|
||||
# or Docker Compose deployments - use the main docker/Dockerfile instead.
|
||||
#
|
||||
# Key differences from the standard Dockerfile:
|
||||
# - User is added to supplementary group 0 (root) for OpenShift compatibility
|
||||
# - All files are owned by group 0 and are group-writable (chmod g+w)
|
||||
# - /etc/passwd is made group-writable for dynamic UID injection
|
||||
# - Uses a modified entrypoint that handles arbitrary UID scenarios
|
||||
|
||||
# Setup base image
|
||||
FROM ubuntu:noble-20251013 AS base
|
||||
|
||||
# Build arguments
|
||||
ARG ARG_UID=1000
|
||||
ARG ARG_GID=1000
|
||||
|
||||
FROM base AS build-arm64
|
||||
RUN echo "Preparing build of AnythingLLM image for arm64 architecture"
|
||||
|
||||
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
|
||||
|
||||
# Install system dependencies
|
||||
# hadolint ignore=DL3008,DL3013
|
||||
RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
|
||||
DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
|
||||
unzip curl gnupg libgfortran5 libgbm1 tzdata netcat-openbsd \
|
||||
libasound2t64 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 \
|
||||
libgcc1 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libx11-6 libx11-xcb1 libxcb1 \
|
||||
libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 \
|
||||
libxss1 libxtst6 ca-certificates fonts-liberation libappindicator3-1 libnss3 lsb-release \
|
||||
xdg-utils git build-essential ffmpeg && \
|
||||
mkdir -p /etc/apt/keyrings && \
|
||||
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg && \
|
||||
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_18.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list && \
|
||||
apt-get update && \
|
||||
# Install node and yarn
|
||||
apt-get install -yq --no-install-recommends nodejs && \
|
||||
curl -LO https://github.com/yarnpkg/yarn/releases/download/v1.22.19/yarn_1.22.19_all.deb \
|
||||
&& dpkg -i yarn_1.22.19_all.deb \
|
||||
&& rm yarn_1.22.19_all.deb && \
|
||||
# Install uvx (pinned to 0.6.10) for MCP support
|
||||
curl -LsSf https://astral.sh/uv/0.6.10/install.sh | sh && \
|
||||
mv /root/.local/bin/uv /usr/local/bin/uv && \
|
||||
mv /root/.local/bin/uvx /usr/local/bin/uvx && \
|
||||
echo "Installed uvx! $(uv --version)" && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Create a group and user with specific UID and GID.
|
||||
# The user's primary group is ARG_GID (default 1000) for Docker/Compose backward compatibility.
|
||||
# The user is also added to the root group (GID 0) as a supplementary group so that
|
||||
# OpenShift's arbitrary-UID-with-GID-0 model works via group-writable permissions.
|
||||
RUN (getent passwd "$ARG_UID" && userdel -f "$(getent passwd "$ARG_UID" | cut -d: -f1)") || true && \
|
||||
if [ "$ARG_GID" != "0" ]; then \
|
||||
(getent group "$ARG_GID" && groupdel "$(getent group "$ARG_GID" | cut -d: -f1)") || true && \
|
||||
groupadd -g "$ARG_GID" anythingllm && \
|
||||
useradd -l -u "$ARG_UID" -m -d /app -s /bin/bash -g anythingllm -G 0 anythingllm; \
|
||||
else \
|
||||
useradd -l -u "$ARG_UID" -m -d /app -s /bin/bash -g 0 anythingllm; \
|
||||
fi && \
|
||||
mkdir -p /app/frontend/ /app/server/ /app/collector/ /app/.cache /app/.yarn && \
|
||||
chown -R anythingllm:0 /app && \
|
||||
chmod -R g+w /app && \
|
||||
chmod g=u /etc/passwd
|
||||
|
||||
# Copy docker helper scripts (use OpenShift-specific entrypoint)
|
||||
COPY ./cloud-deployments/openshift/docker-entrypoint.sh /usr/local/bin/
|
||||
COPY ./docker/docker-healthcheck.sh /usr/local/bin/
|
||||
COPY --chown=anythingllm:0 ./docker/.env.example /app/server/.env
|
||||
|
||||
# Ensure the scripts are executable
|
||||
RUN chmod +x /usr/local/bin/docker-entrypoint.sh && \
|
||||
chmod +x /usr/local/bin/docker-healthcheck.sh
|
||||
|
||||
USER anythingllm
|
||||
WORKDIR /app
|
||||
|
||||
# Puppeteer does not ship with an ARM86 compatible build for Chromium
|
||||
# so web-scraping would be broken in arm docker containers unless we patch it
|
||||
# by manually installing a compatible chromedriver.
|
||||
RUN echo "Need to patch Puppeteer x Chromium support for ARM86 - installing dep!" && \
|
||||
curl -fSL https://webassets.anythingllm.com/chromium-1088-linux-arm64.zip -o chrome-linux.zip && \
|
||||
unzip chrome-linux.zip && \
|
||||
rm -rf chrome-linux.zip
|
||||
|
||||
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
|
||||
ENV CHROME_PATH=/app/chrome-linux/chrome
|
||||
ENV PUPPETEER_EXECUTABLE_PATH=/app/chrome-linux/chrome
|
||||
|
||||
RUN echo "Done running arm64 specific installation steps"
|
||||
|
||||
#############################################
|
||||
|
||||
# amd64-specific stage
|
||||
FROM base AS build-amd64
|
||||
RUN echo "Preparing build of AnythingLLM image for non-ARM architecture"
|
||||
|
||||
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
|
||||
|
||||
# Install system dependencies
|
||||
# hadolint ignore=DL3008,DL3013
|
||||
RUN DEBIAN_FRONTEND=noninteractive apt-get update && \
|
||||
DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
|
||||
curl gnupg libgfortran5 libgbm1 tzdata netcat-openbsd \
|
||||
libasound2t64 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 \
|
||||
libgcc1 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libx11-6 libx11-xcb1 libxcb1 \
|
||||
libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 \
|
||||
libxss1 libxtst6 ca-certificates fonts-liberation libappindicator3-1 libnss3 lsb-release \
|
||||
xdg-utils git build-essential ffmpeg && \
|
||||
mkdir -p /etc/apt/keyrings && \
|
||||
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg && \
|
||||
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_18.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list && \
|
||||
apt-get update && \
|
||||
# Install node and yarn
|
||||
apt-get install -yq --no-install-recommends nodejs && \
|
||||
curl -LO https://github.com/yarnpkg/yarn/releases/download/v1.22.19/yarn_1.22.19_all.deb \
|
||||
&& dpkg -i yarn_1.22.19_all.deb \
|
||||
&& rm yarn_1.22.19_all.deb && \
|
||||
# Install uvx (pinned to 0.6.10) for MCP support
|
||||
curl -LsSf https://astral.sh/uv/0.6.10/install.sh | sh && \
|
||||
mv /root/.local/bin/uv /usr/local/bin/uv && \
|
||||
mv /root/.local/bin/uvx /usr/local/bin/uvx && \
|
||||
echo "Installed uvx! $(uv --version)" && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Create a group and user with specific UID and GID.
|
||||
# The user's primary group is ARG_GID (default 1000) for Docker/Compose backward compatibility.
|
||||
# The user is also added to the root group (GID 0) as a supplementary group so that
|
||||
# OpenShift's arbitrary-UID-with-GID-0 model works via group-writable permissions.
|
||||
RUN (getent passwd "$ARG_UID" && userdel -f "$(getent passwd "$ARG_UID" | cut -d: -f1)") || true && \
|
||||
if [ "$ARG_GID" != "0" ]; then \
|
||||
(getent group "$ARG_GID" && groupdel "$(getent group "$ARG_GID" | cut -d: -f1)") || true && \
|
||||
groupadd -g "$ARG_GID" anythingllm && \
|
||||
useradd -l -u "$ARG_UID" -m -d /app -s /bin/bash -g anythingllm -G 0 anythingllm; \
|
||||
else \
|
||||
useradd -l -u "$ARG_UID" -m -d /app -s /bin/bash -g 0 anythingllm; \
|
||||
fi && \
|
||||
mkdir -p /app/frontend/ /app/server/ /app/collector/ /app/.cache /app/.yarn && \
|
||||
chown -R anythingllm:0 /app && \
|
||||
chmod -R g+w /app && \
|
||||
chmod g=u /etc/passwd
|
||||
|
||||
# Copy docker helper scripts (use OpenShift-specific entrypoint)
|
||||
COPY ./cloud-deployments/openshift/docker-entrypoint.sh /usr/local/bin/
|
||||
COPY ./docker/docker-healthcheck.sh /usr/local/bin/
|
||||
COPY --chown=anythingllm:0 ./docker/.env.example /app/server/.env
|
||||
|
||||
# Ensure the scripts are executable
|
||||
RUN chmod +x /usr/local/bin/docker-entrypoint.sh && \
|
||||
chmod +x /usr/local/bin/docker-healthcheck.sh
|
||||
|
||||
#############################################
|
||||
# COMMON BUILD FLOW FOR ALL ARCHS
|
||||
#############################################
|
||||
|
||||
# hadolint ignore=DL3006
|
||||
FROM build-${TARGETARCH} AS build
|
||||
RUN echo "Running common build flow of AnythingLLM image for all architectures"
|
||||
|
||||
USER anythingllm
|
||||
WORKDIR /app
|
||||
|
||||
# Install & Build frontend layer
|
||||
# Use BUILDPLATFORM to run on the native host architecture (not emulated).
|
||||
# This avoids esbuild crashing under QEMU when cross-compiling.
|
||||
# The output (static HTML/CSS/JS) is platform-independent.
|
||||
FROM --platform=$BUILDPLATFORM node:18-slim AS frontend-build
|
||||
WORKDIR /app/frontend
|
||||
COPY ./frontend/package.json ./frontend/yarn.lock ./
|
||||
RUN yarn install --network-timeout 100000 && yarn cache clean
|
||||
COPY ./frontend/ ./
|
||||
RUN yarn build
|
||||
WORKDIR /app
|
||||
|
||||
# Install server layer
|
||||
# Also pull and build collector deps (chromium issues prevent bad bindings)
|
||||
FROM build AS backend-build
|
||||
COPY --chown=anythingllm:0 ./server /app/server/
|
||||
WORKDIR /app/server
|
||||
RUN yarn install --production --network-timeout 100000 && yarn cache clean
|
||||
WORKDIR /app
|
||||
|
||||
# Install collector dependencies
|
||||
COPY --chown=anythingllm:0 ./collector/ ./collector/
|
||||
WORKDIR /app/collector
|
||||
ENV PUPPETEER_DOWNLOAD_BASE_URL=https://storage.googleapis.com/chrome-for-testing-public
|
||||
RUN yarn install --production --network-timeout 100000 && yarn cache clean
|
||||
|
||||
WORKDIR /app
|
||||
USER anythingllm
|
||||
|
||||
# Since we are building from backend-build we just need to move built frontend into server/public
|
||||
FROM backend-build AS production-build
|
||||
WORKDIR /app
|
||||
COPY --chown=anythingllm:0 --from=frontend-build /app/frontend/dist /app/server/public
|
||||
|
||||
# Ensure all app files are owned by group 0 (root) and group-writable so that:
|
||||
# - Docker/Compose: access works via user ownership (UID match)
|
||||
# - OpenShift: access works via group ownership (GID 0 match + g+w)
|
||||
# - Generic Kubernetes: access works via user ownership or fsGroup
|
||||
# This must run after all COPY and yarn install steps to cover node_modules, etc.
|
||||
USER root
|
||||
RUN chown -R anythingllm:0 /app && \
|
||||
chmod -R g+w /app && \
|
||||
mkdir -p /app/server/storage
|
||||
|
||||
# Setup the environment
|
||||
ENV NODE_ENV=production
|
||||
ENV ANYTHING_LLM_RUNTIME=docker
|
||||
ENV DEPLOYMENT_VERSION=1.15.0
|
||||
ENV HOME=/app
|
||||
|
||||
# Setup the healthcheck
|
||||
HEALTHCHECK --interval=1m --timeout=10s --start-period=1m \
|
||||
CMD /bin/bash /usr/local/bin/docker-healthcheck.sh || exit 1
|
||||
|
||||
USER anythingllm
|
||||
|
||||
# Run the server
|
||||
# CMD ["sh", "-c", "tail -f /dev/null"] # For development: keep container open
|
||||
ENTRYPOINT ["/bin/bash", "/usr/local/bin/docker-entrypoint.sh"]
|
||||
@@ -0,0 +1,146 @@
|
||||
> [!IMPORTANT]
|
||||
> This is a community-maintained template and is not officially supported by the AnythingLLM team. You could encounter issues or even deployment failures in future versions of AnythingLLM. We do our best to keep this template and all community contributions backwards compatible, but we cannot guarantee it.
|
||||
|
||||
# OpenShift Deployment Template for AnythingLLM
|
||||
|
||||
This directory contains a specialized Dockerfile and entrypoint script for deploying AnythingLLM on **Red Hat OpenShift** clusters.
|
||||
|
||||
## Why This Template Exists
|
||||
|
||||
OpenShift has a unique security model that differs from standard Docker/Kubernetes deployments:
|
||||
|
||||
1. **Arbitrary UIDs**: OpenShift runs containers with randomly assigned user IDs (UIDs) that don't exist in `/etc/passwd`
|
||||
2. **GID 0 Requirement**: All containers run with GID 0 (root group) as the primary group
|
||||
3. **Restricted SCCs**: The default Security Context Constraints (SCCs) prevent containers from running as specific users
|
||||
|
||||
These requirements are incompatible with the standard AnythingLLM Docker image, which uses a fixed `anythingllm` user with UID/GID 1000.
|
||||
|
||||
## Key Differences from Standard Dockerfile
|
||||
|
||||
| Feature | Standard Docker | OpenShift Template |
|
||||
|---------|-----------------|-------------------|
|
||||
| File ownership | `anythingllm:anythingllm` | `anythingllm:0` (root group) |
|
||||
| File permissions | Standard | Group-writable (`g+w`) |
|
||||
| `/etc/passwd` | Read-only | Group-writable for UID injection |
|
||||
| Supplementary groups | None | Added to group 0 |
|
||||
| Entrypoint | Standard | Handles arbitrary UID scenarios |
|
||||
|
||||
## When to Use This Template
|
||||
|
||||
Use this template **only** if you are deploying to:
|
||||
- Red Hat OpenShift (any version)
|
||||
- OKD (OpenShift Origin)
|
||||
- Any Kubernetes cluster with OpenShift-style restricted SCCs
|
||||
|
||||
**Do NOT use this for:**
|
||||
- Standard Docker deployments
|
||||
- Docker Compose
|
||||
- Generic Kubernetes (use the standard image with appropriate `securityContext`)
|
||||
- Cloud container services (AWS ECS, Google Cloud Run, Azure Container Instances)
|
||||
|
||||
## Building the Image
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
docker build -f cloud-deployments/openshift/Dockerfile -t anythingllm:openshift .
|
||||
```
|
||||
|
||||
For multi-architecture builds:
|
||||
|
||||
```bash
|
||||
docker buildx build \
|
||||
--platform linux/amd64,linux/arm64 \
|
||||
-f cloud-deployments/openshift/Dockerfile \
|
||||
-t your-registry/anythingllm:openshift \
|
||||
--push .
|
||||
```
|
||||
|
||||
## Deploying to OpenShift
|
||||
|
||||
### Using `oc` CLI
|
||||
|
||||
```bash
|
||||
# Create a new project (namespace)
|
||||
oc new-project anythingllm
|
||||
|
||||
# Create a deployment
|
||||
oc new-app your-registry/anythingllm:openshift
|
||||
|
||||
# Expose the service
|
||||
oc expose svc/anythingllm --port=3001
|
||||
|
||||
# Set required environment variables
|
||||
oc set env deployment/anythingllm \
|
||||
STORAGE_DIR=/app/server/storage \
|
||||
JWT_SECRET=$(openssl rand -hex 32)
|
||||
```
|
||||
|
||||
### Using a DeploymentConfig YAML
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: anythingllm
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: anythingllm
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: anythingllm
|
||||
spec:
|
||||
containers:
|
||||
- name: anythingllm
|
||||
image: your-registry/anythingllm:openshift
|
||||
ports:
|
||||
- containerPort: 3001
|
||||
env:
|
||||
- name: STORAGE_DIR
|
||||
value: /app/server/storage
|
||||
- name: JWT_SECRET
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: anythingllm-secrets
|
||||
key: jwt-secret
|
||||
volumeMounts:
|
||||
- name: storage
|
||||
mountPath: /app/server/storage
|
||||
volumes:
|
||||
- name: storage
|
||||
persistentVolumeClaim:
|
||||
claimName: anythingllm-storage
|
||||
```
|
||||
|
||||
## Persistent Storage
|
||||
|
||||
OpenShift PersistentVolumeClaims work with this image. Ensure the PVC is created before deployment:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: anythingllm-storage
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Permission Denied Errors
|
||||
|
||||
If you see permission errors, verify:
|
||||
1. You're using this OpenShift-specific image, not the standard one
|
||||
2. The PVC has correct access modes
|
||||
3. No custom SCCs are overriding the default behavior
|
||||
|
||||
### User Not Found in passwd
|
||||
|
||||
The entrypoint script automatically handles this by injecting a passwd entry at runtime. If issues persist, check that `/etc/passwd` is group-writable in your image.
|
||||
@@ -0,0 +1,39 @@
|
||||
#!/bin/bash
|
||||
|
||||
# OpenShift runs containers with an arbitrary UID that may not exist in /etc/passwd.
|
||||
# Many tools (npm, prisma, git, etc.) expect a passwd entry for the running user.
|
||||
# If the current UID has no entry, dynamically add one using nss_wrapper-style injection.
|
||||
if ! whoami &> /dev/null 2>&1; then
|
||||
if [ -w /etc/passwd ]; then
|
||||
echo "anythingllm:x:$(id -u):0:AnythingLLM User:/app:/bin/bash" >> /etc/passwd
|
||||
fi
|
||||
fi
|
||||
export HOME=/app
|
||||
|
||||
# Check if STORAGE_DIR is set
|
||||
if [ -z "$STORAGE_DIR" ]; then
|
||||
echo "================================================================"
|
||||
echo "⚠️ ⚠️ ⚠️ WARNING: STORAGE_DIR environment variable is not set! ⚠️ ⚠️ ⚠️"
|
||||
echo ""
|
||||
echo "Not setting this will result in data loss on container restart since"
|
||||
echo "the application will not have a persistent storage location."
|
||||
echo "It can also result in weird errors in various parts of the application."
|
||||
echo ""
|
||||
echo "Please run the container with the official docker command at"
|
||||
echo "https://docs.anythingllm.com/installation-docker/quickstart"
|
||||
echo ""
|
||||
echo "⚠️ ⚠️ ⚠️ WARNING: STORAGE_DIR environment variable is not set! ⚠️ ⚠️ ⚠️"
|
||||
echo "================================================================"
|
||||
fi
|
||||
|
||||
{
|
||||
cd /app/server/ &&
|
||||
# Disable Prisma CLI telemetry (https://www.prisma.io/docs/orm/tools/prisma-cli#how-to-opt-out-of-data-collection)
|
||||
export CHECKPOINT_DISABLE=1 &&
|
||||
npx prisma generate --schema=./prisma/schema.prisma &&
|
||||
npx prisma migrate deploy --schema=./prisma/schema.prisma &&
|
||||
node /app/server/index.js
|
||||
} &
|
||||
{ node /app/collector/index.js; } &
|
||||
wait -n
|
||||
exit $?
|
||||
@@ -0,0 +1,9 @@
|
||||
# Placeholder .env file for collector runtime
|
||||
|
||||
# Port the collector listens on. The server must use the same COLLECTOR_PORT value when running services separately.
|
||||
# COLLECTOR_PORT=8888
|
||||
|
||||
# This enables HTTP request/response logging in development. Set value to truthy string to enable, leave empty value or comment out to disable
|
||||
# ENABLE_HTTP_LOGGER=""
|
||||
# This enables timestamps for the HTTP Logger. Set value to true to enable, leave empty or comment out to disable
|
||||
# ENABLE_HTTP_LOGGER_TIMESTAMPS=""
|
||||
@@ -0,0 +1,9 @@
|
||||
hotdir/*
|
||||
!hotdir/__HOTDIR__.md
|
||||
yarn-error.log
|
||||
!yarn.lock
|
||||
outputs
|
||||
scripts
|
||||
.env.development
|
||||
.env.production
|
||||
.env.test
|
||||
@@ -0,0 +1 @@
|
||||
v18.18.0
|
||||
@@ -0,0 +1,77 @@
|
||||
process.env.STORAGE_DIR = "test-storage";
|
||||
const fs = require("fs");
|
||||
const path = require("path");
|
||||
|
||||
// Mock fix-path as a noop to prevent SIGSEGV (segfault)
|
||||
// Returns ESM-style default export for dynamic import()
|
||||
jest.mock("fix-path", () => ({ default: jest.fn() }));
|
||||
|
||||
const { FFMPEGWrapper } = require("../../../../utils/WhisperProviders/ffmpeg");
|
||||
|
||||
const describeRunner = process.env.GITHUB_ACTIONS ? describe.skip : describe;
|
||||
|
||||
describeRunner("FFMPEGWrapper", () => {
|
||||
/** @type { import("../../../../utils/WhisperProviders/ffmpeg/index").FFMPEGWrapper } */
|
||||
let ffmpeg;
|
||||
const testDir = path.resolve(__dirname, "../../../../storage/tmp");
|
||||
const inputPath = path.resolve(testDir, "test-input.wav");
|
||||
const outputPath = path.resolve(testDir, "test-output.wav");
|
||||
|
||||
beforeEach(() => {
|
||||
ffmpeg = new FFMPEGWrapper();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
if (fs.existsSync(inputPath)) fs.rmSync(inputPath);
|
||||
if (fs.existsSync(outputPath)) fs.rmSync(outputPath);
|
||||
});
|
||||
|
||||
it("should find ffmpeg executable", async () => {
|
||||
const knownPath = await ffmpeg.ffmpegPath();
|
||||
expect(knownPath).toBeDefined();
|
||||
expect(typeof knownPath).toBe("string");
|
||||
expect(knownPath.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it("should validate ffmpeg executable", async () => {
|
||||
const knownPath = await ffmpeg.ffmpegPath();
|
||||
expect(ffmpeg.isValidFFMPEG(knownPath)).toBe(true);
|
||||
});
|
||||
|
||||
it("should return false for invalid ffmpeg path", () => {
|
||||
expect(ffmpeg.isValidFFMPEG("/invalid/path/to/ffmpeg")).toBe(false);
|
||||
});
|
||||
|
||||
it("should convert audio file to wav format", async () => {
|
||||
if (!fs.existsSync(testDir)) fs.mkdirSync(testDir, { recursive: true });
|
||||
|
||||
const sampleUrl =
|
||||
"https://github.com/ringcentral/ringcentral-api-docs/blob/main/resources/sample1.wav?raw=true";
|
||||
|
||||
const response = await fetch(sampleUrl);
|
||||
if (!response.ok)
|
||||
throw new Error(
|
||||
`Failed to download sample file: ${response.statusText}`
|
||||
);
|
||||
|
||||
const buffer = await response.arrayBuffer();
|
||||
fs.writeFileSync(inputPath, Buffer.from(buffer));
|
||||
|
||||
const result = await ffmpeg.convertAudioToWav(inputPath, outputPath);
|
||||
|
||||
expect(result).toBe(true);
|
||||
expect(fs.existsSync(outputPath)).toBe(true);
|
||||
|
||||
const stats = fs.statSync(outputPath);
|
||||
expect(stats.size).toBeGreaterThan(0);
|
||||
}, 30000);
|
||||
|
||||
it("should throw error when conversion fails", () => {
|
||||
const nonExistentFile = path.resolve(testDir, "non-existent-file.wav");
|
||||
const outputPath = path.resolve(testDir, "test-output-fail.wav");
|
||||
|
||||
expect(async () => {
|
||||
return await ffmpeg.convertAudioToWav(nonExistentFile, outputPath);
|
||||
}).rejects.toThrow(`Input file ${nonExistentFile} does not exist.`);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,96 @@
|
||||
const path = require("path");
|
||||
const { SUPPORTED_FILETYPE_CONVERTERS } = require("../../../utils/constants");
|
||||
const { mimeToExtension } = require("../../../utils/downloadURIToFile");
|
||||
|
||||
/**
|
||||
* Simulates the filename-building logic from downloadURIToFile
|
||||
* to verify extension inference works correctly.
|
||||
*/
|
||||
function buildFilenameWithExtension(sluggedFilename, contentType) {
|
||||
const existingExt = path.extname(sluggedFilename).toLowerCase();
|
||||
if (!SUPPORTED_FILETYPE_CONVERTERS.hasOwnProperty(existingExt)) {
|
||||
const mimeType = contentType?.toLowerCase()?.split(";")[0]?.trim();
|
||||
const inferredExt = mimeToExtension(mimeType);
|
||||
if (inferredExt) {
|
||||
return sluggedFilename + inferredExt;
|
||||
}
|
||||
}
|
||||
return sluggedFilename;
|
||||
}
|
||||
|
||||
describe("mimeToExtension", () => {
|
||||
test("returns null for invalid or unknown input", () => {
|
||||
expect(mimeToExtension(null)).toBeNull();
|
||||
expect(mimeToExtension(undefined)).toBeNull();
|
||||
expect(mimeToExtension("application/octet-stream")).toBeNull();
|
||||
});
|
||||
|
||||
test("returns first extension from ACCEPTED_MIMES for known types", () => {
|
||||
expect(mimeToExtension("application/pdf")).toBe(".pdf");
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildFilenameWithExtension", () => {
|
||||
test("appends .pdf when URL path has no recognized extension (arxiv case)", () => {
|
||||
// Simulates: https://arxiv.org/pdf/2307.10265
|
||||
// slugify produces something like "arxiv.org-pdf-230710265"
|
||||
const filename = "arxiv.org-pdf-230710265";
|
||||
const result = buildFilenameWithExtension(filename, "application/pdf");
|
||||
expect(result).toBe("arxiv.org-pdf-230710265.pdf");
|
||||
});
|
||||
|
||||
test("appends .pdf when URL has numeric-looking extension", () => {
|
||||
// path.extname("arxiv.org-pdf-2307.10265") => ".10265" which is not in SUPPORTED_FILETYPE_CONVERTERS
|
||||
const filename = "arxiv.org-pdf-2307.10265";
|
||||
const result = buildFilenameWithExtension(
|
||||
filename,
|
||||
"application/pdf; charset=utf-8"
|
||||
);
|
||||
expect(result).toBe("arxiv.org-pdf-2307.10265.pdf");
|
||||
});
|
||||
|
||||
test("does NOT append extension when file already has a supported extension", () => {
|
||||
const filename = "example.com-document.pdf";
|
||||
const result = buildFilenameWithExtension(filename, "application/pdf");
|
||||
expect(result).toBe("example.com-document.pdf");
|
||||
});
|
||||
|
||||
test("does NOT append extension when file has .txt extension", () => {
|
||||
const filename = "example.com-readme.txt";
|
||||
const result = buildFilenameWithExtension(filename, "text/plain");
|
||||
expect(result).toBe("example.com-readme.txt");
|
||||
});
|
||||
|
||||
test("does not append extension for unknown content type", () => {
|
||||
const filename = "example.com-binary-blob";
|
||||
const result = buildFilenameWithExtension(
|
||||
filename,
|
||||
"application/octet-stream"
|
||||
);
|
||||
expect(result).toBe("example.com-binary-blob");
|
||||
});
|
||||
|
||||
test("does not append extension when content type is null", () => {
|
||||
const filename = "example.com-unknown";
|
||||
const result = buildFilenameWithExtension(filename, null);
|
||||
expect(result).toBe("example.com-unknown");
|
||||
});
|
||||
|
||||
test("appends .docx for word document MIME type", () => {
|
||||
const filename = "sharepoint.com-documents-report";
|
||||
const result = buildFilenameWithExtension(
|
||||
filename,
|
||||
"application/vnd.openxmlformats-officedocument.wordprocessingml.document"
|
||||
);
|
||||
expect(result).toBe("sharepoint.com-documents-report.docx");
|
||||
});
|
||||
|
||||
test("handles content type with charset parameter correctly", () => {
|
||||
const filename = "api.example.com-export-data";
|
||||
const result = buildFilenameWithExtension(
|
||||
filename,
|
||||
"text/csv; charset=utf-8"
|
||||
);
|
||||
expect(result).toBe("api.example.com-export-data.csv");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,125 @@
|
||||
/* eslint-env jest, node */
|
||||
process.env.STORAGE_DIR = "test-storage";
|
||||
|
||||
const { resolveConfluenceBaseUrl } = require("../../../../utils/extensions/Confluence");
|
||||
const {
|
||||
ConfluencePagesLoader,
|
||||
} = require("../../../../utils/extensions/Confluence/ConfluenceLoader");
|
||||
|
||||
describe("resolveConfluenceBaseUrl", () => {
|
||||
test("cloud: strips path and returns origin only", () => {
|
||||
expect(
|
||||
resolveConfluenceBaseUrl("https://example.atlassian.net/wiki/spaces/SP", true)
|
||||
).toBe("https://example.atlassian.net");
|
||||
});
|
||||
|
||||
test("self-hosted: preserves context path, strips trailing slash", () => {
|
||||
expect(
|
||||
resolveConfluenceBaseUrl("https://my.domain.com/confluence/", false)
|
||||
).toBe("https://my.domain.com/confluence");
|
||||
});
|
||||
|
||||
test("self-hosted: returns origin when no context path", () => {
|
||||
expect(
|
||||
resolveConfluenceBaseUrl("https://my.domain.com/", false)
|
||||
).toBe("https://my.domain.com");
|
||||
});
|
||||
});
|
||||
|
||||
describe("ConfluencePagesLoader", () => {
|
||||
afterEach(() => {
|
||||
jest.restoreAllMocks();
|
||||
});
|
||||
|
||||
describe("cloud mode", () => {
|
||||
test("API requests include /wiki prefix", async () => {
|
||||
const fetchMock = jest.spyOn(global, "fetch").mockResolvedValue({
|
||||
ok: true,
|
||||
json: jest.fn().mockResolvedValue({ size: 0, results: [] }),
|
||||
});
|
||||
const loader = new ConfluencePagesLoader({
|
||||
baseUrl: resolveConfluenceBaseUrl("https://example.atlassian.net/wiki/spaces/SP", true),
|
||||
spaceKey: "SP",
|
||||
username: "user",
|
||||
accessToken: "token",
|
||||
cloud: true,
|
||||
});
|
||||
|
||||
await loader.fetchAllPagesInSpace();
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
"https://example.atlassian.net/wiki/rest/api/content?spaceKey=SP&limit=25&start=0&expand=body.storage,version",
|
||||
expect.any(Object)
|
||||
);
|
||||
});
|
||||
|
||||
test("page URLs include /wiki prefix", () => {
|
||||
const loader = new ConfluencePagesLoader({
|
||||
baseUrl: resolveConfluenceBaseUrl("https://example.atlassian.net/wiki", true),
|
||||
spaceKey: "SP",
|
||||
username: "user",
|
||||
accessToken: "token",
|
||||
cloud: true,
|
||||
});
|
||||
|
||||
const document = loader.createDocumentFromPage({
|
||||
id: "123",
|
||||
status: "current",
|
||||
title: "Cloud page",
|
||||
type: "page",
|
||||
body: { storage: { value: "<p>Hello</p>" } },
|
||||
version: { number: 1, by: { displayName: "User" }, when: "2026-01-01T00:00:00.000Z" },
|
||||
});
|
||||
|
||||
expect(document.metadata.url).toBe(
|
||||
"https://example.atlassian.net/wiki/spaces/SP/pages/123"
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("self-hosted mode", () => {
|
||||
test("API requests use context path without /wiki", async () => {
|
||||
const fetchMock = jest.spyOn(global, "fetch").mockResolvedValue({
|
||||
ok: true,
|
||||
json: jest.fn().mockResolvedValue({ size: 0, results: [] }),
|
||||
});
|
||||
const loader = new ConfluencePagesLoader({
|
||||
baseUrl: resolveConfluenceBaseUrl("https://my.domain.com/confluence/", false),
|
||||
spaceKey: "SP",
|
||||
username: "user",
|
||||
accessToken: "token",
|
||||
cloud: false,
|
||||
});
|
||||
|
||||
await loader.fetchAllPagesInSpace();
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
"https://my.domain.com/confluence/rest/api/content?spaceKey=SP&limit=25&start=0&expand=body.storage,version",
|
||||
expect.any(Object)
|
||||
);
|
||||
});
|
||||
|
||||
test("page URLs use context path without /wiki", () => {
|
||||
const loader = new ConfluencePagesLoader({
|
||||
baseUrl: resolveConfluenceBaseUrl("https://my.domain.com/confluence/", false),
|
||||
spaceKey: "SP",
|
||||
username: "user",
|
||||
accessToken: "token",
|
||||
cloud: false,
|
||||
});
|
||||
|
||||
const document = loader.createDocumentFromPage({
|
||||
id: "123",
|
||||
status: "current",
|
||||
title: "Self-hosted page",
|
||||
type: "page",
|
||||
body: { storage: { value: "<p>Hello</p>" } },
|
||||
version: { number: 1, by: { displayName: "User" }, when: "2026-01-01T00:00:00.000Z" },
|
||||
});
|
||||
|
||||
expect(document.metadata.url).toBe(
|
||||
"https://my.domain.com/confluence/spaces/SP/pages/123"
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,197 @@
|
||||
process.env.STORAGE_DIR = "test-storage"; // needed for tests to run
|
||||
const { validURL, validateURL, validYoutubeVideoUrl } = require("../../../utils/url");
|
||||
|
||||
// Mock the RuntimeSettings module
|
||||
jest.mock("../../../utils/runtimeSettings", () => {
|
||||
const mockInstance = {
|
||||
get: jest.fn(),
|
||||
set: jest.fn(),
|
||||
};
|
||||
return jest.fn().mockImplementation(() => mockInstance);
|
||||
});
|
||||
|
||||
describe("validURL", () => {
|
||||
let mockRuntimeSettings;
|
||||
|
||||
beforeEach(() => {
|
||||
const RuntimeSettings = require("../../../utils/runtimeSettings");
|
||||
mockRuntimeSettings = new RuntimeSettings();
|
||||
jest.clearAllMocks();
|
||||
});
|
||||
|
||||
it("should validate a valid URL", () => {
|
||||
mockRuntimeSettings.get.mockImplementation((key) => {
|
||||
if (key === "allowAnyIp") return false;
|
||||
if (key === "seenAnyIpWarning") return true; // silence the warning for tests
|
||||
return false;
|
||||
});
|
||||
|
||||
expect(validURL("https://www.google.com")).toBe(true);
|
||||
expect(validURL("http://www.google.com")).toBe(true);
|
||||
|
||||
// JS URL does not require extensions, so in theory
|
||||
// these should be valid
|
||||
expect(validURL("https://random")).toBe(true);
|
||||
expect(validURL("http://123")).toBe(true);
|
||||
|
||||
// missing protocols
|
||||
expect(validURL("www.google.com")).toBe(false);
|
||||
expect(validURL("google.com")).toBe(false);
|
||||
|
||||
// invalid protocols
|
||||
expect(validURL("ftp://www.google.com")).toBe(false);
|
||||
expect(validURL("mailto://www.google.com")).toBe(false);
|
||||
expect(validURL("tel://www.google.com")).toBe(false);
|
||||
expect(validURL("data://www.google.com")).toBe(false);
|
||||
});
|
||||
|
||||
it("should block private/local IPs when allowAnyIp is false (default behavior)", () => {
|
||||
mockRuntimeSettings.get.mockImplementation((key) => {
|
||||
if (key === "allowAnyIp") return false;
|
||||
if (key === "seenAnyIpWarning") return true; // silence the warning for tests
|
||||
return false;
|
||||
});
|
||||
|
||||
expect(validURL("http://192.168.1.1")).toBe(false);
|
||||
expect(validURL("http://10.0.0.1")).toBe(false);
|
||||
expect(validURL("http://172.16.0.1")).toBe(false);
|
||||
|
||||
// But localhost should still be allowed
|
||||
expect(validURL("http://127.0.0.1")).toBe(true);
|
||||
expect(validURL("http://0.0.0.0")).toBe(true);
|
||||
});
|
||||
|
||||
it("should allow any IP when allowAnyIp is true", () => {
|
||||
mockRuntimeSettings.get.mockImplementation((key) => {
|
||||
if (key === "allowAnyIp") return true;
|
||||
if (key === "seenAnyIpWarning") return true; // silence the warning for tests
|
||||
return false;
|
||||
});
|
||||
|
||||
expect(validURL("http://192.168.1.1")).toBe(true);
|
||||
expect(validURL("http://10.0.0.1")).toBe(true);
|
||||
expect(validURL("http://172.16.0.1")).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("validateURL", () => {
|
||||
it("should return the same URL if it's already valid", () => {
|
||||
expect(validateURL("https://www.google.com")).toBe(
|
||||
"https://www.google.com"
|
||||
);
|
||||
expect(validateURL("http://www.google.com")).toBe("http://www.google.com");
|
||||
expect(validateURL("https://random")).toBe("https://random");
|
||||
|
||||
// With numbers as a url this will turn into an ip
|
||||
expect(validateURL("123")).toBe("https://0.0.0.123");
|
||||
expect(validateURL("123.123.123.123")).toBe("https://123.123.123.123");
|
||||
expect(validateURL("http://127.0.123.45")).toBe("http://127.0.123.45");
|
||||
});
|
||||
|
||||
it("should assume https:// if the URL doesn't have a protocol", () => {
|
||||
expect(validateURL("www.google.com")).toBe("https://www.google.com");
|
||||
expect(validateURL("google.com")).toBe("https://google.com");
|
||||
expect(validateURL("EXAMPLE.com/ABCDEF/q1=UPPER")).toBe("https://example.com/ABCDEF/q1=UPPER");
|
||||
expect(validateURL("ftp://www.google.com")).toBe("ftp://www.google.com");
|
||||
expect(validateURL("mailto://www.google.com")).toBe(
|
||||
"mailto://www.google.com"
|
||||
);
|
||||
expect(validateURL("tel://www.google.com")).toBe("tel://www.google.com");
|
||||
expect(validateURL("data://www.google.com")).toBe("data://www.google.com");
|
||||
});
|
||||
|
||||
it("should remove trailing slashes post-validation", () => {
|
||||
expect(validateURL("https://www.google.com/")).toBe(
|
||||
"https://www.google.com"
|
||||
);
|
||||
expect(validateURL("http://www.google.com/")).toBe("http://www.google.com");
|
||||
expect(validateURL("https://random/")).toBe("https://random");
|
||||
expect(validateURL("https://example.com/ABCDEF/")).toBe("https://example.com/ABCDEF");
|
||||
});
|
||||
|
||||
it("should handle edge cases and bad data inputs", () => {
|
||||
expect(validateURL({})).toBe("");
|
||||
expect(validateURL(null)).toBe("");
|
||||
expect(validateURL(undefined)).toBe("");
|
||||
expect(validateURL(124512)).toBe("");
|
||||
expect(validateURL("")).toBe("");
|
||||
expect(validateURL(" ")).toBe("");
|
||||
expect(validateURL(" look here! ")).toBe("look here!");
|
||||
});
|
||||
|
||||
it("should preserve case of characters in URL pathname", () => {
|
||||
expect(validateURL("https://example.com/To/ResOURce?q1=Value&qZ22=UPPE!R"))
|
||||
.toBe("https://example.com/To/ResOURce?q1=Value&qZ22=UPPE!R");
|
||||
expect(validateURL("https://sample.com/uPeRCaSe"))
|
||||
.toBe("https://sample.com/uPeRCaSe");
|
||||
expect(validateURL("Example.com/PATH/To/Resource?q2=Value&q1=UPPER"))
|
||||
.toBe("https://example.com/PATH/To/Resource?q2=Value&q1=UPPER");
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
describe("validYoutubeVideoUrl", () => {
|
||||
const ID = "dQw4w9WgXcQ"; // 11-char valid video id
|
||||
|
||||
it("returns true for youtube watch URLs with v param", () => {
|
||||
expect(validYoutubeVideoUrl(`https://www.youtube.com/watch?v=${ID}`)).toBe(
|
||||
true
|
||||
);
|
||||
expect(validYoutubeVideoUrl(`https://youtube.com/watch?v=${ID}&t=10s`)).toBe(
|
||||
true
|
||||
);
|
||||
expect(validYoutubeVideoUrl(`https://m.youtube.com/watch?v=${ID}`)).toBe(true);
|
||||
expect(validYoutubeVideoUrl(`youtube.com/watch?v=${ID}`)).toBe(true);
|
||||
});
|
||||
|
||||
it("returns true for youtu.be short URLs", () => {
|
||||
expect(validYoutubeVideoUrl(`https://youtu.be/${ID}`)).toBe(true);
|
||||
expect(validYoutubeVideoUrl(`https://youtu.be/${ID}?si=abc`)).toBe(true);
|
||||
// extra path segments after id should still validate the id component
|
||||
expect(validYoutubeVideoUrl(`https://youtu.be/${ID}/extra`)).toBe(true);
|
||||
});
|
||||
|
||||
it("returns true for embed and v path formats", () => {
|
||||
expect(validYoutubeVideoUrl(`https://www.youtube.com/embed/${ID}`)).toBe(true);
|
||||
expect(validYoutubeVideoUrl(`https://youtube.com/v/${ID}`)).toBe(true);
|
||||
});
|
||||
|
||||
it("returns false for non-YouTube hosts", () => {
|
||||
expect(validYoutubeVideoUrl("https://example.com/watch?v=dQw4w9WgXcQ")).toBe(
|
||||
false
|
||||
);
|
||||
expect(validYoutubeVideoUrl("https://vimeo.com/123456")).toBe(false);
|
||||
});
|
||||
|
||||
it("returns false for unrelated YouTube paths without a video id", () => {
|
||||
expect(validYoutubeVideoUrl("https://www.youtube.com/user/somechannel")).toBe(
|
||||
false
|
||||
);
|
||||
expect(validYoutubeVideoUrl("https://www.youtube.com/")).toBe(false);
|
||||
});
|
||||
|
||||
it("returns false for empty or bad inputs", () => {
|
||||
expect(validYoutubeVideoUrl("")).toBe(false);
|
||||
expect(validYoutubeVideoUrl(null)).toBe(false);
|
||||
expect(validYoutubeVideoUrl(undefined)).toBe(false);
|
||||
});
|
||||
|
||||
it("returns the video ID for valid YouTube video URLs", () => {
|
||||
expect(validYoutubeVideoUrl(`https://www.youtube.com/watch?v=${ID}`, true)).toBe(ID);
|
||||
expect(validYoutubeVideoUrl(`https://youtube.com/watch?v=${ID}&t=10s`, true)).toBe(ID);
|
||||
expect(validYoutubeVideoUrl(`https://m.youtube.com/watch?v=${ID}`, true)).toBe(ID);
|
||||
expect(validYoutubeVideoUrl(`youtube.com/watch?v=${ID}`, true)).toBe(ID);
|
||||
expect(validYoutubeVideoUrl(`https://youtu.be/${ID}`, true)).toBe(ID);
|
||||
expect(validYoutubeVideoUrl(`https://youtu.be/${ID}?si=abc`, true)).toBe(ID);
|
||||
expect(validYoutubeVideoUrl(`https://youtu.be/${ID}/extra`, true)).toBe(ID);
|
||||
expect(validYoutubeVideoUrl(`https://www.youtube.com/embed/${ID}`, true)).toBe(ID);
|
||||
expect(validYoutubeVideoUrl(`https://youtube.com/v/${ID}`, true)).toBe(ID);
|
||||
// invalid video IDs
|
||||
expect(validYoutubeVideoUrl(`https://www.youtube.com/watch?v=invalid`, true)).toBe(null);
|
||||
expect(validYoutubeVideoUrl(`https://youtube.com/watch?v=invalid`, true)).toBe(null);
|
||||
expect(validYoutubeVideoUrl(`https://m.youtube.com/watch?v=invalid`, true)).toBe(null);
|
||||
expect(validYoutubeVideoUrl(`youtube.com/watch`, true)).toBe(null);
|
||||
expect(validYoutubeVideoUrl(`https://youtu.be/invalid`, true)).toBe(null);
|
||||
expect(validYoutubeVideoUrl(`https://youtu.be/invalid?si=abc`, true)).toBe(null);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,54 @@
|
||||
const fs = require("fs/promises");
|
||||
const path = require("path");
|
||||
const { WATCH_DIRECTORY } = require("../utils/constants");
|
||||
const { isWithin, trashFile, normalizePath } = require("../utils/files");
|
||||
const { FFMPEGWrapper } = require("../utils/WhisperProviders/ffmpeg");
|
||||
|
||||
/**
|
||||
* Convert an audio file in the hotdir to a 16kHz mono WAV in the same directory.
|
||||
* The source file is trashed; caller must read and trash the resulting wav.
|
||||
* @param {string} filename - The filename of the source audio in the hotdir
|
||||
* @returns {Promise<{success: boolean, reason: string, wavFilename: string|null}>}
|
||||
*/
|
||||
async function convertAudioToWav(filename) {
|
||||
if (!filename)
|
||||
return {
|
||||
success: false,
|
||||
reason: "No filename provided.",
|
||||
wavFilename: null,
|
||||
};
|
||||
|
||||
const inputPath = normalizePath(path.resolve(WATCH_DIRECTORY, filename));
|
||||
if (!isWithin(path.resolve(WATCH_DIRECTORY), inputPath))
|
||||
return {
|
||||
success: false,
|
||||
reason: "Filename is outside the hotdir.",
|
||||
wavFilename: null,
|
||||
};
|
||||
|
||||
try {
|
||||
await fs.access(inputPath);
|
||||
} catch {
|
||||
return {
|
||||
success: false,
|
||||
reason: `${filename} does not exist in hotdir.`,
|
||||
wavFilename: null,
|
||||
};
|
||||
}
|
||||
|
||||
const wavFilename = `${path.parse(filename).name}.wav`;
|
||||
const outputPath = path.resolve(WATCH_DIRECTORY, wavFilename);
|
||||
|
||||
try {
|
||||
const ffmpeg = new FFMPEGWrapper();
|
||||
await ffmpeg.convertAudioToWav(inputPath, outputPath);
|
||||
return { success: true, reason: null, wavFilename };
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
return { success: false, reason: e.message, wavFilename: null };
|
||||
} finally {
|
||||
trashFile(inputPath);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { convertAudioToWav };
|
||||
@@ -0,0 +1,38 @@
|
||||
import js from "@eslint/js";
|
||||
import globals from "globals";
|
||||
import { defineConfig } from "eslint/config";
|
||||
import pluginPrettier from "eslint-plugin-prettier";
|
||||
import configPrettier from "eslint-config-prettier";
|
||||
import unusedImports from "eslint-plugin-unused-imports";
|
||||
|
||||
export default defineConfig([
|
||||
{ ignores: ["__tests__/**"] },
|
||||
{
|
||||
files: ["**/*.{js,mjs,cjs}"],
|
||||
plugins: { js, prettier: pluginPrettier, "unused-imports": unusedImports },
|
||||
extends: ["js/recommended"],
|
||||
languageOptions: { globals: { ...globals.node, ...globals.browser } },
|
||||
rules: {
|
||||
...configPrettier.rules,
|
||||
"prettier/prettier": "error",
|
||||
"no-case-declarations": "off",
|
||||
"no-prototype-builtins": "off",
|
||||
"no-async-promise-executor": "off",
|
||||
"no-extra-boolean-cast": "off",
|
||||
"no-empty": "off",
|
||||
"no-unused-private-class-members": "warn",
|
||||
"no-unused-vars": "off",
|
||||
"unused-imports/no-unused-imports": "error",
|
||||
"unused-imports/no-unused-vars": [
|
||||
"error",
|
||||
{
|
||||
vars: "all",
|
||||
varsIgnorePattern: "^_",
|
||||
args: "after-used",
|
||||
argsIgnorePattern: "^_",
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
{ files: ["**/*.js"], languageOptions: { sourceType: "commonjs" } },
|
||||
]);
|
||||
@@ -0,0 +1,239 @@
|
||||
const { setDataSigner } = require("../middleware/setDataSigner");
|
||||
const { verifyPayloadIntegrity } = require("../middleware/verifyIntegrity");
|
||||
const {
|
||||
resolveRepoLoader,
|
||||
resolveRepoLoaderFunction,
|
||||
} = require("../utils/extensions/RepoLoader");
|
||||
const { reqBody } = require("../utils/http");
|
||||
const { validURL, validateURL } = require("../utils/url");
|
||||
const RESYNC_METHODS = require("./resync");
|
||||
const { loadObsidianVault } = require("../utils/extensions/ObsidianVault");
|
||||
|
||||
function extensions(app) {
|
||||
if (!app) return;
|
||||
|
||||
app.post(
|
||||
"/ext/resync-source-document",
|
||||
[verifyPayloadIntegrity, setDataSigner],
|
||||
async function (request, response) {
|
||||
try {
|
||||
const { type, options } = reqBody(request);
|
||||
if (!RESYNC_METHODS.hasOwnProperty(type))
|
||||
throw new Error(`Type "${type}" is not a valid type to sync.`);
|
||||
return await RESYNC_METHODS[type](options, response);
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(200).json({
|
||||
success: false,
|
||||
content: null,
|
||||
reason: e.message || "A processing error occurred.",
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/ext/:repo_platform-repo",
|
||||
[verifyPayloadIntegrity, setDataSigner],
|
||||
async function (request, response) {
|
||||
try {
|
||||
const loadRepo = resolveRepoLoaderFunction(
|
||||
request.params.repo_platform
|
||||
);
|
||||
const { success, reason, data } = await loadRepo(
|
||||
reqBody(request),
|
||||
response
|
||||
);
|
||||
response.status(200).json({
|
||||
success,
|
||||
reason,
|
||||
data,
|
||||
});
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(200).json({
|
||||
success: false,
|
||||
reason: e.message || "A processing error occurred.",
|
||||
data: {},
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
|
||||
// gets all branches for a specific repo
|
||||
app.post(
|
||||
"/ext/:repo_platform-repo/branches",
|
||||
[verifyPayloadIntegrity],
|
||||
async function (request, response) {
|
||||
try {
|
||||
const RepoLoader = resolveRepoLoader(request.params.repo_platform);
|
||||
const allBranches = await new RepoLoader(
|
||||
reqBody(request)
|
||||
).getRepoBranches();
|
||||
response.status(200).json({
|
||||
success: true,
|
||||
reason: null,
|
||||
data: {
|
||||
branches: allBranches,
|
||||
},
|
||||
});
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(400).json({
|
||||
success: false,
|
||||
reason: e.message,
|
||||
data: {
|
||||
branches: [],
|
||||
},
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/ext/youtube-transcript",
|
||||
[verifyPayloadIntegrity],
|
||||
async function (request, response) {
|
||||
try {
|
||||
const {
|
||||
loadYouTubeTranscript,
|
||||
} = require("../utils/extensions/YoutubeTranscript");
|
||||
const { success, reason, data } = await loadYouTubeTranscript(
|
||||
reqBody(request)
|
||||
);
|
||||
response.status(200).json({ success, reason, data });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(400).json({
|
||||
success: false,
|
||||
reason: e.message,
|
||||
data: {
|
||||
title: null,
|
||||
author: null,
|
||||
},
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/ext/website-depth",
|
||||
[verifyPayloadIntegrity],
|
||||
async function (request, response) {
|
||||
try {
|
||||
const websiteDepth = require("../utils/extensions/WebsiteDepth");
|
||||
const { url, depth = 1, maxLinks = 20 } = reqBody(request);
|
||||
const validatedUrl = validateURL(url);
|
||||
if (!validURL(validatedUrl)) throw new Error("Not a valid URL.");
|
||||
const scrapedData = await websiteDepth(validatedUrl, depth, maxLinks);
|
||||
response.status(200).json({ success: true, data: scrapedData });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(400).json({ success: false, reason: e.message });
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/ext/confluence",
|
||||
[verifyPayloadIntegrity, setDataSigner],
|
||||
async function (request, response) {
|
||||
try {
|
||||
const { loadConfluence } = require("../utils/extensions/Confluence");
|
||||
const { success, reason, data } = await loadConfluence(
|
||||
reqBody(request),
|
||||
response
|
||||
);
|
||||
response.status(200).json({ success, reason, data });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(400).json({
|
||||
success: false,
|
||||
reason: e.message,
|
||||
data: {
|
||||
title: null,
|
||||
author: null,
|
||||
},
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/ext/drupalwiki",
|
||||
[verifyPayloadIntegrity, setDataSigner],
|
||||
async function (request, response) {
|
||||
try {
|
||||
const {
|
||||
loadAndStoreSpaces,
|
||||
} = require("../utils/extensions/DrupalWiki");
|
||||
const { success, reason, data } = await loadAndStoreSpaces(
|
||||
reqBody(request),
|
||||
response
|
||||
);
|
||||
response.status(200).json({ success, reason, data });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(400).json({
|
||||
success: false,
|
||||
reason: e.message,
|
||||
data: {
|
||||
title: null,
|
||||
author: null,
|
||||
},
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/ext/obsidian/vault",
|
||||
[verifyPayloadIntegrity, setDataSigner],
|
||||
async function (request, response) {
|
||||
try {
|
||||
const { files } = reqBody(request);
|
||||
const result = await loadObsidianVault({ files });
|
||||
response.status(200).json(result);
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(400).json({
|
||||
success: false,
|
||||
reason: e.message,
|
||||
data: null,
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/ext/paperless-ngx",
|
||||
[verifyPayloadIntegrity, setDataSigner],
|
||||
async function (request, response) {
|
||||
try {
|
||||
const {
|
||||
loadPaperlessNgx,
|
||||
} = require("../utils/extensions/PaperlessNgx");
|
||||
const result = await loadPaperlessNgx(reqBody(request), response);
|
||||
response.status(200).json(result);
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(400).json({
|
||||
success: false,
|
||||
reason: e.message,
|
||||
data: null,
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
module.exports = extensions;
|
||||
@@ -0,0 +1,197 @@
|
||||
const { getLinkText } = require("../../processLink");
|
||||
|
||||
/**
|
||||
* Fetches the content of a raw link. Returns the content as a text string of the link in question.
|
||||
* @param {object} data - metadata from document (eg: link)
|
||||
* @param {import("../../middleware/setDataSigner").ResponseWithSigner} response
|
||||
*/
|
||||
async function resyncLink({ link }, response) {
|
||||
if (!link) throw new Error("Invalid link provided");
|
||||
try {
|
||||
const { success, content = null, reason } = await getLinkText(link);
|
||||
if (!success) throw new Error(`Failed to sync link content. ${reason}`);
|
||||
response.status(200).json({ success, content });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(200).json({
|
||||
success: false,
|
||||
content: null,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetches the content of a YouTube link. Returns the content as a text string of the video in question.
|
||||
* We offer this as there may be some videos where a transcription could be manually edited after initial scraping
|
||||
* but in general - transcriptions often never change.
|
||||
* @param {object} data - metadata from document (eg: link)
|
||||
* @param {import("../../middleware/setDataSigner").ResponseWithSigner} response
|
||||
*/
|
||||
async function resyncYouTube({ link }, response) {
|
||||
if (!link) throw new Error("Invalid link provided");
|
||||
try {
|
||||
const {
|
||||
fetchVideoTranscriptContent,
|
||||
} = require("../../utils/extensions/YoutubeTranscript");
|
||||
const { success, reason, content } = await fetchVideoTranscriptContent({
|
||||
url: link,
|
||||
});
|
||||
if (!success)
|
||||
throw new Error(`Failed to sync YouTube video transcript. ${reason}`);
|
||||
response.status(200).json({ success, content });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(200).json({
|
||||
success: false,
|
||||
content: null,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetches the content of a specific confluence page via its chunkSource.
|
||||
* Returns the content as a text string of the page in question and only that page.
|
||||
* @param {object} data - metadata from document (eg: chunkSource)
|
||||
* @param {import("../../middleware/setDataSigner").ResponseWithSigner} response
|
||||
*/
|
||||
async function resyncConfluence({ chunkSource }, response) {
|
||||
if (!chunkSource) throw new Error("Invalid source property provided");
|
||||
try {
|
||||
// Confluence data is `payload` encrypted. So we need to expand its
|
||||
// encrypted payload back into query params so we can reFetch the page with same access token/params.
|
||||
const source = response.locals.encryptionWorker.expandPayload(chunkSource);
|
||||
const {
|
||||
fetchConfluencePage,
|
||||
} = require("../../utils/extensions/Confluence");
|
||||
const { success, reason, content } = await fetchConfluencePage({
|
||||
pageUrl: `https:${source.pathname}`, // need to add back the real protocol
|
||||
baseUrl: source.searchParams.get("baseUrl"),
|
||||
spaceKey: source.searchParams.get("spaceKey"),
|
||||
accessToken: source.searchParams.get("token"),
|
||||
username: source.searchParams.get("username"),
|
||||
cloud: source.searchParams.get("cloud") === "true",
|
||||
bypassSSL: source.searchParams.get("bypassSSL") === "true",
|
||||
});
|
||||
|
||||
if (!success)
|
||||
throw new Error(`Failed to sync Confluence page content. ${reason}`);
|
||||
response.status(200).json({ success, content });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(200).json({
|
||||
success: false,
|
||||
content: null,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetches the content of a specific confluence page via its chunkSource.
|
||||
* Returns the content as a text string of the page in question and only that page.
|
||||
* @param {object} data - metadata from document (eg: chunkSource)
|
||||
* @param {import("../../middleware/setDataSigner").ResponseWithSigner} response
|
||||
*/
|
||||
async function resyncGithub({ chunkSource }, response) {
|
||||
if (!chunkSource) throw new Error("Invalid source property provided");
|
||||
try {
|
||||
// Github file data is `payload` encrypted (might contain PAT). So we need to expand its
|
||||
// encrypted payload back into query params so we can reFetch the page with same access token/params.
|
||||
const source = response.locals.encryptionWorker.expandPayload(chunkSource);
|
||||
const {
|
||||
fetchGithubFile,
|
||||
} = require("../../utils/extensions/RepoLoader/GithubRepo");
|
||||
const { success, reason, content } = await fetchGithubFile({
|
||||
repoUrl: `https:${source.pathname}`, // need to add back the real protocol
|
||||
branch: source.searchParams.get("branch"),
|
||||
accessToken: source.searchParams.get("pat"),
|
||||
sourceFilePath: source.searchParams.get("path"),
|
||||
});
|
||||
|
||||
if (!success)
|
||||
throw new Error(`Failed to sync GitHub file content. ${reason}`);
|
||||
response.status(200).json({ success, content });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(200).json({
|
||||
success: false,
|
||||
content: null,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetches the content of a specific DrupalWiki page via its chunkSource.
|
||||
* Returns the content as a text string of the page in question and only that page.
|
||||
* @param {object} data - metadata from document (eg: chunkSource)
|
||||
* @param {import("../../middleware/setDataSigner").ResponseWithSigner} response
|
||||
*/
|
||||
async function resyncDrupalWiki({ chunkSource }, response) {
|
||||
if (!chunkSource) throw new Error("Invalid source property provided");
|
||||
try {
|
||||
// DrupalWiki data is `payload` encrypted. So we need to expand its
|
||||
// encrypted payload back into query params so we can reFetch the page with same access token/params.
|
||||
const source = response.locals.encryptionWorker.expandPayload(chunkSource);
|
||||
const { loadPage } = require("../../utils/extensions/DrupalWiki");
|
||||
const { success, reason, content } = await loadPage({
|
||||
baseUrl: source.searchParams.get("baseUrl"),
|
||||
pageId: source.searchParams.get("pageId"),
|
||||
accessToken: source.searchParams.get("accessToken"),
|
||||
});
|
||||
|
||||
if (!success) {
|
||||
console.error(`Failed to sync DrupalWiki page content. ${reason}`);
|
||||
response.status(200).json({
|
||||
success: false,
|
||||
content: null,
|
||||
});
|
||||
} else {
|
||||
response.status(200).json({ success, content });
|
||||
}
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(200).json({
|
||||
success: false,
|
||||
content: null,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetches the content of a specific Paperless-ngx document via its chunkSource.
|
||||
* Returns the content as a text string of the document.
|
||||
* @param {object} data - metadata from document (eg: chunkSource)
|
||||
* @param {import("../../middleware/setDataSigner").ResponseWithSigner} response
|
||||
*/
|
||||
async function resyncPaperlessNgx({ chunkSource }, response) {
|
||||
if (!chunkSource) throw new Error("Invalid source property provided");
|
||||
try {
|
||||
const source = response.locals.encryptionWorker.expandPayload(chunkSource);
|
||||
const {
|
||||
PaperlessNgxLoader,
|
||||
} = require("../../utils/extensions/PaperlessNgx/PaperlessNgxLoader");
|
||||
const loader = new PaperlessNgxLoader({
|
||||
baseUrl: source.searchParams.get("baseUrl"),
|
||||
apiToken: source.searchParams.get("token"),
|
||||
});
|
||||
const documentId = source.pathname.split("//")[1];
|
||||
const content = await loader.fetchDocumentContent(documentId);
|
||||
|
||||
if (!content) throw new Error("Failed to fetch document content");
|
||||
response.status(200).json({ success: true, content });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(200).json({
|
||||
success: false,
|
||||
content: null,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
link: resyncLink,
|
||||
youtube: resyncYouTube,
|
||||
confluence: resyncConfluence,
|
||||
github: resyncGithub,
|
||||
drupalwiki: resyncDrupalWiki,
|
||||
"paperless-ngx": resyncPaperlessNgx,
|
||||
};
|
||||
@@ -0,0 +1,3 @@
|
||||
### What is the "Hot directory"
|
||||
|
||||
This is a pre-set file location that documents will be written to when uploaded by AnythingLLM. There is really no need to touch it.
|
||||
@@ -0,0 +1,228 @@
|
||||
process.env.NODE_ENV === "development"
|
||||
? require("dotenv").config({ path: `.env.${process.env.NODE_ENV}` })
|
||||
: require("dotenv").config();
|
||||
|
||||
require("./utils/logger")();
|
||||
const express = require("express");
|
||||
const bodyParser = require("body-parser");
|
||||
const cors = require("cors");
|
||||
const path = require("path");
|
||||
const { ACCEPTED_MIMES } = require("./utils/constants");
|
||||
const { reqBody, getCollectorPort } = require("./utils/http");
|
||||
const { processSingleFile } = require("./processSingleFile");
|
||||
const { processLink, getLinkText } = require("./processLink");
|
||||
const { wipeCollectorStorage } = require("./utils/files");
|
||||
const extensions = require("./extensions");
|
||||
const { processRawText } = require("./processRawText");
|
||||
const { convertAudioToWav } = require("./convertAudioToWav");
|
||||
const { verifyPayloadIntegrity } = require("./middleware/verifyIntegrity");
|
||||
const { httpLogger } = require("./middleware/httpLogger");
|
||||
const app = express();
|
||||
const FILE_LIMIT = "3GB";
|
||||
const COLLECTOR_PORT = getCollectorPort();
|
||||
|
||||
// Only log HTTP requests in development mode and if the ENABLE_HTTP_LOGGER environment variable is set to true
|
||||
if (
|
||||
process.env.NODE_ENV === "development" &&
|
||||
!!process.env.ENABLE_HTTP_LOGGER
|
||||
) {
|
||||
app.use(
|
||||
httpLogger({
|
||||
enableTimestamps: !!process.env.ENABLE_HTTP_LOGGER_TIMESTAMPS,
|
||||
})
|
||||
);
|
||||
}
|
||||
app.use(cors({ origin: true }));
|
||||
app.use(
|
||||
bodyParser.text({ limit: FILE_LIMIT }),
|
||||
bodyParser.json({ limit: FILE_LIMIT }),
|
||||
bodyParser.urlencoded({
|
||||
limit: FILE_LIMIT,
|
||||
extended: true,
|
||||
})
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/process",
|
||||
[verifyPayloadIntegrity],
|
||||
async function (request, response) {
|
||||
const { filename, options = {}, metadata = {} } = reqBody(request);
|
||||
try {
|
||||
const targetFilename = path
|
||||
.normalize(filename)
|
||||
.replace(/^(\.\.(\/|\\|$))+/, "");
|
||||
const {
|
||||
success,
|
||||
reason,
|
||||
documents = [],
|
||||
} = await processSingleFile(targetFilename, options, metadata);
|
||||
response
|
||||
.status(200)
|
||||
.json({ filename: targetFilename, success, reason, documents });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(200).json({
|
||||
filename: filename,
|
||||
success: false,
|
||||
reason: "A processing error occurred.",
|
||||
documents: [],
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/parse",
|
||||
[verifyPayloadIntegrity],
|
||||
async function (request, response) {
|
||||
const { filename, options = {} } = reqBody(request);
|
||||
try {
|
||||
const targetFilename = path
|
||||
.normalize(filename)
|
||||
.replace(/^(\.\.(\/|\\|$))+/, "");
|
||||
const {
|
||||
success,
|
||||
reason,
|
||||
documents = [],
|
||||
} = await processSingleFile(targetFilename, {
|
||||
...options,
|
||||
parseOnly: true,
|
||||
absolutePath: options.absolutePath || null,
|
||||
});
|
||||
response
|
||||
.status(200)
|
||||
.json({ filename: targetFilename, success, reason, documents });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(200).json({
|
||||
filename: filename,
|
||||
success: false,
|
||||
reason: "A processing error occurred.",
|
||||
documents: [],
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/process-link",
|
||||
[verifyPayloadIntegrity],
|
||||
async function (request, response) {
|
||||
const { link, scraperHeaders = {}, metadata = {} } = reqBody(request);
|
||||
try {
|
||||
const {
|
||||
success,
|
||||
reason,
|
||||
documents = [],
|
||||
} = await processLink(link, scraperHeaders, metadata);
|
||||
response.status(200).json({ url: link, success, reason, documents });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(200).json({
|
||||
url: link,
|
||||
success: false,
|
||||
reason: "A processing error occurred.",
|
||||
documents: [],
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/util/get-link",
|
||||
[verifyPayloadIntegrity],
|
||||
async function (request, response) {
|
||||
const { link, captureAs = "text" } = reqBody(request);
|
||||
try {
|
||||
const { success, content = null } = await getLinkText(link, captureAs);
|
||||
response.status(200).json({ url: link, success, content });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(200).json({
|
||||
url: link,
|
||||
success: false,
|
||||
content: null,
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/util/convert-audio-to-wav",
|
||||
[verifyPayloadIntegrity],
|
||||
async function (request, response) {
|
||||
const { filename } = reqBody(request);
|
||||
try {
|
||||
const {
|
||||
success,
|
||||
reason,
|
||||
wavFilename = null,
|
||||
} = await convertAudioToWav(filename);
|
||||
response.status(200).json({ filename, success, reason, wavFilename });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(200).json({
|
||||
filename,
|
||||
success: false,
|
||||
reason: "An audio conversion error occurred.",
|
||||
wavFilename: null,
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
|
||||
app.post(
|
||||
"/process-raw-text",
|
||||
[verifyPayloadIntegrity],
|
||||
async function (request, response) {
|
||||
const { textContent, metadata } = reqBody(request);
|
||||
try {
|
||||
const {
|
||||
success,
|
||||
reason,
|
||||
documents = [],
|
||||
} = await processRawText(textContent, metadata);
|
||||
response
|
||||
.status(200)
|
||||
.json({ filename: metadata.title, success, reason, documents });
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
response.status(200).json({
|
||||
filename: metadata?.title || "Unknown-doc.txt",
|
||||
success: false,
|
||||
reason: "A processing error occurred.",
|
||||
documents: [],
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
);
|
||||
|
||||
extensions(app);
|
||||
|
||||
app.get("/accepts", function (_, response) {
|
||||
response.status(200).json(ACCEPTED_MIMES);
|
||||
});
|
||||
|
||||
app.all("*", function (_, response) {
|
||||
response.sendStatus(200);
|
||||
});
|
||||
|
||||
app
|
||||
.listen(COLLECTOR_PORT, async () => {
|
||||
await wipeCollectorStorage();
|
||||
console.log(`Document processor app listening on port ${COLLECTOR_PORT}`);
|
||||
})
|
||||
.on("error", function (_) {
|
||||
process.once("SIGUSR2", function () {
|
||||
process.kill(process.pid, "SIGUSR2");
|
||||
});
|
||||
process.on("SIGINT", function () {
|
||||
process.kill(process.pid, "SIGINT");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,29 @@
|
||||
const httpLogger =
|
||||
({ enableTimestamps = false }) =>
|
||||
(req, res, next) => {
|
||||
// Capture the original res.end to log response status
|
||||
const originalEnd = res.end;
|
||||
|
||||
res.end = function (chunk, encoding) {
|
||||
// Log the request method, status code, and path
|
||||
const statusColor = res.statusCode >= 400 ? "\x1b[31m" : "\x1b[32m"; // Red for errors, green for success
|
||||
console.log(
|
||||
`\x1b[32m[HTTP]\x1b[0m ${statusColor}${res.statusCode}\x1b[0m ${
|
||||
req.method
|
||||
} -> ${req.path} ${
|
||||
enableTimestamps
|
||||
? `@ ${new Date().toLocaleTimeString("en-US", { hour12: true })}`
|
||||
: ""
|
||||
}`.trim()
|
||||
);
|
||||
|
||||
// Call the original end method
|
||||
return originalEnd.call(this, chunk, encoding);
|
||||
};
|
||||
|
||||
next();
|
||||
};
|
||||
|
||||
module.exports = {
|
||||
httpLogger,
|
||||
};
|
||||
@@ -0,0 +1,44 @@
|
||||
const { EncryptionWorker } = require("../utils/EncryptionWorker");
|
||||
const { CommunicationKey } = require("../utils/comKey");
|
||||
|
||||
/**
|
||||
* Express Response Object interface with defined encryptionWorker attached to locals property.
|
||||
* @typedef {import("express").Response & import("express").Response['locals'] & {encryptionWorker: EncryptionWorker} } ResponseWithSigner
|
||||
*/
|
||||
|
||||
// You can use this middleware to assign the EncryptionWorker to the response locals
|
||||
// property so that if can be used to encrypt/decrypt arbitrary data via response object.
|
||||
// eg: Encrypting API keys in chunk sources.
|
||||
|
||||
// The way this functions is that the rolling RSA Communication Key is used server-side to private-key encrypt the raw
|
||||
// key of the persistent EncryptionManager credentials. Since EncryptionManager credentials do _not_ roll, we should not send them
|
||||
// even between server<>collector in plaintext because if the user configured the server/collector to be public they could technically
|
||||
// be exposing the key in transit via the X-Payload-Signer header. Even if this risk is minimal we should not do this.
|
||||
|
||||
// This middleware uses the CommunicationKey public key to first decrypt the base64 representation of the EncryptionManager credentials
|
||||
// and then loads that in to the EncryptionWorker as a buffer so we can use the same credentials across the system. Should we ever break the
|
||||
// collector out into its own service this would still work without SSL/TLS.
|
||||
|
||||
/**
|
||||
*
|
||||
* @param {import("express").Request} request
|
||||
* @param {import("express").Response} response
|
||||
* @param {import("express").NextFunction} next
|
||||
*/
|
||||
function setDataSigner(request, response, next) {
|
||||
const comKey = new CommunicationKey();
|
||||
const encryptedPayloadSigner = request.header("X-Payload-Signer");
|
||||
if (!encryptedPayloadSigner)
|
||||
console.log(
|
||||
"Failed to find signed-payload to set encryption worker! Encryption calls will fail."
|
||||
);
|
||||
|
||||
const decryptedPayloadSignerKey = comKey.decrypt(encryptedPayloadSigner);
|
||||
const encryptionWorker = new EncryptionWorker(decryptedPayloadSignerKey);
|
||||
response.locals.encryptionWorker = encryptionWorker;
|
||||
next();
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
setDataSigner,
|
||||
};
|
||||
@@ -0,0 +1,32 @@
|
||||
const { CommunicationKey } = require("../utils/comKey");
|
||||
const RuntimeSettings = require("../utils/runtimeSettings");
|
||||
const runtimeSettings = new RuntimeSettings();
|
||||
|
||||
function verifyPayloadIntegrity(request, response, next) {
|
||||
const comKey = new CommunicationKey();
|
||||
if (process.env.NODE_ENV === "development") {
|
||||
comKey.log("verifyPayloadIntegrity is skipped in development.");
|
||||
runtimeSettings.parseOptionsFromRequest(request);
|
||||
next();
|
||||
return;
|
||||
}
|
||||
|
||||
const signature = request.header("X-Integrity");
|
||||
if (!signature)
|
||||
return response
|
||||
.status(400)
|
||||
.json({ msg: "Failed integrity signature check." });
|
||||
|
||||
const validSignedPayload = comKey.verify(signature, request.body);
|
||||
if (!validSignedPayload)
|
||||
return response
|
||||
.status(400)
|
||||
.json({ msg: "Failed integrity signature check." });
|
||||
|
||||
runtimeSettings.parseOptionsFromRequest(request);
|
||||
next();
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
verifyPayloadIntegrity,
|
||||
};
|
||||
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"events": {}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
{
|
||||
"name": "anything-llm-document-collector",
|
||||
"version": "1.15.0",
|
||||
"description": "Document collector server endpoints",
|
||||
"main": "index.js",
|
||||
"author": "Timothy Carambat (Mintplex Labs)",
|
||||
"license": "MIT",
|
||||
"private": false,
|
||||
"engines": {
|
||||
"node": ">=18.12.1"
|
||||
},
|
||||
"scripts": {
|
||||
"dev": "cross-env NODE_ENV=development nodemon --ignore hotdir --ignore storage --trace-warnings index.js",
|
||||
"start": "cross-env NODE_ENV=production node index.js",
|
||||
"lint": "eslint --fix .",
|
||||
"lint:check": "eslint ."
|
||||
},
|
||||
"dependencies": {
|
||||
"@langchain/community": "^0.2.23",
|
||||
"@xenova/transformers": "^2.14.0",
|
||||
"body-parser": "^1.20.3",
|
||||
"cors": "^2.8.5",
|
||||
"dotenv": "^16.0.3",
|
||||
"epub2": "git+https://github.com/Mintplex-Labs/epub2-static.git#main",
|
||||
"express": "^4.21.2",
|
||||
"fix-path": "^4.0.0",
|
||||
"html-to-text": "^9.0.5",
|
||||
"ignore": "^5.3.0",
|
||||
"js-tiktoken": "^1.0.8",
|
||||
"langchain": "0.1.36",
|
||||
"mammoth": "^1.6.0",
|
||||
"mbox-parser": "^1.0.1",
|
||||
"mime": "^3.0.0",
|
||||
"moment": "^2.29.4",
|
||||
"node-html-parser": "^6.1.13",
|
||||
"node-xlsx": "^0.24.0",
|
||||
"officeparser": "^4.0.5",
|
||||
"openai": "4.95.1",
|
||||
"pdf-parse": "^1.1.1",
|
||||
"puppeteer": "~21.5.2",
|
||||
"sharp": "^0.33.5",
|
||||
"slugify": "^1.6.6",
|
||||
"strip-ansi": "^7.1.2",
|
||||
"tesseract.js": "^6.0.0",
|
||||
"turndown": "7.2.4",
|
||||
"url-pattern": "^1.0.3",
|
||||
"uuid": "^9.0.0",
|
||||
"wavefile": "^11.0.0",
|
||||
"winston": "^3.13.0",
|
||||
"youtube-transcript-plus": "^1.1.2",
|
||||
"youtubei.js": "^9.1.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@eslint/js": "^9.0.0",
|
||||
"cross-env": "^7.0.3",
|
||||
"eslint": "^9.0.0",
|
||||
"eslint-config-prettier": "^9.0.0",
|
||||
"eslint-plugin-prettier": "^5.0.0",
|
||||
"eslint-plugin-unused-imports": "^4.0.0",
|
||||
"globals": "^17.4.0",
|
||||
"nodemon": "^2.0.22",
|
||||
"prettier": "^2.4.1"
|
||||
},
|
||||
"resolutions": {
|
||||
"string-width": "^4.2.3",
|
||||
"wrap-ansi": "^7.0.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,240 @@
|
||||
const { v4 } = require("uuid");
|
||||
const {
|
||||
PuppeteerWebBaseLoader,
|
||||
} = require("langchain/document_loaders/web/puppeteer");
|
||||
const { writeToServerDocuments } = require("../../utils/files");
|
||||
const { tokenizeString } = require("../../utils/tokenizer");
|
||||
const { default: slugify } = require("slugify");
|
||||
const {
|
||||
returnResult,
|
||||
determineContentType,
|
||||
processAsFile,
|
||||
} = require("../helpers");
|
||||
const {
|
||||
loadYouTubeTranscript,
|
||||
} = require("../../utils/extensions/YoutubeTranscript");
|
||||
const RuntimeSettings = require("../../utils/runtimeSettings");
|
||||
const { htmlToMarkdown } = require("../helpers/htmlToMarkdown");
|
||||
|
||||
/**
|
||||
* Scrape a generic URL and return the content in the specified format
|
||||
* @param {Object} config - The configuration object
|
||||
* @param {string} config.link - The URL to scrape
|
||||
* @param {('html' | 'text')} config.captureAs - The format to capture the page content as. Default is 'text'
|
||||
* @param {{[key: string]: string}} config.scraperHeaders - Custom headers to use when making the request
|
||||
* @param {{[key: string]: string}} config.metadata - Metadata to use when creating the document
|
||||
* @param {boolean} config.saveAsDocument - Whether to save the content as a document. Default is true
|
||||
* @returns {Promise<Object>} - The content of the page
|
||||
*/
|
||||
async function scrapeGenericUrl({
|
||||
link,
|
||||
captureAs = "text",
|
||||
scraperHeaders = {},
|
||||
metadata = {},
|
||||
saveAsDocument = true,
|
||||
}) {
|
||||
/** @type {'web' | 'file' | 'youtube'} */
|
||||
console.log(`-- Working URL ${link} => (captureAs: ${captureAs}) --`);
|
||||
let { contentType, processVia } = await determineContentType(link);
|
||||
console.log(`-- URL determined to be ${contentType} (${processVia}) --`);
|
||||
|
||||
/**
|
||||
* When the content is a file or a YouTube video, we can use the existing processing functions
|
||||
* These are self-contained and will return the correct response based on the saveAsDocument flag already
|
||||
* so we can return the content immediately.
|
||||
*/
|
||||
if (processVia === "file")
|
||||
return await processAsFile({ uri: link, saveAsDocument });
|
||||
else if (processVia === "youtube")
|
||||
return await loadYouTubeTranscript(
|
||||
{ url: link },
|
||||
{ parseOnly: saveAsDocument === false }
|
||||
);
|
||||
|
||||
// Otherwise, assume the content is a webpage and scrape the content from the webpage
|
||||
const content = await getPageContent({
|
||||
link,
|
||||
captureAs,
|
||||
headers: scraperHeaders,
|
||||
});
|
||||
if (!content || !content.length) {
|
||||
console.error(`Resulting URL content was empty at ${link}.`);
|
||||
return returnResult({
|
||||
success: false,
|
||||
reason: `No URL content found at ${link}.`,
|
||||
documents: [],
|
||||
content: null,
|
||||
saveAsDocument,
|
||||
});
|
||||
}
|
||||
|
||||
// If the captureAs is text, return the content as a string immediately
|
||||
// so that we dont save the content as a document
|
||||
if (!saveAsDocument)
|
||||
return returnResult({
|
||||
success: true,
|
||||
content,
|
||||
saveAsDocument,
|
||||
});
|
||||
|
||||
// Save the content as a document from the URL
|
||||
const url = new URL(link);
|
||||
const decodedPathname = decodeURIComponent(url.pathname);
|
||||
const filename = `${url.hostname}${decodedPathname.replace(/\//g, "_")}`;
|
||||
const data = {
|
||||
id: v4(),
|
||||
url: "file://" + slugify(filename) + ".html",
|
||||
title: metadata.title || slugify(filename) + ".html",
|
||||
docAuthor: metadata.docAuthor || "no author found",
|
||||
description: metadata.description || "No description found.",
|
||||
docSource: metadata.docSource || "URL link uploaded by the user.",
|
||||
chunkSource: `link://${link}`,
|
||||
published: new Date().toLocaleString(),
|
||||
wordCount: content.split(" ").length,
|
||||
pageContent: content,
|
||||
token_count_estimate: tokenizeString(content),
|
||||
};
|
||||
|
||||
const document = writeToServerDocuments({
|
||||
data,
|
||||
filename: `url-${slugify(filename)}-${data.id}`,
|
||||
});
|
||||
console.log(`[SUCCESS]: URL ${link} converted & ready for embedding.\n`);
|
||||
return { success: true, reason: null, documents: [document] };
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate the headers object
|
||||
* - Keys & Values must be strings and not empty
|
||||
* - Assemble a new object with only the valid keys and values
|
||||
* @param {{[key: string]: string}} headers - The headers object to validate
|
||||
* @returns {{[key: string]: string}} - The validated headers object
|
||||
*/
|
||||
function validatedHeaders(headers = {}) {
|
||||
try {
|
||||
if (Object.keys(headers).length === 0) return {};
|
||||
let validHeaders = {};
|
||||
for (const key of Object.keys(headers)) {
|
||||
if (!key?.trim()) continue;
|
||||
if (typeof headers[key] !== "string" || !headers[key]?.trim()) continue;
|
||||
validHeaders[key] = headers[key].trim();
|
||||
}
|
||||
return validHeaders;
|
||||
} catch (error) {
|
||||
console.error("Error validating headers", error);
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the content of a page
|
||||
* @param {Object} config - The configuration object
|
||||
* @param {string} config.link - The URL to get the content of
|
||||
* @param {('html' | 'text')} config.captureAs - The format to capture the page content as. Default is 'text'
|
||||
* @param {{[key: string]: string}} config.headers - Custom headers to use when making the request
|
||||
* @returns {Promise<string>} - The content of the page
|
||||
*/
|
||||
async function getPageContent({ link, captureAs = "text", headers = {} }) {
|
||||
try {
|
||||
let pageContents = [];
|
||||
const runtimeSettings = new RuntimeSettings();
|
||||
|
||||
/** @type {import('puppeteer').PuppeteerLaunchOptions} */
|
||||
let launchConfig = { headless: "new" };
|
||||
|
||||
/* On MacOS 15.1, the headless=new option causes the browser to crash immediately.
|
||||
* It is not clear why this is the case, but it is reproducible. Since AnythinglLM
|
||||
* in production runs in a container, we can disable headless mode to workaround the issue for development purposes.
|
||||
*
|
||||
* This may show a popup window when scraping a page in development mode.
|
||||
* This is expected behavior if seen in development mode on MacOS 15+
|
||||
*/
|
||||
if (
|
||||
process.platform === "darwin" &&
|
||||
process.env.NODE_ENV === "development"
|
||||
) {
|
||||
console.log(
|
||||
"Darwin Development Mode: Disabling headless mode to prevent Chromium from crashing."
|
||||
);
|
||||
launchConfig.headless = "false";
|
||||
}
|
||||
|
||||
const loader = new PuppeteerWebBaseLoader(link, {
|
||||
launchOptions: {
|
||||
headless: launchConfig.headless,
|
||||
ignoreHTTPSErrors: true,
|
||||
args: runtimeSettings.get("browserLaunchArgs"),
|
||||
},
|
||||
gotoOptions: {
|
||||
waitUntil: "networkidle2",
|
||||
},
|
||||
async evaluate(page, browser) {
|
||||
const innerHTML = await page.evaluate(
|
||||
() => document.documentElement.innerHTML
|
||||
);
|
||||
await browser.close();
|
||||
if (captureAs === "html") return innerHTML;
|
||||
return htmlToMarkdown(innerHTML, link);
|
||||
},
|
||||
});
|
||||
|
||||
// Override scrape method if headers are available
|
||||
let overrideHeaders = validatedHeaders(headers);
|
||||
if (Object.keys(overrideHeaders).length > 0) {
|
||||
loader.scrape = async function () {
|
||||
const { launch } = await PuppeteerWebBaseLoader.imports();
|
||||
const browser = await launch({
|
||||
headless: "new",
|
||||
defaultViewport: null,
|
||||
ignoreDefaultArgs: ["--disable-extensions"],
|
||||
...this.options?.launchOptions,
|
||||
});
|
||||
const page = await browser.newPage();
|
||||
await page.setExtraHTTPHeaders(overrideHeaders);
|
||||
|
||||
await page.goto(this.webPath, {
|
||||
timeout: 180000,
|
||||
waitUntil: "networkidle2",
|
||||
...this.options?.gotoOptions,
|
||||
});
|
||||
|
||||
const bodyHTML = this.options?.evaluate
|
||||
? await this.options.evaluate(page, browser)
|
||||
: await page.evaluate(() => document.body.innerHTML);
|
||||
|
||||
await browser.close();
|
||||
return bodyHTML;
|
||||
};
|
||||
}
|
||||
|
||||
const docs = await loader.load();
|
||||
for (const doc of docs) pageContents.push(doc.pageContent);
|
||||
return pageContents.join(" ");
|
||||
} catch (error) {
|
||||
console.error(
|
||||
"getPageContent failed to be fetched by puppeteer - falling back to fetch!",
|
||||
error
|
||||
);
|
||||
}
|
||||
|
||||
try {
|
||||
const pageText = await fetch(link, {
|
||||
method: "GET",
|
||||
headers: {
|
||||
"Content-Type": "text/plain",
|
||||
"User-Agent":
|
||||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36,gzip(gfe)",
|
||||
...validatedHeaders(headers),
|
||||
},
|
||||
}).then((res) => res.text());
|
||||
return htmlToMarkdown(pageText, link);
|
||||
} catch (error) {
|
||||
console.error("getPageContent failed to be fetched by any method.", error);
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
scrapeGenericUrl,
|
||||
};
|
||||
@@ -0,0 +1,203 @@
|
||||
const TurndownService = require("turndown");
|
||||
const { parse } = require("node-html-parser");
|
||||
|
||||
// Image URL base paths to strip entirely - these are typically proxied badges,
|
||||
// tracking pixels, avatars, or other non-content images that bloat token counts.
|
||||
const IGNORED_IMG_BASEPATHS = [
|
||||
"https://camo.githubusercontent.com",
|
||||
"https://avatars.githubusercontent.com",
|
||||
"https://img.shields.io",
|
||||
"https://badge.fury.io",
|
||||
"https://badges.gitter.im",
|
||||
"https://coveralls.io/repos",
|
||||
"https://travis-ci.org",
|
||||
"https://circleci.com",
|
||||
"https://github.com/favicon",
|
||||
];
|
||||
|
||||
/**
|
||||
* Convert raw page HTML into clean markdown using Turndown.
|
||||
* Strips non-content elements (nav, footer, ads, etc.), hidden elements,
|
||||
* base64 images, scripts, styles, and resolves relative URLs before converting.
|
||||
* @param {string} html - Raw HTML string
|
||||
* @param {string} baseUrl - The URL the page was scraped from
|
||||
* @returns {string} Cleaned markdown string
|
||||
*/
|
||||
function htmlToMarkdown(html, baseUrl) {
|
||||
if (!html || typeof html !== "string") return "";
|
||||
try {
|
||||
let root = parse(html);
|
||||
|
||||
// Prefer the narrowest content container to avoid page chrome.
|
||||
// article > main because sites like GitHub put the file tree inside <main>
|
||||
// but scope the README to <article>.
|
||||
const content =
|
||||
root.querySelector("article") ||
|
||||
root.querySelector("main") ||
|
||||
root.querySelector('[role="main"]');
|
||||
if (content) root = content;
|
||||
|
||||
const junkSelectors = [
|
||||
"script",
|
||||
"style",
|
||||
"noscript",
|
||||
"nav",
|
||||
"footer",
|
||||
"header",
|
||||
"aside",
|
||||
"iframe",
|
||||
"svg",
|
||||
'[role="navigation"]',
|
||||
'[role="banner"]',
|
||||
'[role="contentinfo"]',
|
||||
'[aria-hidden="true"]',
|
||||
"[hidden]",
|
||||
];
|
||||
for (const sel of junkSelectors) {
|
||||
root.querySelectorAll(sel).forEach((el) => el.remove());
|
||||
}
|
||||
|
||||
for (const el of root.querySelectorAll("[style]")) {
|
||||
const style = el.getAttribute("style") || "";
|
||||
if (
|
||||
/display\s*:\s*none/i.test(style) ||
|
||||
/visibility\s*:\s*hidden/i.test(style)
|
||||
) {
|
||||
el.remove();
|
||||
}
|
||||
}
|
||||
|
||||
if (baseUrl) {
|
||||
resolveUrls(root, baseUrl);
|
||||
}
|
||||
|
||||
stripCitations(root);
|
||||
|
||||
const cleanedHtml = root.toString();
|
||||
|
||||
const turndown = new TurndownService({
|
||||
headingStyle: "atx",
|
||||
codeBlockStyle: "fenced",
|
||||
bulletListMarker: "-",
|
||||
});
|
||||
turndown.remove(["script", "style", "noscript", "iframe", "svg"]);
|
||||
|
||||
turndown.addRule("compactLinks", {
|
||||
filter: "a",
|
||||
replacement: function (content, node) {
|
||||
const href = node.getAttribute("href");
|
||||
if (!href) return content;
|
||||
const text = content.replace(/\s+/g, " ").trim();
|
||||
if (!text) return "";
|
||||
return `[${text}](${href})`;
|
||||
},
|
||||
});
|
||||
|
||||
let markdown = turndown.turndown(cleanedHtml);
|
||||
|
||||
// Strip links and images with excessively long URLs (200+ chars)
|
||||
markdown = markdown.replace(/!\[[^\]]*\]\([^)]{200,}\)\s*/g, "");
|
||||
markdown = markdown.replace(/\[[^\]]*\]\([^)]{200,}\)/g, (match) => {
|
||||
const textMatch = match.match(/\[([^\]]*)\]/);
|
||||
return textMatch ? textMatch[1] : "";
|
||||
});
|
||||
|
||||
markdown = markdown.replace(/\[[\d]+\]/g, "");
|
||||
markdown = markdown.replace(/\[#cite[^\]]*\]/g, "");
|
||||
markdown = markdown.replace(/\[edit\]/gi, "");
|
||||
|
||||
markdown = markdown.replace(/\n{4,}/g, "\n\n\n").trim();
|
||||
return markdown;
|
||||
} catch (error) {
|
||||
console.error("htmlToMarkdown failed:", error);
|
||||
try {
|
||||
return parse(html).text.trim();
|
||||
} catch {
|
||||
return "";
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve relative URLs for links and images. Strips base64 and
|
||||
* ignored-basepath images entirely. Gives alt-less images a filename-based alt.
|
||||
* @param {import('node-html-parser').HTMLElement} root
|
||||
* @param {string} baseUrl
|
||||
*/
|
||||
function resolveUrls(root, baseUrl) {
|
||||
for (const a of root.querySelectorAll("a[href]")) {
|
||||
const href = a.getAttribute("href");
|
||||
if (!href || /^(https?:|mailto:|tel:|javascript:|#)/i.test(href)) continue;
|
||||
try {
|
||||
a.setAttribute("href", new URL(href, baseUrl).toString());
|
||||
} catch {}
|
||||
}
|
||||
|
||||
for (const img of root.querySelectorAll("img[src]")) {
|
||||
const src = img.getAttribute("src");
|
||||
if (!src) {
|
||||
img.remove();
|
||||
continue;
|
||||
}
|
||||
if (src.startsWith("data:")) {
|
||||
img.remove();
|
||||
continue;
|
||||
}
|
||||
|
||||
const resolvedSrc = /^https?:/i.test(src)
|
||||
? src
|
||||
: (() => {
|
||||
try {
|
||||
return new URL(src, baseUrl).toString();
|
||||
} catch {
|
||||
return src;
|
||||
}
|
||||
})();
|
||||
|
||||
if (IGNORED_IMG_BASEPATHS.some((base) => resolvedSrc.startsWith(base))) {
|
||||
img.remove();
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!/^https?:/i.test(src)) {
|
||||
try {
|
||||
img.setAttribute("src", new URL(src, baseUrl).toString());
|
||||
} catch {}
|
||||
}
|
||||
|
||||
const alt = (img.getAttribute("alt") || "").trim();
|
||||
if (!alt) {
|
||||
try {
|
||||
const pathname = new URL(img.getAttribute("src")).pathname;
|
||||
const filename = pathname.split("/").pop() || "image";
|
||||
img.setAttribute("alt", filename);
|
||||
} catch {
|
||||
img.setAttribute("alt", "image");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Strip citation/reference superscripts and reference sections
|
||||
* commonly found on Wikipedia and similar sites.
|
||||
* @param {import('node-html-parser').HTMLElement} root
|
||||
*/
|
||||
function stripCitations(root) {
|
||||
for (const sup of root.querySelectorAll("sup.reference, sup.noprint")) {
|
||||
sup.remove();
|
||||
}
|
||||
|
||||
for (const sel of [
|
||||
".reflist",
|
||||
".references",
|
||||
".refbegin",
|
||||
"#References",
|
||||
".catlinks",
|
||||
".mw-authority-control",
|
||||
]) {
|
||||
root.querySelectorAll(sel).forEach((el) => el.remove());
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { htmlToMarkdown };
|
||||
@@ -0,0 +1,190 @@
|
||||
const path = require("path");
|
||||
const { validURL } = require("../../utils/url");
|
||||
const { processSingleFile } = require("../../processSingleFile");
|
||||
const { downloadURIToFile } = require("../../utils/downloadURIToFile");
|
||||
const { ACCEPTED_MIMES } = require("../../utils/constants");
|
||||
const { validYoutubeVideoUrl } = require("../../utils/url");
|
||||
|
||||
/**
|
||||
* Parse a Content-Type header value and return the MIME type without charset or other parameters.
|
||||
* @param {string|null} contentTypeHeader - The raw Content-Type header value
|
||||
* @returns {string|null} - The MIME type (e.g., "application/pdf") or null
|
||||
*/
|
||||
function parseContentType(contentTypeHeader) {
|
||||
if (!contentTypeHeader) return null;
|
||||
return contentTypeHeader.toLowerCase().split(";")[0].trim() || null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the content type of a resource
|
||||
* - Sends a HEAD request to the URL and returns the Content-Type header with a 5 second timeout
|
||||
* @param {string} url - The URL to get the content type of
|
||||
* @returns {Promise<{success: boolean, reason: string|null, contentType: string|null}>} - The content type of the resource
|
||||
*/
|
||||
async function getContentTypeFromURL(url) {
|
||||
try {
|
||||
if (!url || typeof url !== "string" || !validURL(url))
|
||||
return { success: false, reason: "Not a valid URL.", contentType: null };
|
||||
|
||||
const abortController = new AbortController();
|
||||
const timeout = setTimeout(() => {
|
||||
abortController.abort();
|
||||
console.error("Timeout fetching content type for URL:", url.toString());
|
||||
}, 5_000);
|
||||
|
||||
const res = await fetch(url, {
|
||||
method: "HEAD",
|
||||
signal: abortController.signal,
|
||||
}).finally(() => clearTimeout(timeout));
|
||||
|
||||
if (!res.ok)
|
||||
return {
|
||||
success: false,
|
||||
reason: `HTTP ${res.status}: ${res.statusText}`,
|
||||
contentType: null,
|
||||
};
|
||||
|
||||
const contentTypeWithoutCharset = parseContentType(
|
||||
res.headers.get("Content-Type")
|
||||
);
|
||||
if (!contentTypeWithoutCharset)
|
||||
return {
|
||||
success: false,
|
||||
reason: "No Content-Type found.",
|
||||
contentType: null,
|
||||
};
|
||||
return {
|
||||
success: true,
|
||||
reason: null,
|
||||
contentType: contentTypeWithoutCharset,
|
||||
};
|
||||
} catch (error) {
|
||||
return {
|
||||
success: false,
|
||||
reason: `Error: ${error.message}`,
|
||||
contentType: null,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize the result object based on the saveAsDocument flag
|
||||
* @param {Object} result - The result object to normalize
|
||||
* @param {boolean} result.success - Whether the result is successful
|
||||
* @param {string|null} result.reason - The reason for the result
|
||||
* @param {Object[]} result.documents - The documents from the result
|
||||
* @param {string|null} result.content - The content of the result
|
||||
* @param {boolean} result.saveAsDocument - Whether to save the content as a document. Default is true
|
||||
* @returns {{success: boolean, reason: string|null, documents: Object[], content: string|null}} - The normalized result object
|
||||
*/
|
||||
function returnResult({
|
||||
success,
|
||||
reason,
|
||||
documents,
|
||||
content,
|
||||
saveAsDocument = true,
|
||||
} = {}) {
|
||||
if (!saveAsDocument) {
|
||||
return {
|
||||
success,
|
||||
content,
|
||||
};
|
||||
} else return { success, reason, documents };
|
||||
}
|
||||
|
||||
/**
|
||||
* Determine the content type of a link - should be a URL
|
||||
* @param {string} uri - The link to determine the content type of
|
||||
* @returns {Promise<{contentType: string|null, processVia: 'web' | 'file' | 'youtube'}>} - The content type of the link
|
||||
*/
|
||||
async function determineContentType(uri) {
|
||||
let processVia = "web";
|
||||
|
||||
// Dont check for content type if it is a YouTube video URL
|
||||
if (validYoutubeVideoUrl(uri))
|
||||
return { contentType: "text/html", processVia: "youtube" };
|
||||
|
||||
return await getContentTypeFromURL(uri)
|
||||
.then((result) => {
|
||||
if (!!result.reason) console.error(result.reason);
|
||||
|
||||
// If the content type is not text/html or text/plain, and it is in the ACCEPTED_MIMES,
|
||||
// then we can process it as a file
|
||||
if (
|
||||
!!result.contentType &&
|
||||
!["text/html", "text/plain"].includes(result.contentType) &&
|
||||
result.contentType in ACCEPTED_MIMES
|
||||
)
|
||||
processVia = "file";
|
||||
|
||||
return { contentType: result.contentType, processVia };
|
||||
})
|
||||
.catch((error) => {
|
||||
console.error("Error getting content type from URL", error);
|
||||
return { contentType: null, processVia };
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Process a link as a file
|
||||
* @param {string} uri - The link to process as a file
|
||||
* @param {boolean} saveAsDocument - Whether to save the content as a document. Default is true
|
||||
* @returns {Promise<{success: boolean, reason: string|null, documents: Object[], content: string|null, saveAsDocument: boolean}>} - The content of the file
|
||||
*/
|
||||
async function processAsFile({ uri, saveAsDocument = true }) {
|
||||
const fileContentResult = await downloadURIToFile(uri);
|
||||
if (!fileContentResult.success)
|
||||
return returnResult({
|
||||
success: false,
|
||||
reason: fileContentResult.reason,
|
||||
documents: [],
|
||||
content: null,
|
||||
saveAsDocument,
|
||||
});
|
||||
|
||||
const fileFilePath = fileContentResult.fileLocation;
|
||||
const targetFilename = path.basename(fileFilePath);
|
||||
|
||||
/**
|
||||
* If the saveAsDocument is false, we are only interested in the text content
|
||||
* and can ignore the file as a document by using `parseOnly` in the options.
|
||||
* This will send the file to the Direct Uploads folder instead of the Documents folder.
|
||||
* that will be deleted by the cleanup-orphan-documents job that runs frequently. The trade off
|
||||
* is that since it still is in FS we can debug its output or even potentially reuse it for other purposes.
|
||||
*
|
||||
* TODO: Improve this process via a new option that will instantly delete the file after processing
|
||||
* if we find we dont need this file ever after processing.
|
||||
*/
|
||||
const processSingleFileResult = await processSingleFile(targetFilename, {
|
||||
parseOnly: saveAsDocument === false,
|
||||
});
|
||||
if (!processSingleFileResult.success) {
|
||||
return returnResult({
|
||||
success: false,
|
||||
reason: processSingleFileResult.reason,
|
||||
documents: [],
|
||||
content: null,
|
||||
saveAsDocument,
|
||||
});
|
||||
}
|
||||
|
||||
// If we intend to return only the text content, return the content from the file
|
||||
// and then delete the file - otherwise it will be saved as a document
|
||||
if (!saveAsDocument) {
|
||||
return returnResult({
|
||||
success: true,
|
||||
content: processSingleFileResult.documents[0].pageContent,
|
||||
saveAsDocument,
|
||||
});
|
||||
}
|
||||
|
||||
return processSingleFileResult;
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
parseContentType,
|
||||
returnResult,
|
||||
getContentTypeFromURL,
|
||||
determineContentType,
|
||||
processAsFile,
|
||||
};
|
||||
@@ -0,0 +1,47 @@
|
||||
const { validURL } = require("../utils/url");
|
||||
const { scrapeGenericUrl } = require("./convert/generic");
|
||||
const { validateURL } = require("../utils/url");
|
||||
|
||||
/**
|
||||
* Process a link and return the text content. This util will save the link as a document
|
||||
* so it can be used for embedding later.
|
||||
* @param {string} link - The link to process
|
||||
* @param {{[key: string]: string}} scraperHeaders - Custom headers to apply when scraping the link
|
||||
* @param {Object} metadata - Optional metadata to attach to the document
|
||||
* @returns {Promise<{success: boolean, content: string}>} - Response from collector
|
||||
*/
|
||||
async function processLink(link, scraperHeaders = {}, metadata = {}) {
|
||||
const validatedLink = validateURL(link);
|
||||
if (!validURL(validatedLink))
|
||||
return { success: false, reason: "Not a valid URL." };
|
||||
return await scrapeGenericUrl({
|
||||
link: validatedLink,
|
||||
captureAs: "text",
|
||||
scraperHeaders,
|
||||
metadata,
|
||||
saveAsDocument: true,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the text content of a link - does not save the link as a document
|
||||
* Mostly used in agentic flows/tools calls to get the text content of a link
|
||||
* @param {string} link - The link to get the text content of
|
||||
* @param {('html' | 'text' | 'json')} captureAs - The format to capture the page content as
|
||||
* @returns {Promise<{success: boolean, content: string}>} - Response from collector
|
||||
*/
|
||||
async function getLinkText(link, captureAs = "text") {
|
||||
const validatedLink = validateURL(link);
|
||||
if (!validURL(validatedLink))
|
||||
return { success: false, reason: "Not a valid URL." };
|
||||
return await scrapeGenericUrl({
|
||||
link: validatedLink,
|
||||
captureAs,
|
||||
saveAsDocument: false,
|
||||
});
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
processLink,
|
||||
getLinkText,
|
||||
};
|
||||
@@ -0,0 +1,83 @@
|
||||
const { v4 } = require("uuid");
|
||||
const { writeToServerDocuments } = require("../utils/files");
|
||||
const { tokenizeString } = require("../utils/tokenizer");
|
||||
const { default: slugify } = require("slugify");
|
||||
|
||||
// Will remove the last .extension from the input
|
||||
// and stringify the input + move to lowercase.
|
||||
function stripAndSlug(input) {
|
||||
if (!input.includes(".")) return slugify(input, { lower: true });
|
||||
return slugify(input.split(".").slice(0, -1).join("-"), { lower: true });
|
||||
}
|
||||
|
||||
const METADATA_KEYS = {
|
||||
possible: {
|
||||
url: ({ url, title }) => {
|
||||
let validUrl;
|
||||
try {
|
||||
const u = new URL(url);
|
||||
validUrl = ["https:", "http:"].includes(u.protocol);
|
||||
} catch {}
|
||||
|
||||
if (validUrl) return `web://${url.toLowerCase()}.website`;
|
||||
return `file://${stripAndSlug(title)}.txt`;
|
||||
},
|
||||
title: ({ title }) => `${stripAndSlug(title)}.txt`,
|
||||
docAuthor: ({ docAuthor }) => {
|
||||
return typeof docAuthor === "string" ? docAuthor : "no author specified";
|
||||
},
|
||||
description: ({ description }) => {
|
||||
return typeof description === "string"
|
||||
? description
|
||||
: "no description found";
|
||||
},
|
||||
docSource: ({ docSource }) => {
|
||||
return typeof docSource === "string" ? docSource : "no source set";
|
||||
},
|
||||
chunkSource: ({ chunkSource, title }) => {
|
||||
return typeof chunkSource === "string"
|
||||
? chunkSource
|
||||
: `${stripAndSlug(title)}.txt`;
|
||||
},
|
||||
published: ({ published }) => {
|
||||
if (isNaN(Number(published))) return new Date().toLocaleString();
|
||||
return new Date(Number(published)).toLocaleString();
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
async function processRawText(textContent, metadata) {
|
||||
console.log(`-- Working Raw Text doc ${metadata.title} --`);
|
||||
if (!textContent || textContent.length === 0) {
|
||||
return {
|
||||
success: false,
|
||||
reason: "textContent was empty - nothing to process.",
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
|
||||
const data = {
|
||||
id: v4(),
|
||||
url: METADATA_KEYS.possible.url(metadata),
|
||||
title: METADATA_KEYS.possible.title(metadata),
|
||||
docAuthor: METADATA_KEYS.possible.docAuthor(metadata),
|
||||
description: METADATA_KEYS.possible.description(metadata),
|
||||
docSource: METADATA_KEYS.possible.docSource(metadata),
|
||||
chunkSource: METADATA_KEYS.possible.chunkSource(metadata),
|
||||
published: METADATA_KEYS.possible.published(metadata),
|
||||
wordCount: textContent.split(" ").length,
|
||||
pageContent: textContent,
|
||||
token_count_estimate: tokenizeString(textContent),
|
||||
};
|
||||
|
||||
const document = writeToServerDocuments({
|
||||
data,
|
||||
filename: `raw-${stripAndSlug(metadata.title)}-${data.id}`,
|
||||
});
|
||||
console.log(
|
||||
`[SUCCESS]: Raw text and metadata saved & ready for embedding.\n`
|
||||
);
|
||||
return { success: true, reason: null, documents: [document] };
|
||||
}
|
||||
|
||||
module.exports = { processRawText };
|
||||
@@ -0,0 +1,83 @@
|
||||
const { v4 } = require("uuid");
|
||||
const {
|
||||
createdDate,
|
||||
trashFile,
|
||||
writeToServerDocuments,
|
||||
} = require("../../utils/files");
|
||||
const { tokenizeString } = require("../../utils/tokenizer");
|
||||
const { default: slugify } = require("slugify");
|
||||
const { LocalWhisper } = require("../../utils/WhisperProviders/localWhisper");
|
||||
const { OpenAiWhisper } = require("../../utils/WhisperProviders/OpenAiWhisper");
|
||||
const {
|
||||
GenericOpenAiWhisper,
|
||||
} = require("../../utils/WhisperProviders/GenericOpenAiWhisper");
|
||||
|
||||
const WHISPER_PROVIDERS = {
|
||||
openai: OpenAiWhisper,
|
||||
"generic-openai": GenericOpenAiWhisper,
|
||||
local: LocalWhisper,
|
||||
};
|
||||
|
||||
async function asAudio({
|
||||
fullFilePath = "",
|
||||
filename = "",
|
||||
options = {},
|
||||
metadata = {},
|
||||
}) {
|
||||
const WhisperProvider = WHISPER_PROVIDERS.hasOwnProperty(
|
||||
options?.whisperProvider
|
||||
)
|
||||
? WHISPER_PROVIDERS[options?.whisperProvider]
|
||||
: WHISPER_PROVIDERS.local;
|
||||
|
||||
console.log(`-- Working ${filename} --`);
|
||||
const whisper = new WhisperProvider({ options });
|
||||
const { content, error } = await whisper.processFile(fullFilePath, filename);
|
||||
|
||||
if (!!error) {
|
||||
console.error(`Error encountered for parsing of ${filename}.`);
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
return {
|
||||
success: false,
|
||||
reason: error,
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
|
||||
if (!content?.length) {
|
||||
console.error(`Resulting text content was empty for ${filename}.`);
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
return {
|
||||
success: false,
|
||||
reason: `No text content found in ${filename}.`,
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
|
||||
const data = {
|
||||
id: v4(),
|
||||
url: "file://" + fullFilePath,
|
||||
title: metadata.title || filename,
|
||||
docAuthor: metadata.docAuthor || "no author found",
|
||||
description: metadata.description || "No description found.",
|
||||
docSource: metadata.docSource || "audio file uploaded by the user.",
|
||||
chunkSource: metadata.chunkSource || "",
|
||||
published: createdDate(fullFilePath),
|
||||
wordCount: content.split(" ").length,
|
||||
pageContent: content,
|
||||
token_count_estimate: tokenizeString(content),
|
||||
};
|
||||
|
||||
const document = writeToServerDocuments({
|
||||
data,
|
||||
filename: `${slugify(filename)}-${data.id}`,
|
||||
options: { parseOnly: options.parseOnly },
|
||||
});
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
console.log(
|
||||
`[SUCCESS]: ${filename} transcribed, converted & ready for embedding.\n`
|
||||
);
|
||||
return { success: true, reason: null, documents: [document] };
|
||||
}
|
||||
|
||||
module.exports = asAudio;
|
||||
@@ -0,0 +1,63 @@
|
||||
const { v4 } = require("uuid");
|
||||
const { DocxLoader } = require("langchain/document_loaders/fs/docx");
|
||||
const {
|
||||
createdDate,
|
||||
trashFile,
|
||||
writeToServerDocuments,
|
||||
} = require("../../utils/files");
|
||||
const { tokenizeString } = require("../../utils/tokenizer");
|
||||
const { default: slugify } = require("slugify");
|
||||
|
||||
async function asDocX({
|
||||
fullFilePath = "",
|
||||
filename = "",
|
||||
options = {},
|
||||
metadata = {},
|
||||
}) {
|
||||
const loader = new DocxLoader(fullFilePath);
|
||||
|
||||
console.log(`-- Working ${filename} --`);
|
||||
let pageContent = [];
|
||||
const docs = await loader.load();
|
||||
for (const doc of docs) {
|
||||
console.log(`-- Parsing content from docx page --`);
|
||||
if (!doc.pageContent.length) continue;
|
||||
pageContent.push(doc.pageContent);
|
||||
}
|
||||
|
||||
if (!pageContent.length) {
|
||||
console.error(`Resulting text content was empty for ${filename}.`);
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
return {
|
||||
success: false,
|
||||
reason: `No text content found in ${filename}.`,
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
|
||||
const content = pageContent.join("");
|
||||
const data = {
|
||||
id: v4(),
|
||||
url: "file://" + fullFilePath,
|
||||
title: metadata.title || filename,
|
||||
docAuthor: metadata.docAuthor || "no author found",
|
||||
description: metadata.description || "No description found.",
|
||||
docSource: metadata.docSource || "docx file uploaded by the user.",
|
||||
chunkSource: metadata.chunkSource || "",
|
||||
published: createdDate(fullFilePath),
|
||||
wordCount: content.split(" ").length,
|
||||
pageContent: content,
|
||||
token_count_estimate: tokenizeString(content),
|
||||
};
|
||||
|
||||
const document = writeToServerDocuments({
|
||||
data,
|
||||
filename: `${slugify(filename)}-${data.id}`,
|
||||
options: { parseOnly: options.parseOnly },
|
||||
});
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
console.log(`[SUCCESS]: ${filename} converted & ready for embedding.\n`);
|
||||
return { success: true, reason: null, documents: [document] };
|
||||
}
|
||||
|
||||
module.exports = asDocX;
|
||||
@@ -0,0 +1,61 @@
|
||||
const { v4 } = require("uuid");
|
||||
const { EPubLoader } = require("langchain/document_loaders/fs/epub");
|
||||
const { tokenizeString } = require("../../utils/tokenizer");
|
||||
const {
|
||||
createdDate,
|
||||
trashFile,
|
||||
writeToServerDocuments,
|
||||
} = require("../../utils/files");
|
||||
const { default: slugify } = require("slugify");
|
||||
|
||||
async function asEPub({
|
||||
fullFilePath = "",
|
||||
filename = "",
|
||||
options = {},
|
||||
metadata = {},
|
||||
}) {
|
||||
let content = "";
|
||||
try {
|
||||
const loader = new EPubLoader(fullFilePath, { splitChapters: false });
|
||||
const docs = await loader.load();
|
||||
docs.forEach((doc) => (content += doc.pageContent));
|
||||
} catch (err) {
|
||||
console.error("Could not read epub file!", err);
|
||||
}
|
||||
|
||||
if (!content?.length) {
|
||||
console.error(`Resulting text content was empty for ${filename}.`);
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
return {
|
||||
success: false,
|
||||
reason: `No text content found in ${filename}.`,
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
|
||||
console.log(`-- Working ${filename} --`);
|
||||
const data = {
|
||||
id: v4(),
|
||||
url: "file://" + fullFilePath,
|
||||
title: metadata.title || filename,
|
||||
docAuthor: metadata.docAuthor || "Unknown",
|
||||
description: metadata.description || "Unknown",
|
||||
docSource: metadata.docSource || "epub file uploaded by the user.",
|
||||
chunkSource: metadata.chunkSource || "",
|
||||
published: createdDate(fullFilePath),
|
||||
wordCount: content.split(" ").length,
|
||||
pageContent: content,
|
||||
token_count_estimate: tokenizeString(content),
|
||||
};
|
||||
|
||||
const document = writeToServerDocuments({
|
||||
data,
|
||||
filename: `${slugify(filename)}-${data.id}`,
|
||||
options: { parseOnly: options.parseOnly },
|
||||
});
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
console.log(`[SUCCESS]: ${filename} converted & ready for embedding.\n`);
|
||||
return { success: true, reason: null, documents: [document] };
|
||||
}
|
||||
|
||||
module.exports = asEPub;
|
||||
@@ -0,0 +1,56 @@
|
||||
const { v4 } = require("uuid");
|
||||
const { tokenizeString } = require("../../utils/tokenizer");
|
||||
const {
|
||||
createdDate,
|
||||
trashFile,
|
||||
writeToServerDocuments,
|
||||
} = require("../../utils/files");
|
||||
const OCRLoader = require("../../utils/OCRLoader");
|
||||
const { default: slugify } = require("slugify");
|
||||
|
||||
async function asImage({
|
||||
fullFilePath = "",
|
||||
filename = "",
|
||||
options = {},
|
||||
metadata = {},
|
||||
}) {
|
||||
let content = await new OCRLoader({
|
||||
targetLanguages: options?.ocr?.langList,
|
||||
}).ocrImage(fullFilePath);
|
||||
|
||||
if (!content?.length) {
|
||||
console.error(`Resulting text content was empty for ${filename}.`);
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
return {
|
||||
success: false,
|
||||
reason: `No text content found in ${filename}.`,
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
|
||||
console.log(`-- Working ${filename} --`);
|
||||
const data = {
|
||||
id: v4(),
|
||||
url: "file://" + fullFilePath,
|
||||
title: metadata.title || filename,
|
||||
docAuthor: metadata.docAuthor || "Unknown",
|
||||
description: metadata.description || "Unknown",
|
||||
docSource: metadata.docSource || "image file uploaded by the user.",
|
||||
chunkSource: metadata.chunkSource || "",
|
||||
published: createdDate(fullFilePath),
|
||||
wordCount: content.split(" ").length,
|
||||
pageContent: content,
|
||||
token_count_estimate: tokenizeString(content),
|
||||
};
|
||||
|
||||
const document = writeToServerDocuments({
|
||||
data,
|
||||
filename: `${slugify(filename)}-${data.id}`,
|
||||
options: { parseOnly: options.parseOnly },
|
||||
});
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
console.log(`[SUCCESS]: ${filename} converted & ready for embedding.\n`);
|
||||
return { success: true, reason: null, documents: [document] };
|
||||
}
|
||||
|
||||
module.exports = asImage;
|
||||
@@ -0,0 +1,83 @@
|
||||
const { v4 } = require("uuid");
|
||||
const fs = require("fs");
|
||||
const { mboxParser } = require("mbox-parser");
|
||||
const {
|
||||
createdDate,
|
||||
trashFile,
|
||||
writeToServerDocuments,
|
||||
} = require("../../utils/files");
|
||||
const { tokenizeString } = require("../../utils/tokenizer");
|
||||
const { default: slugify } = require("slugify");
|
||||
|
||||
async function asMbox({
|
||||
fullFilePath = "",
|
||||
filename = "",
|
||||
options = {},
|
||||
metadata = {},
|
||||
}) {
|
||||
console.log(`-- Working ${filename} --`);
|
||||
|
||||
const mails = await mboxParser(fs.createReadStream(fullFilePath))
|
||||
.then((mails) => mails)
|
||||
.catch((error) => {
|
||||
console.log(`Could not parse mail items`, error);
|
||||
return [];
|
||||
});
|
||||
|
||||
if (!mails.length) {
|
||||
console.error(`Resulting mail items was empty for ${filename}.`);
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
return {
|
||||
success: false,
|
||||
reason: `No mail items found in ${filename}.`,
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
|
||||
let item = 1;
|
||||
const documents = [];
|
||||
for (const mail of mails) {
|
||||
if (!mail.hasOwnProperty("text")) continue;
|
||||
|
||||
const content = mail.text;
|
||||
if (!content) continue;
|
||||
console.log(
|
||||
`-- Working on message "${mail.subject || "Unknown subject"}" --`
|
||||
);
|
||||
|
||||
const data = {
|
||||
id: v4(),
|
||||
url: "file://" + fullFilePath,
|
||||
title:
|
||||
metadata.title ||
|
||||
(mail?.subject
|
||||
? slugify(mail?.subject?.replace(".", "")) + ".mbox"
|
||||
: `msg_${item}-${filename}`),
|
||||
docAuthor: metadata.docAuthor || mail?.from?.text,
|
||||
description: metadata.description || "No description found.",
|
||||
docSource:
|
||||
metadata.docSource || "Mbox message file uploaded by the user.",
|
||||
chunkSource: metadata.chunkSource || "",
|
||||
published: createdDate(fullFilePath),
|
||||
wordCount: content.split(" ").length,
|
||||
pageContent: content,
|
||||
token_count_estimate: tokenizeString(content),
|
||||
};
|
||||
|
||||
item++;
|
||||
const document = writeToServerDocuments({
|
||||
data,
|
||||
filename: `${slugify(filename)}-${data.id}-msg-${item}`,
|
||||
options: { parseOnly: options.parseOnly },
|
||||
});
|
||||
documents.push(document);
|
||||
}
|
||||
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
console.log(
|
||||
`[SUCCESS]: ${filename} messages converted & ready for embedding.\n`
|
||||
);
|
||||
return { success: true, reason: null, documents };
|
||||
}
|
||||
|
||||
module.exports = asMbox;
|
||||
@@ -0,0 +1,59 @@
|
||||
const { v4 } = require("uuid");
|
||||
const officeParser = require("officeparser");
|
||||
const {
|
||||
createdDate,
|
||||
trashFile,
|
||||
writeToServerDocuments,
|
||||
} = require("../../utils/files");
|
||||
const { tokenizeString } = require("../../utils/tokenizer");
|
||||
const { default: slugify } = require("slugify");
|
||||
|
||||
async function asOfficeMime({
|
||||
fullFilePath = "",
|
||||
filename = "",
|
||||
options = {},
|
||||
metadata = {},
|
||||
}) {
|
||||
console.log(`-- Working ${filename} --`);
|
||||
let content = "";
|
||||
try {
|
||||
content = await officeParser.parseOfficeAsync(fullFilePath);
|
||||
} catch (error) {
|
||||
console.error(`Could not parse office or office-like file`, error);
|
||||
}
|
||||
|
||||
if (!content.length) {
|
||||
console.error(`Resulting text content was empty for ${filename}.`);
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
return {
|
||||
success: false,
|
||||
reason: `No text content found in ${filename}.`,
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
|
||||
const data = {
|
||||
id: v4(),
|
||||
url: "file://" + fullFilePath,
|
||||
title: metadata.title || filename,
|
||||
docAuthor: metadata.docAuthor || "no author found",
|
||||
description: metadata.description || "No description found.",
|
||||
docSource: metadata.docSource || "Office file uploaded by the user.",
|
||||
chunkSource: metadata.chunkSource || "",
|
||||
published: createdDate(fullFilePath),
|
||||
wordCount: content.split(" ").length,
|
||||
pageContent: content,
|
||||
token_count_estimate: tokenizeString(content),
|
||||
};
|
||||
|
||||
const document = writeToServerDocuments({
|
||||
data,
|
||||
filename: `${slugify(filename)}-${data.id}`,
|
||||
options: { parseOnly: options.parseOnly },
|
||||
});
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
console.log(`[SUCCESS]: ${filename} converted & ready for embedding.\n`);
|
||||
return { success: true, reason: null, documents: [document] };
|
||||
}
|
||||
|
||||
module.exports = asOfficeMime;
|
||||
@@ -0,0 +1,97 @@
|
||||
const fs = require("fs").promises;
|
||||
|
||||
class PDFLoader {
|
||||
constructor(filePath, { splitPages = true } = {}) {
|
||||
this.filePath = filePath;
|
||||
this.splitPages = splitPages;
|
||||
}
|
||||
|
||||
async load() {
|
||||
const buffer = await fs.readFile(this.filePath);
|
||||
const { getDocument, version } = await this.getPdfJS();
|
||||
|
||||
const pdf = await getDocument({
|
||||
data: new Uint8Array(buffer),
|
||||
useWorkerFetch: false,
|
||||
isEvalSupported: false,
|
||||
useSystemFonts: true,
|
||||
}).promise;
|
||||
|
||||
const meta = await pdf.getMetadata().catch(() => null);
|
||||
const documents = [];
|
||||
|
||||
for (let i = 1; i <= pdf.numPages; i += 1) {
|
||||
const page = await pdf.getPage(i);
|
||||
const content = await page.getTextContent();
|
||||
|
||||
if (content.items.length === 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let lastY;
|
||||
const textItems = [];
|
||||
for (const item of content.items) {
|
||||
if ("str" in item) {
|
||||
if (lastY === item.transform[5] || !lastY) {
|
||||
textItems.push(item.str);
|
||||
} else {
|
||||
textItems.push(`\n${item.str}`);
|
||||
}
|
||||
lastY = item.transform[5];
|
||||
}
|
||||
}
|
||||
|
||||
const text = textItems.join("");
|
||||
documents.push({
|
||||
pageContent: text.trim(),
|
||||
metadata: {
|
||||
source: this.filePath,
|
||||
pdf: {
|
||||
version,
|
||||
info: meta?.info,
|
||||
metadata: meta?.metadata,
|
||||
totalPages: pdf.numPages,
|
||||
},
|
||||
loc: { pageNumber: i },
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
if (this.splitPages) {
|
||||
return documents;
|
||||
}
|
||||
|
||||
if (documents.length === 0) {
|
||||
return [];
|
||||
}
|
||||
|
||||
return [
|
||||
{
|
||||
pageContent: documents.map((doc) => doc.pageContent).join("\n\n"),
|
||||
metadata: {
|
||||
source: this.filePath,
|
||||
pdf: {
|
||||
version,
|
||||
info: meta?.info,
|
||||
metadata: meta?.metadata,
|
||||
totalPages: pdf.numPages,
|
||||
},
|
||||
},
|
||||
},
|
||||
];
|
||||
}
|
||||
|
||||
async getPdfJS() {
|
||||
try {
|
||||
const pdfjs = await import("pdf-parse/lib/pdf.js/v1.10.100/build/pdf.js");
|
||||
return { getDocument: pdfjs.getDocument, version: pdfjs.version };
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
throw new Error(
|
||||
"Failed to load pdf-parse. Please install it with eg. `npm install pdf-parse`."
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = PDFLoader;
|
||||
@@ -0,0 +1,86 @@
|
||||
const { v4 } = require("uuid");
|
||||
const {
|
||||
createdDate,
|
||||
trashFile,
|
||||
writeToServerDocuments,
|
||||
} = require("../../../utils/files");
|
||||
const { tokenizeString } = require("../../../utils/tokenizer");
|
||||
const { default: slugify } = require("slugify");
|
||||
const PDFLoader = require("./PDFLoader");
|
||||
const OCRLoader = require("../../../utils/OCRLoader");
|
||||
|
||||
async function asPdf({
|
||||
fullFilePath = "",
|
||||
filename = "",
|
||||
options = {},
|
||||
metadata = {},
|
||||
}) {
|
||||
const pdfLoader = new PDFLoader(fullFilePath, {
|
||||
splitPages: true,
|
||||
});
|
||||
|
||||
console.log(`-- Working ${filename} --`);
|
||||
const pageContent = [];
|
||||
let docs = await pdfLoader.load();
|
||||
|
||||
if (docs.length === 0) {
|
||||
console.log(
|
||||
`[asPDF] No text content found for ${filename}. Will attempt OCR parse.`
|
||||
);
|
||||
docs = await new OCRLoader({
|
||||
targetLanguages: options?.ocr?.langList,
|
||||
}).ocrPDF(fullFilePath);
|
||||
}
|
||||
|
||||
for (const doc of docs) {
|
||||
console.log(
|
||||
`-- Parsing content from pg ${
|
||||
doc.metadata?.loc?.pageNumber || "unknown"
|
||||
} --`
|
||||
);
|
||||
if (!doc.pageContent || !doc.pageContent.length) continue;
|
||||
pageContent.push(doc.pageContent);
|
||||
}
|
||||
|
||||
if (!pageContent.length) {
|
||||
console.error(`[asPDF] Resulting text content was empty for ${filename}.`);
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
return {
|
||||
success: false,
|
||||
reason: `No text content found in ${filename}.`,
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
|
||||
const content = pageContent.join("");
|
||||
const data = {
|
||||
id: v4(),
|
||||
url: "file://" + fullFilePath,
|
||||
title: metadata.title || filename,
|
||||
docAuthor:
|
||||
metadata.docAuthor ||
|
||||
docs[0]?.metadata?.pdf?.info?.Creator ||
|
||||
"no author found",
|
||||
description:
|
||||
metadata.description ||
|
||||
docs[0]?.metadata?.pdf?.info?.Title ||
|
||||
"No description found.",
|
||||
docSource: metadata.docSource || "pdf file uploaded by the user.",
|
||||
chunkSource: metadata.chunkSource || "",
|
||||
published: createdDate(fullFilePath),
|
||||
wordCount: content.split(" ").length,
|
||||
pageContent: content,
|
||||
token_count_estimate: tokenizeString(content),
|
||||
};
|
||||
|
||||
const document = writeToServerDocuments({
|
||||
data,
|
||||
filename: `${slugify(filename)}-${data.id}`,
|
||||
options: { parseOnly: options.parseOnly },
|
||||
});
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
console.log(`[SUCCESS]: ${filename} converted & ready for embedding.\n`);
|
||||
return { success: true, reason: null, documents: [document] };
|
||||
}
|
||||
|
||||
module.exports = asPdf;
|
||||
@@ -0,0 +1,59 @@
|
||||
const { v4 } = require("uuid");
|
||||
const fs = require("fs");
|
||||
const { tokenizeString } = require("../../utils/tokenizer");
|
||||
const {
|
||||
createdDate,
|
||||
trashFile,
|
||||
writeToServerDocuments,
|
||||
} = require("../../utils/files");
|
||||
const { default: slugify } = require("slugify");
|
||||
|
||||
async function asTxt({
|
||||
fullFilePath = "",
|
||||
filename = "",
|
||||
options = {},
|
||||
metadata = {},
|
||||
}) {
|
||||
let content = "";
|
||||
try {
|
||||
content = fs.readFileSync(fullFilePath, "utf8");
|
||||
} catch (err) {
|
||||
console.error("Could not read file!", err);
|
||||
}
|
||||
|
||||
if (!content?.length) {
|
||||
console.error(`Resulting text content was empty for ${filename}.`);
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
return {
|
||||
success: false,
|
||||
reason: `No text content found in ${filename}.`,
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
|
||||
console.log(`-- Working ${filename} --`);
|
||||
const data = {
|
||||
id: v4(),
|
||||
url: "file://" + fullFilePath,
|
||||
title: metadata.title || filename,
|
||||
docAuthor: metadata.docAuthor || "Unknown",
|
||||
description: metadata.description || "Unknown",
|
||||
docSource: metadata.docSource || "a text file uploaded by the user.",
|
||||
chunkSource: metadata.chunkSource || "",
|
||||
published: createdDate(fullFilePath),
|
||||
wordCount: content.split(" ").length,
|
||||
pageContent: content,
|
||||
token_count_estimate: tokenizeString(content),
|
||||
};
|
||||
|
||||
const document = writeToServerDocuments({
|
||||
data,
|
||||
filename: `${slugify(filename)}-${data.id}`,
|
||||
options: { parseOnly: options.parseOnly },
|
||||
});
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
console.log(`[SUCCESS]: ${filename} converted & ready for embedding.\n`);
|
||||
return { success: true, reason: null, documents: [document] };
|
||||
}
|
||||
|
||||
module.exports = asTxt;
|
||||
@@ -0,0 +1,193 @@
|
||||
const { v4 } = require("uuid");
|
||||
const xlsx = require("node-xlsx").default;
|
||||
const path = require("path");
|
||||
const fs = require("fs");
|
||||
const {
|
||||
createdDate,
|
||||
trashFile,
|
||||
writeToServerDocuments,
|
||||
documentsFolder,
|
||||
} = require("../../utils/files");
|
||||
const { tokenizeString } = require("../../utils/tokenizer");
|
||||
const { default: slugify } = require("slugify");
|
||||
|
||||
function convertToCSV(data) {
|
||||
return data
|
||||
.map((row) =>
|
||||
row
|
||||
.map((cell) => {
|
||||
if (cell === null || cell === undefined) return "";
|
||||
if (typeof cell === "string" && cell.includes(","))
|
||||
return `"${cell}"`;
|
||||
return cell;
|
||||
})
|
||||
.join(",")
|
||||
)
|
||||
.join("\n");
|
||||
}
|
||||
|
||||
async function asXlsx({
|
||||
fullFilePath = "",
|
||||
filename = "",
|
||||
options = {},
|
||||
metadata = {},
|
||||
}) {
|
||||
const documents = [];
|
||||
|
||||
try {
|
||||
const workSheetsFromFile = xlsx.parse(fullFilePath);
|
||||
|
||||
if (options.parseOnly) {
|
||||
const allSheetContents = [];
|
||||
let totalWordCount = 0;
|
||||
const sheetNames = [];
|
||||
|
||||
for (const sheet of workSheetsFromFile) {
|
||||
const processed = processSheet(sheet);
|
||||
if (!processed) continue;
|
||||
|
||||
const { name, content, wordCount } = processed;
|
||||
sheetNames.push(name);
|
||||
allSheetContents.push(`\nSheet: ${name}\n${content}`);
|
||||
totalWordCount += wordCount;
|
||||
}
|
||||
|
||||
if (allSheetContents.length === 0) {
|
||||
console.log(`No valid sheets found in ${filename}.`);
|
||||
return {
|
||||
success: false,
|
||||
reason: `No valid sheets found in ${filename}.`,
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
|
||||
const combinedContent = allSheetContents.join("\n");
|
||||
const sheetListText =
|
||||
sheetNames.length > 1
|
||||
? ` (Sheets: ${sheetNames.join(", ")})`
|
||||
: ` (Sheet: ${sheetNames[0]})`;
|
||||
|
||||
const combinedData = {
|
||||
id: v4(),
|
||||
url: `file://${fullFilePath}`,
|
||||
title: metadata.title || `${filename}${sheetListText}`,
|
||||
docAuthor: metadata.docAuthor || "Unknown",
|
||||
description:
|
||||
metadata.description ||
|
||||
`Spreadsheet data from ${filename} containing ${sheetNames.length} ${
|
||||
sheetNames.length === 1 ? "sheet" : "sheets"
|
||||
}`,
|
||||
docSource: metadata.docSource || "an xlsx file uploaded by the user.",
|
||||
chunkSource: metadata.chunkSource || "",
|
||||
published: createdDate(fullFilePath),
|
||||
wordCount: totalWordCount,
|
||||
pageContent: combinedContent,
|
||||
token_count_estimate: tokenizeString(combinedContent),
|
||||
};
|
||||
|
||||
const document = writeToServerDocuments({
|
||||
data: combinedData,
|
||||
filename: `${slugify(path.basename(filename))}-${combinedData.id}`,
|
||||
destinationOverride: null,
|
||||
options: { parseOnly: true },
|
||||
});
|
||||
documents.push(document);
|
||||
console.log(`[SUCCESS]: ${filename} converted & ready for embedding.`);
|
||||
} else {
|
||||
const folderName = slugify(
|
||||
`${path.basename(filename)}-${v4().slice(0, 4)}`,
|
||||
{
|
||||
lower: true,
|
||||
trim: true,
|
||||
}
|
||||
);
|
||||
const outFolderPath = path.resolve(documentsFolder, folderName);
|
||||
if (!fs.existsSync(outFolderPath))
|
||||
fs.mkdirSync(outFolderPath, { recursive: true });
|
||||
|
||||
for (const sheet of workSheetsFromFile) {
|
||||
const processed = processSheet(sheet);
|
||||
if (!processed) continue;
|
||||
|
||||
const { name, content, wordCount } = processed;
|
||||
const sheetData = {
|
||||
id: v4(),
|
||||
url: `file://${path.join(outFolderPath, `${slugify(name)}.csv`)}`,
|
||||
title: metadata.title || `${filename} - Sheet:${name}`,
|
||||
docAuthor: metadata.docAuthor || "Unknown",
|
||||
description:
|
||||
metadata.description || `Spreadsheet data from sheet: ${name}`,
|
||||
docSource: metadata.docSource || "an xlsx file uploaded by the user.",
|
||||
chunkSource: metadata.chunkSource || "",
|
||||
published: createdDate(fullFilePath),
|
||||
wordCount: wordCount,
|
||||
pageContent: content,
|
||||
token_count_estimate: tokenizeString(content),
|
||||
};
|
||||
|
||||
const document = writeToServerDocuments({
|
||||
data: sheetData,
|
||||
filename: `sheet-${slugify(name)}`,
|
||||
destinationOverride: outFolderPath,
|
||||
options: { parseOnly: options.parseOnly },
|
||||
});
|
||||
documents.push(document);
|
||||
console.log(
|
||||
`[SUCCESS]: Sheet "${name}" converted & ready for embedding.`
|
||||
);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
console.error("Could not process xlsx file!", err);
|
||||
return {
|
||||
success: false,
|
||||
reason: `Error processing ${filename}: ${err.message}`,
|
||||
documents: [],
|
||||
};
|
||||
} finally {
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
}
|
||||
|
||||
if (documents.length === 0) {
|
||||
console.error(`No valid sheets found in ${filename}.`);
|
||||
return {
|
||||
success: false,
|
||||
reason: `No valid sheets found in ${filename}.`,
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
|
||||
console.log(
|
||||
`[SUCCESS]: ${filename} fully processed. Created ${documents.length} document(s).\n`
|
||||
);
|
||||
return { success: true, reason: null, documents };
|
||||
}
|
||||
|
||||
/**
|
||||
* Processes a single sheet and returns its content and metadata
|
||||
* @param {{name: string, data: Array<Array<string|number|null|undefined>>}} sheet - Parsed sheet with name and 2D array of cell values
|
||||
* @returns {{name: string, content: string, wordCount: number}|null} - Object with name, CSV content, and word count, or null if sheet is empty
|
||||
*/
|
||||
function processSheet(sheet) {
|
||||
try {
|
||||
const { name, data } = sheet;
|
||||
const content = convertToCSV(data);
|
||||
|
||||
if (!content?.length) {
|
||||
console.log(`Sheet "${name}" is empty. Skipping.`);
|
||||
return null;
|
||||
}
|
||||
|
||||
console.log(`-- Processing sheet: ${name} --`);
|
||||
return {
|
||||
name,
|
||||
content,
|
||||
wordCount: content.split(/\s+/).length,
|
||||
};
|
||||
} catch (err) {
|
||||
console.error(`Error processing sheet "${sheet.name}":`, err);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = asXlsx;
|
||||
@@ -0,0 +1,95 @@
|
||||
const path = require("path");
|
||||
const fs = require("fs");
|
||||
const {
|
||||
WATCH_DIRECTORY,
|
||||
SUPPORTED_FILETYPE_CONVERTERS,
|
||||
} = require("../utils/constants");
|
||||
const {
|
||||
trashFile,
|
||||
isTextType,
|
||||
normalizePath,
|
||||
isWithin,
|
||||
} = require("../utils/files");
|
||||
const RESERVED_FILES = ["__HOTDIR__.md"];
|
||||
|
||||
/**
|
||||
* Process a single file and return the documents
|
||||
* @param {string} targetFilename - The filename to process
|
||||
* @param {Object} options - The options for the file processing
|
||||
* @param {boolean} options.parseOnly - If true, the file will not be saved as a document even when `writeToServerDocuments` is called in the handler. Must be explicitly set to true to use.
|
||||
* @param {string} options.absolutePath - If provided, use this absolute path instead of resolving relative to WATCH_DIRECTORY. For internal use only.
|
||||
* @param {Object} metadata - The metadata for the file processing
|
||||
* @returns {Promise<{success: boolean, reason: string, documents: Object[]}>} - The documents from the file processing
|
||||
*/
|
||||
async function processSingleFile(targetFilename, options = {}, metadata = {}) {
|
||||
const fullFilePath = normalizePath(
|
||||
options.absolutePath || path.resolve(WATCH_DIRECTORY, targetFilename)
|
||||
);
|
||||
|
||||
// If absolute path is not provided, check if the file is within the watch directory
|
||||
// to prevent unauthorized paths from being processed.
|
||||
if (
|
||||
!options.absolutePath &&
|
||||
!isWithin(path.resolve(WATCH_DIRECTORY), fullFilePath)
|
||||
)
|
||||
return {
|
||||
success: false,
|
||||
reason: "Filename is a not a valid path to process.",
|
||||
documents: [],
|
||||
};
|
||||
|
||||
if (RESERVED_FILES.includes(targetFilename))
|
||||
return {
|
||||
success: false,
|
||||
reason: "Filename is a reserved filename and cannot be processed.",
|
||||
documents: [],
|
||||
};
|
||||
|
||||
if (!fs.existsSync(fullFilePath))
|
||||
return {
|
||||
success: false,
|
||||
reason: "File does not exist in upload directory.",
|
||||
documents: [],
|
||||
};
|
||||
|
||||
const fileExtension = path.extname(fullFilePath).toLowerCase();
|
||||
if (fullFilePath.includes(".") && !fileExtension) {
|
||||
return {
|
||||
success: false,
|
||||
reason: `No file extension found. This file cannot be processed.`,
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
|
||||
let processFileAs = fileExtension;
|
||||
if (!SUPPORTED_FILETYPE_CONVERTERS.hasOwnProperty(fileExtension)) {
|
||||
if (isTextType(fullFilePath)) {
|
||||
console.log(
|
||||
`\x1b[33m[Collector]\x1b[0m The provided filetype of ${fileExtension} does not have a preset and will be processed as .txt.`
|
||||
);
|
||||
processFileAs = ".txt";
|
||||
} else {
|
||||
// If absolute path is provided, do NOT trash the file since it is a user provided path.
|
||||
if (!options.absolutePath) trashFile(fullFilePath);
|
||||
return {
|
||||
success: false,
|
||||
reason: `File extension ${fileExtension} not supported for parsing and cannot be assumed as text file type.`,
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
const FileTypeProcessor = require(SUPPORTED_FILETYPE_CONVERTERS[
|
||||
processFileAs
|
||||
]);
|
||||
return await FileTypeProcessor({
|
||||
fullFilePath,
|
||||
filename: targetFilename,
|
||||
options,
|
||||
metadata,
|
||||
});
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
processSingleFile,
|
||||
};
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user