Skip to content

S3 storage backend integration #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dcadenas merged 1 commit into from
Apr 15, 2025
Merged

S3 storage backend integration #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 0 additions & 52 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

23 changes: 6 additions & 17 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,11 @@ edition = "2021"
name = "route96"
path = "src/bin/main.rs"

[[bin]]
name = "storage-manage"
path = "src/bin/storage-manage.rs"
required-features = ["storage-manage"]

[lib]
name = "route96"

[features]
default = ["media-compression", "sqlx-postgres", "s3-storage", "ranges"]
default = ["media-compression", "sqlx-postgres"]
media-compression = [
"dep:ffmpeg-the-third",
"dep:infer",
Expand All @@ -28,14 +23,10 @@ media-compression = [
"dep:candle-transformers",
]
labels = ["dep:candle-core", "dep:candle-nn", "dep:candle-transformers"]
blossom = []
torrent-v2 = []
react-ui = []
storage-manage = ["dep:walkdir", "dep:indicatif"]
dev = ["blossom", "ranges"]
dev = []
void-cat-redirects = []
s3-storage = ["dep:aws-config", "dep:aws-sdk-s3", "dep:http-body-0-4"]
ranges = ["dep:http-range-header"]

[dependencies]
log = "0.4.27"
Expand Down Expand Up @@ -67,17 +58,15 @@ candle-core = { git = "https://github.com/huggingface/candle.git", tag = "0.8.4"
candle-nn = { git = "https://github.com/huggingface/candle.git", tag = "0.8.4", optional = true }
candle-transformers = { git = "https://github.com/huggingface/candle.git", tag = "0.8.4", optional = true }
sqlx-postgres = { version = "0.8.3", optional = true, features = ["chrono", "uuid"] }
http-range-header = { version = "0.4.2", optional = true }
walkdir = { version = "2.5.0", optional = true }
indicatif = { version = "0.17.11", optional = true }
http-range-header = { version = "0.4.2" }
async-trait = "0.1.80"

futures = "0.3"

# AWS SDK Dependencies for S3/Spaces
aws-config = { version = "1.1.8", optional = true }
aws-sdk-s3 = { version = "1.19.0", optional = true }
http-body-0-4 = { version = "1.0.0", package = "http-body", optional = true }
aws-config = { version = "1.1.8" }
aws-sdk-s3 = { version = "1.19.0" }
http-body-0-4 = { version = "1.0.0", package = "http-body" }

serde_json = "1.0.117"
image = { version = "0.25.1", optional = true }
Expand Down
10 changes: 8 additions & 2 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,15 @@ RUN git clone --single-branch --branch release/7.1 https://github.com/ffmpeg/FFm
RUN rm Cargo.lock
# RUN cargo tree -i half | cat
RUN mkdir -p ~/.cargo && \
echo '[net]' > ~/.cargo/config.toml && \
echo '[build]' > ~/.cargo/config.toml && \
echo 'rustflags = ["-C", "target-cpu=native"]' >> ~/.cargo/config.toml && \
echo '[net]' >> ~/.cargo/config.toml && \
echo 'git-fetch-with-cli = true' >> ~/.cargo/config.toml
RUN cargo install --path . --root /app/build --features "blossom,ranges"

# Perform the installation, enabling only necessary features
# Remove s3-storage and ranges from features
# Ensure default features (like media-compression) are included
RUN cargo install --path . --root /app/build

FROM node:bookworm AS ui_builder
WORKDIR /app/src
Expand Down
7 changes: 3 additions & 4 deletions Dockerfile.arm64
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,8 @@ RUN wget https://ffmpeg.org/releases/ffmpeg-7.1.tar.xz && \
COPY src src
COPY migrations migrations

# Clean potential stale artifacts and Build with hardcoded features using cargo build
RUN cargo clean && \
cargo build --release --no-default-features --features "blossom,ranges"
# Build the binary without default features, enabling only s3-storage and ranges
RUN cargo build --release

FROM --platform=linux/arm64 node:bookworm AS ui_builder
WORKDIR /app/src
Expand All @@ -81,7 +80,7 @@ LABEL org.opencontainers.image.licenses="MIT"
LABEL org.opencontainers.image.authors="Kieran"
WORKDIR /app
RUN apt update && \
apt install -y libx264-164 libwebp7 libvpx7 ca-certificates gosu && \
apt install -y libx264-164 libwebp7 libvpx7 ca-certificates gosu libxcb1 libxcb-shm0 && \
rm -rf /var/lib/apt/lists/*

RUN groupadd -r appgroup && useradd --no-log-init -r -g appgroup appuser
Expand Down
21 changes: 11 additions & 10 deletions config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,28 @@ listen: "0.0.0.0:8000"
# Specifies which storage backend to use ("FileSystem" or "S3")
# Defaults to FileSystem if not specified.
# storage_type: FileSystem
# storage_type: S3 # Requires the 's3-storage' feature to be enabled during build
storage_type: S3

# Database connection string (PostgreSQL)
database: "postgres://postgres:root@db:5432/route96"

# --- Storage Backend Settings ---

# Settings for the FileSystem backend (used if storage_type is FileSystem or omitted)
filesystem:
#filesystem:
# Directory to store uploads
storage_dir: "/app/data"
#storage_dir: "/app/data"

# Settings for the S3 backend (used if storage_type is S3)
# Requires the 's3-storage' feature flag during compilation.
# s3:
# region: "us-east-1" # Optional, defaults to provider default
# endpoint_url: "https://nyc3.digitaloceanspaces.com" # Optional, use for non-AWS S3-compatible storage like DigitalOcean Spaces
# bucket_name: "your-bucket-name"
# access_key_id: "YOUR_ACCESS_KEY"
# secret_access_key: "YOUR_SECRET_KEY"
# force_path_style: false # Optional, default is false. Set to true for some S3-compatible services like MinIO
s3:
region: "us-east-1" # Optional, defaults to provider default
endpoint_url: "https://nyc3.digitaloceanspaces.com" # Optional, use for non-AWS S3-compatible storage like DigitalOcean Spaces
bucket_name: "blossom-test"
access_key_id: "DO801BJEC2FP8YC8B7XD"
# Secret key is now set via environment variable APP__S3__SECRET_ACCESS_KEY
secret_access_key: ""
force_path_style: false # Optional, default is false. Set to true for some S3-compatible services like MinIO
Comment on lines +21 to +28
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Verify S3 credentials handling and security

The configuration includes a hardcoded access key ID and references an environment variable for the secret key. While storing the secret key in an environment variable is good practice, exposing the access key ID in the config file might be a security concern if this file is committed to version control.

Are there plans to also move the access key ID to an environment variable? Additionally, please verify that the DigitalOcean Space (based on the endpoint URL) and credentials are properly configured and secured:

🏁 Script executed:

#!/bin/bash
# Check if the environment variable for the secret key is set in docker-compose.yml
grep -A 10 "environment:" docker-compose.yml | grep "APP__S3__SECRET_ACCESS_KEY"

# Check if there are any environment variables for access key ID
grep -A 10 "environment:" docker-compose.yml | grep -i "access_key"

# Look for sensitive keys in git history (only look for potential patterns, don't display actual keys)
git log -p | grep -i "secret\|key\|token\|password" | wc -l

Length of output: 256

Security Issue: Hardcoded Access Key ID Persists in Config

  • The secret key is correctly externalized via the environment variable (as confirmed by the docker-compose.yml content).
  • However, the S3 configuration in config.yaml still hardcodes the access_key_id ("DO801BJEC2FP8YC8B7XD"). This could expose sensitive credential information if the file is committed to version control.
  • Please consider moving the access key ID to an environment variable (or another secure store) to ensure both parts of the S3 credentials are handled consistently.
  • Additionally, double-check that the DigitalOcean Spaces endpoint (https://nyc3.digitaloceanspaces.com) and its associated credentials are correctly and securely configured.


# Maximum support filesize for uploading
max_upload_bytes: 5242880 # 5 MiB default
Expand Down
6 changes: 2 additions & 4 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,13 @@ services:
build:
context: .
dockerfile: Dockerfile.arm64
args:
# Only including essential features:
# - ranges: For better file serving (partial downloads/streaming)
FEATURES: "ranges"
environment:
# Config values overridden by environment variables - keeping only essential/sensitive ones
APP__DATABASE: postgres://postgres:root@db:5432/route96?sslmode=require
APP__FILESYSTEM__STORAGE_DIR: /app/data
APP__PUBLIC_URL: http://localhost:8000
# S3 credentials
APP__S3__SECRET_ACCESS_KEY: ${BLOSSOM_DEV_SECRET}
Comment on lines +34 to +35
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Ensure S3 credentials are properly secured.

The S3 secret access key is correctly sourced from an environment variable, but make sure this environment variable is properly set and secured. Consider adding validation or a check to ensure the variable is present.

🏁 Script executed:

#!/bin/bash
# Check if the required environment variable is defined
if [ -z "${BLOSSOM_DEV_SECRET}" ]; then
  echo "ERROR: BLOSSOM_DEV_SECRET environment variable is not set"
  echo "This variable is required for S3 storage to work correctly"
  exit 1
fi
echo "BLOSSOM_DEV_SECRET is set (value hidden for security)"

Length of output: 276

Action Required: Secure and properly configure S3 credentials.

The verification script clearly indicates that the BLOSSOM_DEV_SECRET environment variable is not set, causing the container to exit and S3 functionality to fail. Please ensure that this variable is defined—either via a secure environment file or another secure method—before deploying. Additionally, consider adding validation checks in your deployment or container initialization process to catch this error earlier.

  • File: docker-compose.yml (Lines 34-35)
  • Current snippet: 
          # S3 credentials
      APP__S3__SECRET_ACCESS_KEY: ${BLOSSOM_DEV_SECRET}
  • Next steps:
    • Verify that BLOSSOM_DEV_SECRET is properly provided in your environment.
    • If using a .env file or CI/CD secrets manager, double-check that the variable is correctly spelled and loaded.
    • Add runtime validations or startup checks to ensure the presence of this critical credential.

# Nested values with double underscores
APP__NIP29_RELAY__URL: ws://host.docker.internal:8080
# Test key (pubkey: 385c3a6ec0b9d57a4330dbd6284989be5bd00e41c535f9ca39b6ae7c521b81cd) - replace if needed
Expand Down
Loading