Skip to content

README.md

Latest commit

 

History

History
223 lines (158 loc) · 6.04 KB

README.md

File metadata and controls

223 lines (158 loc) · 6.04 KB

Ubuntu Windows MacOS Code Quality Documentation codecov

NovaLLM

A lightweight and efficient C/C++ library for Large Language Model (LLM) inference. The name Nova reflects our goal to bring a new, powerful, and efficient approach to LLM deployment, making it accessible everywhere.

Features

  • 🚀 Lightweight: Minimal dependencies, focusing on core functionality
  • 🔧 Extensible: Easy to extend with custom models and optimizations
  • 🎯 Efficient: Support for low-bit quantization and custom kernels
  • 🛠️ Portable: Support inference on MacOS/Linux/Windows platforms
  • 👨‍💻 Developer-friendly: Easy to use and integrate into other projects

Supported Models

Language Models

Model Parameters Status
Qwen 1.8B 🟡 In Development
7B ⚪ Planned
14B ⚪ Planned
DeepSeek 7B ⚪ Planned
67B ⚪ Planned
Llama 7B ⚪ Planned

Vision Models

Coming soon...

Quick Start

Prerequisites

  • CMake 3.14 or higher
  • C++17 compatible compiler
  • Conan package manager
  • Python 3.10+ (for Conan)

Building

  1. Clone the repository
git clone https://github.com/peterlau123/NovaLLM.git
cd NovaLLM
  1. Install dependencies and build
# Create and enter build directory
mkdir build && cd build

# Install dependencies with Conan
conan install .. --output-folder=. --build=missing

# Configure and build
cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build .

Scripted builds

Option 1: Unified Build Script (Recommended)

The root build.sh provides a comprehensive build system with full control:

# Basic build (Release mode)
./build.sh

# Build with tests
./build.sh -t

# Clean build (removes all build-* and install-* directories)
./build.sh -c -r

# Build everything (main + tests + standalone + package)
./build.sh -a

# Debug build with verbose output
./build.sh -d -v

# Show all options
./build.sh --help

Key features:

  • -c, --clean: Cleans all build-* and install-* directories (including custom directories specified via --build-dir/--install-dir) before building
  • -r, --release: Build in Release mode (default)
  • -d, --debug: Build in Debug mode
  • -t, --tests: Build and run tests
  • -s, --standalone: Build standalone application
  • -p, --package: Create Conan package
  • -a, --all: Build everything
  • -v, --verbose: Enable verbose output

Option 2: Platform-Specific Scripts

Unified wrapper (auto-detects OS):

scripts/build.sh --type Release --enable-logging ON --with-tests

Or call platform-specific scripts:

# macOS
scripts/build_macos.sh --type Release --enable-logging ON --with-tests
# Ubuntu/Linux
scripts/build_ubuntu.sh --type Debug --enable-logging OFF
# Windows (PowerShell)
scripts/build_windows.ps1 -Configuration Release -EnableLogging ON -WithTests

Note: The root build.sh is more feature-rich and recommended for development, while scripts/build.sh is a lightweight wrapper for CI/CD pipelines.

Makefile builds

  • Use scripts via Make: make script-build (honors BUILD_TYPE, ENABLE_LOGGING, ENABLE_TESTS)
# Build and install (Release by default)
make install
# Debug build with logging disabled
make BUILD_TYPE=Debug ENABLE_LOGGING=OFF install
# Build & run tests (scripted)
make ENABLE_TESTS=ON script-test
# Build and install (Release by default)
make install

# Debug build with logging disabled
make BUILD_TYPE=Debug ENABLE_LOGGING=OFF install

# Build & run tests
make ENABLE_TESTS=ON test
  1. Run tests
# Run all tests
ctest --output-on-failure

# Run specific test
./bin/NovaLLM_tests

Development

Code Style

We use clang-format for code formatting. To format your code:

# Install clang-format
pip install clang-format==14.0.6

# Format code
cmake --build build --target fix-format

Building Documentation

# Build documentation
cmake --build build --target docs

# View documentation
open build/docs/html/index.html

Advanced Usage

Model Quantization

#include <NovaLLM/quantization.hpp>

// Example quantization code
auto quantized_model = NovaLLM::quantize_model(model, NovaLLM::QuantizationType::INT8);

Custom Kernel Integration

#include <NovaLLM/kernels.hpp>

// Example custom kernel usage
NovaLLM::register_custom_kernel("my_kernel", kernel_function);

Contributing

We welcome contributions! Please see our Contributing Guide for details.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

Contact

Star History

Star History Chart