Setup python workflow

CLAUDE.md

@@ -0,0 +1,137 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Overview
+
+DNLP-diff-engine is a C library with Python bindings that provides automatic differentiation for nonlinear optimization problems. It builds expression trees (ASTs) from CVXPY problems and computes first and second derivatives (gradients, Jacobians, Hessians) needed by NLP solvers like IPOPT.
+
+## Build Commands
+
+### Python Package (Recommended)
+
+```bash
+# Install in development mode (builds C library + Python bindings)
+pip install -e .
+
+# Run all Python tests
+pytest
+
+# Run specific test file
+pytest tests/python/test_unconstrained.py
+
+# Run specific test
+pytest tests/python/test_unconstrained.py::test_sum_log
+```
+
+### Standalone C Library
+
+```bash
+# Build core C library and tests
+cmake -B build -S .
+cmake --build build
+
+# Run C tests
+./build/all_tests
+```
+
+### Legacy Python Build (without pip)
+
+```bash
+# Build Python bindings manually (from project root)
+cd python && cmake -B build -S . && cmake --build build && cd ..
+
+# Run tests from python/ directory
+cd python && python tests/test_problem_native.py
+```
+
+## Architecture
+
+### Expression Tree System
+
+The core abstraction is the `expr` struct (in `include/expr.h`) representing a node in an expression AST. Each node stores:
+- Shape information (`d1`, `d2`, `size`, `n_vars`)
+- Function pointers for evaluation: `forward`, `jacobian_init`, `eval_jacobian`, `eval_wsum_hess`
+- Computed values: `value`, `jacobian` (CSR), `wsum_hess` (CSR)
+- Child pointers (`left`, `right`) and reference counting (`refcount`)
+
+### Atom Categories
+
+Atoms are organized by mathematical properties in `src/`:
+
+- **`affine/`** - Linear operations: `variable`, `constant`, `add`, `neg`, `sum`, `promote`, `hstack`, `trace`, `linear_op`
+- **`elementwise_univariate/`** - Scalar functions applied elementwise: `log`, `exp`, `entr`, `power`, `logistic`, trigonometric, hyperbolic
+- **`bivariate/`** - Two-argument operations: `multiply`, `quad_over_lin`, `rel_entr`
+- **`other/`** - Special atoms not fitting above categories
+
+Each atom implements its own `forward`, `jacobian_init`, `eval_jacobian`, and `eval_wsum_hess` functions following a consistent pattern.
+
+### Problem Struct
+
+The `problem` struct (in `include/problem.h`) encapsulates an optimization problem:
+- Single `objective` expression (scalar)
+- Array of `constraints` expressions
+- Pre-allocated storage for `constraint_values`, `gradient_values`, `jacobian`, `lagrange_hessian`
+
+Key oracle methods:
+- `problem_objective_forward(prob, u)` - Evaluate objective at point u
+- `problem_constraint_forward(prob, u)` - Evaluate all constraints at u
+- `problem_gradient(prob)` - Compute objective gradient (after forward)
+- `problem_jacobian(prob)` - Compute stacked constraint Jacobian (after forward)
+- `problem_hessian(prob, obj_w, lambda)` - Compute Lagrangian Hessian
+
+### Python Bindings
+
+The Python package `dnlp_diff_engine` (in `src/dnlp_diff_engine/`) provides:
+
+**High-level API** (`__init__.py`):
+- `C_problem` class wraps the C problem struct
+- `convert_problem()` builds expression trees from CVXPY Problem objects
+- Atoms are mapped via `ATOM_CONVERTERS` dictionary
+
+**Low-level C extension** (`_core` module, built from `python/bindings.c`):
+- Atom constructors: `make_variable`, `make_constant`, `make_log`, `make_exp`, `make_add`, etc.
+- Problem interface: `make_problem`, `problem_init_derivatives`, `problem_objective_forward`, `problem_gradient`, `problem_jacobian`, `problem_hessian`
+
+### Derivative Computation Flow
+
+1. Call `problem_init_derivatives()` to allocate Jacobian/Hessian storage and compute sparsity patterns
+2. Call forward pass (`objective_forward` / `constraint_forward`) to propagate values through tree
+3. Call derivative functions (`gradient`, `jacobian`, `hessian`) which traverse tree computing derivatives
+
+Jacobian uses chain rule: each node computes local Jacobian, combined via sparse matrix operations.
+Hessian computes weighted sum: `obj_w * H_obj + sum(lambda_i * H_constraint_i)`
+
+### Sparse Matrix Utilities
+
+`include/utils/` contains CSR and CSC sparse matrix implementations used throughout for efficient derivative storage and computation.
+
+## Key Directories
+
+- `include/` - Header files defining public API
+- `src/` - C implementation files
+- `src/dnlp_diff_engine/` - Python package (installed via pip)
+- `python/` - Python bindings C code and binding headers
+- `python/atoms/` - Python binding headers for each atom type
+- `python/problem/` - Python binding headers for problem interface
+- `tests/` - C tests using minunit framework
+- `tests/python/` - Python tests (run via pytest)
+- `tests/jacobian_tests/` - Jacobian correctness tests (C)
+- `tests/wsum_hess/` - Hessian correctness tests (C)
+
+## Adding a New Atom
+
+1. Create header in `include/` declaring the constructor function
+2. Create implementation in appropriate `src/` subdirectory
+3. Implement: `forward`, `jacobian_init`, `eval_jacobian`, `eval_wsum_hess` (optional), `free_type_data` (if needed)
+4. Add Python binding header in `python/atoms/`
+5. Register in `python/bindings.c` (both include and method table)
+6. Add converter entry in `src/dnlp_diff_engine/__init__.py` `ATOM_CONVERTERS` dict
+7. Rebuild: `pip install -e .`
+8. Add tests in `tests/` (C) and `tests/python/` (Python)
+
+## License Header
+
+```c
+// SPDX-License-Identifier: Apache-2.0
+```

CMakeLists.txt

@@ -1,47 +1,72 @@
-cmake_minimum_required(VERSION 3.10)
+cmake_minimum_required(VERSION 3.15)
 project(DNLP_Diff_Engine C)
 
 set(CMAKE_C_STANDARD 99)
-set(CMAKE_BUILD_TYPE Debug)
 
-# Debug-friendly flags
-add_compile_options(-g -O0)
+# Set build type if not specified (standalone builds)
+if(NOT CMAKE_BUILD_TYPE AND NOT SKBUILD)
+    set(CMAKE_BUILD_TYPE Debug)
+endif()
 
-# Warning flags
+# Warning flags (always enabled)
 add_compile_options(
     -Wall                  # Enable most warnings
     -Wextra                # Extra warnings
     -Wpedantic             # Strict ISO C compliance
     -Wshadow               # Warn about variable shadowing
     -Wformat=2             # Extra format string checks
-    #-Wconversion           # Warn about implicit conversions
-    #-Wsign-conversion      # Warn about sign conversions
     -Wcast-qual            # Warn about cast that removes qualifiers
     -Wcast-align           # Warn about pointer cast alignment issues
     -Wunused               # Warn about unused variables/functions
     -Wdouble-promotion     # Warn about float->double promotion
     -Wnull-dereference     # Warn about null pointer dereference
-    #-Wstrict-prototypes    # Warn about missing prototypes
 )
 
+# Debug flags only for standalone debug builds
+if(CMAKE_BUILD_TYPE STREQUAL "Debug" AND NOT SKBUILD)
+    add_compile_options(-g -O0)
+endif()
+
 # Include directories
 include_directories(${PROJECT_SOURCE_DIR}/include)
-include_directories(${PROJECT_SOURCE_DIR}/tests)
 
 # Source files - automatically gather all .c files from src/
 file(GLOB_RECURSE SOURCES "src/*.c")
 
-# Create library
+# Create core library
 add_library(dnlp_diff ${SOURCES})
 target_link_libraries(dnlp_diff m)
 
-# Enable testing
-enable_testing()
+# =============================================================================
+# Python bindings (built when using scikit-build-core)
+# =============================================================================
+if(SKBUILD)
+    find_package(Python3 REQUIRED COMPONENTS Interpreter Development.Module NumPy)
 
-# Single test executable combining all tests
-add_executable(all_tests
-    tests/all_tests.c
-    tests/test_helpers.c
-)
-target_link_libraries(all_tests dnlp_diff)
-add_test(NAME AllTests COMMAND all_tests)
+    # Create Python extension module
+    Python3_add_library(_core MODULE python/bindings.c)
+    target_include_directories(_core PRIVATE
+        ${PROJECT_SOURCE_DIR}/include
+        ${PROJECT_SOURCE_DIR}/python
+        ${Python3_NumPy_INCLUDE_DIRS}
+    )
+    target_link_libraries(_core PRIVATE dnlp_diff)
+
+    # Install to the package directory
+    install(TARGETS _core LIBRARY DESTINATION dnlp_diff_engine)
+endif()
+
+# =============================================================================
+# C tests (only for standalone builds)
+# =============================================================================
+if(NOT SKBUILD)
+    include_directories(${PROJECT_SOURCE_DIR}/tests)
+    enable_testing()
+
+    add_executable(all_tests
+        tests/all_tests.c
+        tests/test_helpers.c
+    )
+    target_link_libraries(all_tests dnlp_diff)
+    add_test(NAME AllTests COMMAND all_tests)
+endif()

README.md

@@ -1,14 +1,66 @@
-3. more tests for chain rule elementwise univariate hessian
-4. in the refactor, add consts
-5. multiply with one constant vector/scalar argument
-6. AX where X is a matrix. Can that happen? How is that canonicalized? Maybe it can't happen.
-7. Must be able to compute jacobian and hessian of A @ phi(x), so linear operator needs other code! This requires new infrastructure, I think.
-8. Shortcut hessians of affine stuff?
+# DNLP Diff Engine
 
-Going through all atoms to see that sparsity pattern is computed in initialization of jacobian:
-2. trace - not ok
+A C library with Python bindings for automatic differentiation of nonlinear optimization problems. Builds expression trees from CVXPY problems and computes gradients, Jacobians, and Hessians needed by NLP solvers like IPOPT.
 
-Going through all atoms to see that sparsity pattern is computed in initialization of hessian:
-2. hstack - not ok
-3. trace - not ok
+## Installation
 
+### Using uv (recommended)
+
+```bash
+uv venv .venv
+source .venv/bin/activate
+uv pip install -e ".[test]"
+```
+
+### Using pip
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[test]"
+```
+
+## Running Tests
+
+```bash
+# Run all tests
+pytest
+
+# Run specific test file
+pytest tests/python/test_unconstrained.py
+
+# Run specific test
+pytest tests/python/test_unconstrained.py::test_sum_log
+```
+
+## Usage
+
+```python
+import cvxpy as cp
+import numpy as np
+from dnlp_diff_engine import C_problem
+
+# Define a CVXPY problem
+x = cp.Variable(3)
+problem = cp.Problem(cp.Minimize(cp.sum(cp.log(x))))
+
+# Convert to C problem struct
+prob = C_problem(problem)
+prob.init_derivatives()
+
+# Evaluate at a point
+u = np.array([1.0, 2.0, 3.0])
+obj_val = prob.objective_forward(u)
+gradient = prob.gradient()
+
+print(f"Objective: {obj_val}")
+print(f"Gradient: {gradient}")
+```
+
+## Building the C Library (standalone)
+
+```bash
+cmake -B build -S .
+cmake --build build
+./build/all_tests
+```

TODO.md

@@ -0,0 +1,13 @@
+3. more tests for chain rule elementwise univariate hessian
+4. in the refactor, add consts
+5. multiply with one constant vector/scalar argument
+6. AX where X is a matrix. Can that happen? How is that canonicalized? Maybe it can't happen.
+7. Must be able to compute jacobian and hessian of A @ phi(x), so linear operator needs other code! This requires new infrastructure, I think.
+8. Shortcut hessians of affine stuff?
+
+Going through all atoms to see that sparsity pattern is computed in initialization of jacobian:
+2. trace - not ok
+
+Going through all atoms to see that sparsity pattern is computed in initialization of hessian:
+2. hstack - not ok
+3. trace - not ok

pyproject.toml

@@ -0,0 +1,44 @@
+[build-system]
+requires = ["scikit-build-core>=0.5", "numpy"]
+build-backend = "scikit_build_core.build"
+
+[project]
+name = "dnlp-diff-engine"
+version = "0.1.0"
+description = "Automatic differentiation engine for DNLP optimization problems"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+    "numpy",
+    "scipy",
+    "cvxpy",
+]
+
+[project.optional-dependencies]
+test = [
+    "pytest>=7.0",
+]
+dev = [
+    "pytest>=7.0",
+    "ruff",
+]
+
+[tool.scikit-build]
+cmake.source-dir = "."
+wheel.packages = ["src/dnlp_diff_engine"]
+build-dir = "build/{wheel_tag}"
+
+[tool.scikit-build.cmake.define]
+CMAKE_BUILD_TYPE = "Release"
+
+[tool.pytest.ini_options]
+testpaths = ["tests/python"]
+python_files = ["test_*.py"]
+python_functions = ["test_*"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I"]

python/bindings.c

@@ -58,10 +58,10 @@ static PyMethodDef DNLPMethods[] = {
      "Compute Lagrangian Hessian"},
     {NULL, NULL, 0, NULL}};
 
-static struct PyModuleDef dnlp_module = {PyModuleDef_HEAD_INIT, "DNLP_diff_engine",
+static struct PyModuleDef dnlp_module = {PyModuleDef_HEAD_INIT, "dnlp_diff_engine._core",
                                          NULL, -1, DNLPMethods};
 
-PyMODINIT_FUNC PyInit_DNLP_diff_engine(void)
+PyMODINIT_FUNC PyInit__core(void)
 {
     if (ensure_numpy() < 0) return NULL;
     return PyModule_Create(&dnlp_module);