117 structure refactor

.github/workflows/python-tests.yml

@@ -3,11 +3,11 @@ name: Python Tests
 on:
   push:
     paths:
-      - 'src/python/**'
+      - 'futag-package/**'
       - '.github/workflows/python-tests.yml'
   pull_request:
     paths:
-      - 'src/python/**'
+      - 'futag-package/**'
 
 jobs:
   test:
@@ -22,9 +22,9 @@ jobs:
           python-version: ${{ matrix.python-version }}
       - name: Install package with test dependencies
         run: |
-          cd src/python/futag-package
+          cd futag-package
           pip install -e ".[test]"
       - name: Run tests
         run: |
-          cd src/python/futag-package
+          cd futag-package
           python -m pytest tests/ -v --tb=short

.github/workflows/syntax-check.yml

@@ -15,7 +15,7 @@ jobs:
           python -c "
           import ast, sys, pathlib
           errors = []
-          for f in pathlib.Path('src/python/futag-package/src/futag').glob('*.py'):
+          for f in pathlib.Path('futag-package/src/futag').glob('*.py'):
               try:
                   ast.parse(f.read_text())
               except SyntaxError as e:

.gitignore

@@ -7,28 +7,28 @@ futag-llvm
 futag-install
 futag-build
 futag-build/**
-custom-llvm/INFO
-custom-llvm/*.tar.xz
-custom-llvm/futag-llvm/**
-custom-llvm/AFLplusplus-4.02c
-custom-llvm/binutils
-custom-llvm/llvm-project-14.0.6.src.tar.xz
-custom-llvm/llvm-project
-custom-llvm/fuzz-introspector
-custom-llvm/*.tar.gz
-custom-llvm/*.tar.xz
+build-llvm/INFO
+build-llvm/*.tar.xz
+build-llvm/futag-llvm/**
+build-llvm/AFLplusplus-4.02c
+build-llvm/binutils
+build-llvm/llvm-project-14.0.6.src.tar.xz
+build-llvm/llvm-project
+build-llvm/fuzz-introspector
+build-llvm/*.tar.gz
+build-llvm/*.tar.xz
 **/futag-llvm.*.latest.tar.xz
 **/futag-llvm.*.tar.xz
 TODO
-product-tests/*.tar.xz
-src/Checkers/lib/CMakeLists.txt.test
-src/Checkers/include/Checkers.td.test
-src/Checkers/lib/FutagTest.cpp
+integration-tests/*.tar.xz
+analyzers/checkers/lib/CMakeLists.txt.test
+analyzers/checkers/include/Checkers.td.test
+analyzers/checkers/lib/FutagTest.cpp
 futag-llvm*
 Algorithms
-src/python/futag-package/dist/**
-src/python/futag-package/build/**
-src/python/futag-package/src/*.egg-info/
+futag-package/dist/**
+futag-package/build/**
+futag-package/src/*.egg-info/
 
 # Python
 __pycache__/

CLAUDE.md

@@ -0,0 +1,125 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+FUTAG (Fuzz target Automated Generator) is a tool from ISP RAS for automated generation of fuzzing wrappers (fuzz targets) for software libraries. It analyzes library source code via custom Clang/LLVM static analysis checkers and generates fuzz targets in LibFuzzer or AFLplusplus format.
+
+## Build Commands
+
+### Building the custom LLVM/Clang toolchain
+
+```bash
+# 1. Download LLVM sources (interactive - select LLVM version)
+cd build-llvm && ./prepare.sh
+
+# 2. Build LLVM with Futag checkers integrated
+cd ../build && ./build.sh
+
+# Output: futag-llvm/ directory with compiled toolchain
+```
+
+### Building AFLplusplus support (optional)
+
+```bash
+cd futag-llvm && ./buildAFLplusplus.sh
+```
+
+### Installing the Python package
+
+```bash
+cd futag-package && pip install .
+```
+
+### Running integration tests (Docker)
+
+Tests are Dockerized per platform in `integration-tests/`:
+- `integration-tests/build-test/` — build validation on Ubuntu 20.04/22.04/24.04, Alt 11/12
+- `integration-tests/libraries-test/` — end-to-end tests against real libraries (json-c, php, FreeImage, etc.)
+- `integration-tests/package-test/` — Python package tests
+
+## Architecture
+
+### Three-layer design
+
+1. **C++ Clang Checkers** (`analyzers/checkers/`, `analyzers/clang-patches/`) — Static analysis plugins that run inside Clang's StaticAnalyzer to extract function signatures, types, and usage patterns from library code. Version-specific implementations exist for LLVM 14 and 18 (files suffixed `14`/`18`).
+
+2. **Python Orchestration** (`futag-package/src/futag/`) — User-facing API that drives the full pipeline:
+   - `preprocessor.py` — `Builder` builds and analyzes target libraries; `ConsumerBuilder` handles library+consumer pairs
+   - `generator.py` — `Generator` produces fuzz targets from analysis JSON; `ContextGenerator` uses consumer usage contexts
+   - `fuzzer.py` — `Fuzzer` executes generated targets with configurable timeouts, memory limits, and sanitizers
+   - `sysmsg.py` — Constants and error messages (LIBFUZZER, AFLPLUSPLUS engine identifiers, paths)
+
+3. **Build Infrastructure** (`build-llvm/`) — Shell scripts that download LLVM sources, patch in Futag's checkers and clang modifications, and build the complete toolchain via CMake.
+
+### Data flow
+
+`Builder.auto_build()` → `Builder.analyze()` (runs Clang checkers, produces JSON) → `Generator.gen_targets()` (reads JSON, writes C fuzz targets) → `Generator.compile_targets()` (compiles with instrumentation) → `Fuzzer.fuzz()` (runs fuzzing)
+
+### LLVM version handling
+
+The project maintains version-specific copies of several files (e.g., `FutagAnalyzer14.cpp` / `FutagAnalyzer18.cpp`, `ASTMatchFinder14.cpp` / `ASTMatchFinder18.cpp`). The build script (`build.sh`) selects the correct version-specific `CMakeLists.txt` and source files during LLVM compilation.
+
+### Key third-party code
+
+- `vendors/json/` — nlohmann/json for C++ JSON serialization in checkers
+- `.clang-format` — LLVM style formatting for C++ code
+
+### Recent refactoring (2024)
+
+The Python package underwent major refactoring:
+- **BaseGenerator ABC** (`base_generator.py`) — shared infrastructure for all 5 generator subclasses
+- **GeneratorState** (`generator_state.py`) — dataclass replacing 13 mutable instance variables
+- **BaseFuzzer** (`fuzzer.py`) — shared fuzzer logic; Fuzzer and NatchFuzzer are thin subclasses
+- **Custom exceptions** (`exceptions.py`) — FutagError hierarchy replacing sys.exit()
+- Generator classes use single-underscore protected methods for proper inheritance
+
+### Running Python tests
+
+```bash
+cd futag-package
+pip install -e ".[test]"
+python -m pytest tests/ -v
+```
+
+### Python module structure
+
+```
+src/futag/
+    base_generator.py        # BaseGenerator ABC (~2500 lines)
+    generator.py             # Generator + re-exports (~235 lines)
+    fdp_generator.py         # FuzzDataProviderGenerator (~220 lines)
+    blob_stamper_generator.py # BlobStamperGenerator (~40 lines)
+    context_generator.py     # ContextGenerator (~820 lines)
+    natch_generator.py       # NatchGenerator (~1240 lines)
+    generator_state.py       # GeneratorState dataclass (~85 lines)
+    preprocessor.py          # Builder, ConsumerBuilder (~960 lines)
+    fuzzer.py                # BaseFuzzer, Fuzzer, NatchFuzzer (~900 lines)
+    toolchain.py             # ToolchainConfig dataclass (~140 lines)
+    exceptions.py            # FutagError hierarchy
+    sysmsg.py                # Constants and messages
+```
+
+## System Requirements
+
+- CMake >= 3.13.4, GCC >= 7.1.0, Python >= 3.8, pip >= 22.1.1
+- On Ubuntu: `python-is-python3`, `gcc-multilib`, `binutils-gold`, `binutils-dev`
+
+## Typical Python API usage
+
+```python
+from futag.preprocessor import *
+from futag.generator import *
+from futag.toolchain import ToolchainConfig
+
+tc = ToolchainConfig.from_futag_llvm(FUTAG_PATH)
+builder = Builder(library_root, clean=True, toolchain=tc)
+builder.auto_build()
+builder.analyze()
+generator = Generator(library_root, toolchain=tc)
+generator.gen_targets(anonymous=False, max_wrappers=10)
+generator.compile_targets(4)  # parallel workers
+```
+
+See `scripts/template-script.py` for a complete workflow example and `workshop/` for library-specific tutorials.

CONTRIBUTING.md

@@ -11,21 +11,21 @@
 ### Building the LLVM toolchain
 
 ```bash
-cd custom-llvm && ./prepare.sh
+cd build-llvm && ./prepare.sh
 cd ../build && ./build.sh
 ```
 
 ### Installing the Python package (development mode)
 
 ```bash
-cd src/python/futag-package
+cd futag-package
 pip install -e ".[test]"
 ```
 
 ### Running tests
 
 ```bash
-cd src/python/futag-package
+cd futag-package
 python -m pytest tests/ -v
 ```
 

README.en.md

@@ -42,7 +42,7 @@ graph TD
     end
 
     subgraph "Layer 1: Build Infrastructure"
-        D["custom-llvm / build.sh — Download and patch LLVM 14/18/19"]
+        D["build-llvm / build.sh — Download and patch LLVM 14/18/19"]
     end
 
     D -->|"futag-llvm toolchain"| E
@@ -106,7 +106,7 @@ Thank you for acknowledging the authors' work when you use FUTAG or report bugs
 
 # 3. Installation
 ## 3.1. Using the Docker container
-You can try building FUTAG using the provided Dockerfiles for Ubuntu: https://github.com/ispras/Futag/tree/main/product-tests/build-test
+You can try building FUTAG using the provided Dockerfiles for Ubuntu: https://github.com/ispras/Futag/tree/main/integration-tests/build-test
 
 ## 3.2. Using a prepackaged release
 - Download the latest release (for example, futag-llvm.3.0.1.tar.xz) from https://github.com/ispras/Futag/releases/tag/v3.0.1 and extract it. The tool will be installed to the futag-llvm directory.
@@ -145,12 +145,12 @@ On Ubuntu you may also need to install:
 ```bash
   ~$ git clone https://github.com/ispras/Futag
 ```
-- Prepare the "custom-llvm" directory by running the script:
+- Prepare the "build-llvm" directory by running the script:
 
 ```bash
-  ~/Futag/custom-llvm$ ./prepare.sh
+  ~/Futag/build-llvm$ ./prepare.sh
 ```
-This script creates the Futag/build directory and copies Futag/custom-llvm/build.sh into it.
+This script creates the Futag/build directory and copies Futag/build-llvm/build.sh into it.
 
 - Run the copied build script in "Futag/build":
 
@@ -176,45 +176,41 @@ This script creates the Futag/build directory and copies Futag/custom-llvm/build
 ```
 
 ### 4.1. Automatic generation of fuzzing wrappers when usage contexts are absent
-- Run the build, check and analysis when no usage contexts exist:
 
 ```python
 from futag.preprocessor import *
+from futag.generator import *
+from futag.toolchain import ToolchainConfig
 
 FUTAG_PATH = "/home/futag/Futag-tests/futag-llvm/"
 lib_path = "path/to/library/source/code"
+
+# --- Create toolchain ---
+tc = ToolchainConfig.from_futag_llvm(FUTAG_PATH)
+
+# --- Build and analyze the library ---
 build_test = Builder(
-    FUTAG_PATH,
     lib_path,
     clean=True, # remove all folders generated by FUTAG before building
     # intercept=True, # enable compilation with the "intercept" tool to analyze compile_command.json
     # processes=4, # number of build jobs
     # build_ex_params="--with-openssl --with-mhash" # extra params for library build
+    toolchain=tc,
 )
 build_test.auto_build()
 build_test.analyze()
-```
-
-- Generate and compile drivers:
-
-```python
-from futag.generator import *
-
-FUTAG_PATH = "/home/futag/Futag-tests/futag-llvm/"
-lib_path = "path/to/library/source/code"
 
+# --- Generate and compile drivers ---
 generator = Generator(
-    FUTAG_PATH, # path to the "futag-llvm" directory
     lib_path, # path to the directory containing the target software source code
-    # target_type=AFLPLUSPLUS, 
+    # target_type=AFLPLUSPLUS,
+    toolchain=tc,
 )
 
-# Generate fuzzing wrappers
 generator.gen_targets(
     anonymous=False, # option to generate wrappers for non-public functions
-    max_wrappers=10 # limit the number of generated wrappers per function
+    max_wrappers=10, # limit the number of generated wrappers per function
 )
-# Compile fuzz drivers
 generator.compile_targets(
     4, # number of build jobs
     # keep_failed=True, # keep failed targets
@@ -230,25 +226,27 @@ By default, successfully compiled fuzzing wrappers for target functions are plac
 
 ```python
 from futag.preprocessor import *
-from futag.generator import * 
-from futag.fuzzer import * 
+from futag.generator import *
+from futag.fuzzer import *
+from futag.toolchain import ToolchainConfig
 
 FUTAG_PATH = "/home/futag/Futag/futag-llvm"
 library_root = "json-c-json-c-0.16-20220414"
 consumer_root = "libstorj-1.0.3"
 consumber_builder = ConsumerBuilder(
-   FUTAG_PATH, # path to the "futag-llvm" directory
    library_root, # path to the directory with the tested library's source code
    consumer_root, # path to the directory with the consumer application's source code
+   toolchain=tc,
   #  clean=True,
   #  processes=16,
 )
 consumber_builder.auto_build()
 consumber_builder.analyze()
 
+tc = ToolchainConfig.from_futag_llvm(FUTAG_PATH)
 context_generator = ContextGenerator(
-    FUTAG_PATH, 
-    library_root, 
+    library_root,
+    toolchain=tc,
 )
 
 context_generator.gen_context() # generate fuzzing wrappers for contexts
@@ -258,13 +256,13 @@ context_generator.compile_targets( # compile generated fuzzing wrappers
 ```
 
 If multiple fuzzing wrappers are generated for a function, the target function's subdirectory will contain numbered subdirectories (appended to the function name).
-Python package documentation is available at: https://github.com/ispras/Futag/tree/main/src/python/futag-package
+Python package documentation is available at: https://github.com/ispras/Futag/tree/main/futag-package
 
-More information about using FUTAG is available at: https://github.com/ispras/Futag/blob/main/How-to-work-with-Futag.md
+More information about using FUTAG is available at: https://github.com/ispras/Futag/blob/main/docs/how-to-work-with-futag.md
 
-A template for run scripts can be found here: https://github.com/ispras/Futag/blob/main/src/python/template-script.py
+A template for run scripts can be found here: https://github.com/ispras/Futag/blob/main/scripts/template-script.py
 
-A test repository was created at https://github.com/thientc/Futag-tests to test FUTAG on various libraries (json-c, php, FreeImage, etc.). You can try testing using the Docker container at https://github.com/ispras/Futag/tree/main/product-tests/libraries-test.
+A test repository was created at https://github.com/thientc/Futag-tests to test FUTAG on various libraries (json-c, php, FreeImage, etc.). You can try testing using the Docker container at https://github.com/ispras/Futag/tree/main/integration-tests/libraries-test.
 
 ## Documentation