AI Testing Suite

Multi-Agent LangGraph Orchestration for Automated Testing

8 specialized AI agents analyze your project, generate tests, review quality, execute tests, scan for vulnerabilities, and document everything — fully automated.

Installation · Quick Start · Agents · CLI Modes · Security · Reports

Overview

AI Testing Suite is a LangGraph-based multi-agent pipeline that automatically tests Node.js/TypeScript projects. It analyzes project structure and code, plans a test strategy, generates tailored test files, reviews their quality, executes them, scans for security vulnerabilities, and creates comprehensive reports.

Scanner → Analyzer → Strategist → Writer → Reviewer ──→ Runner → Security → Reporter
                                              │              ↑
                                              └── (Retry) ───┘

Why AI Testing Suite?

Extremely easy to integrate — A single command is all it takes to integrate the suite into any existing project
100% project-tailored — Tests are generated based on deep code analysis, not from templates
8 specialized agents — Each agent has a clearly defined role in the pipeline
OWASP Top 10 + Zero-Day — Comprehensive security scanning with 100+ vulnerability patterns
Console-driven — Interactive CLI menu with 8 modes
Production-readiness assessment — Clear verdict on whether the project is production-ready

Project Structure

ai-testing-suite/
├── src/
│   ├── index.ts                    # CLI Entry Point
│   ├── cli.ts                      # Interactive Console Menu
│   ├── config.ts                   # Configuration Management
│   ├── setup.ts                    # Quick Setup for Existing Projects
│   ├── types.ts                    # Complete TypeScript Type System
│   │
│   ├── agents/                     # 8 Specialized Agents
│   │   ├── scanner.agent.ts        # Agent 1: Project Structure Scanner
│   │   ├── analyzer.agent.ts       # Agent 2: Deep Code Analysis
│   │   ├── strategist.agent.ts     # Agent 3: Test Strategy Planner
│   │   ├── writer.agent.ts         # Agent 4: Test Generator
│   │   ├── reviewer.agent.ts       # Agent 5: Test Quality Reviewer
│   │   ├── runner.agent.ts         # Agent 6: Test Executor
│   │   ├── security.agent.ts       # Agent 7: Security Auditor
│   │   └── reporter.agent.ts       # Agent 8: Report Generator
│   │
│   ├── graph/                      # LangGraph Orchestration
│   │   ├── workflow.ts             # Pipeline Definition & Execution
│   │   ├── nodes.ts                # Graph Nodes & Conditional Edges
│   │   └── state.ts                # State Channel Management
│   │
│   └── utils/                      # Utility Functions
│       ├── parser.ts               # Code Parser (Functions, Classes, Patterns)
│       ├── file-utils.ts           # File System Operations
│       └── logger.ts               # Colored Console Output
│
├── tests/                          # Generated Tests (Output)
│   ├── unit/                       # Unit Tests
│   ├── integration/                # Integration Tests
│   ├── e2e/                        # End-to-End Tests
│   ├── security/                   # Security Tests
│   └── performance/                # Performance Tests
│
├── reports/                        # Generated Reports (Output)
│   ├── report-*.json               # Machine-readable
│   ├── report-*.md                 # Markdown Report
│   ├── report-*.html               # Visual HTML Report
│   └── agent-log-*.md              # Agent Activity Log
│
├── package.json
├── tsconfig.json
└── .env.example                    # Configuration Template

Installation

Install via npm (recommended)

npm install --save-dev ai-testing-suite

Then run the setup wizard:

npx ai-test-setup

This will:

Create test directories (tests/unit, tests/integration, tests/e2e, tests/security, tests/performance)
Install the right dependencies (Vitest or Jest, Playwright, etc.)
Add npm scripts to your package.json
Create .env.example with configuration template

Quick Start in 3 Steps

# 1. Install
npm install --save-dev ai-testing-suite

# 2. Configure
cp .env.example .env
# Enter your API key (OpenAI or Anthropic) or use Ollama locally

# 3. Run
npx ai-test --mode full

Setup Flags

npx ai-test-setup              # Default: Vitest + Playwright
npx ai-test-setup --jest        # Use Jest instead of Vitest
npx ai-test-setup --no-playwright  # Skip Playwright installation

Alternative: Clone from GitHub

git clone https://github.com/zurd46/AI-Testing-Suite.git
cd AI-Testing-Suite
npm install
npm run setup -- /path/to/project

Quick Start

Interactive Mode (recommended)

npm start
# or
npx ts-node src/index.ts

Displays the interactive menu:

╔══════════════════════════════════════════════════════════════╗
║  AI TESTING SUITE                                          ║
║  Multi-Agent LangGraph Orchestration                        ║
╠══════════════════════════════════════════════════════════════╣
║  8 Agents | OWASP Top 10 | Zero-Day Scans | 100% Coverage  ║
╚══════════════════════════════════════════════════════════════╝

Select a mode:

  1) Full Pipeline     - Complete analysis, tests & security
  2) Analyze Only      - Project structure & code analysis
  3) Generate Tests    - Analysis + create & review tests
  4) Run Tests         - Execute existing tests
  5) Security Scan     - Security audit only
  6) Custom Pipeline   - Choose your own agent combination
  7) Generate Report   - Create report
  8) Configuration     - View/modify settings
  0) Exit

Direct Modes

# Full pipeline
npm run full -- --path /path/to/project

# Analysis only
npm run analyze -- --path /path/to/project

# Generate tests only
npm run generate -- --path /path/to/project

# Security scan only
npm run security -- --path /path/to/project

# Report only
npm run report -- --path /path/to/project

CLI Options

npx ts-node src/index.ts \
  --mode full \
  --path /path/to/project \
  --output ./tests \
  --reports ./reports \
  --provider openai \
  --model gpt-4o \
  --depth deep \
  --verbose

Option	Description	Default
`-m, --mode`	Mode: `full\|analyze\|generate\|run\|security\|report\|interactive`	`interactive`
`-p, --path`	Path to target project	`./`
`-o, --output`	Test output directory	`./tests`
`-r, --reports`	Report directory	`./reports`
`--provider`	LLM Provider: `openai\|anthropic\|openai_compatible\|ollama`	`openai`
`--model`	LLM Model	`gpt-4o`
`--runner`	Test runner: `vitest\|jest\|node`	`vitest` (auto-detected)
`--e2e`	E2E runner: `playwright\|supertest\|none`	`playwright` (auto-detected)
`--coverage`	Coverage tool: `v8\|c8\|istanbul`	`v8` (auto-detected)
`--depth`	Security scan depth: `basic\|standard\|deep`	`deep`
`-v, --verbose`	Verbose output	`true`
`--no-security`	Disable security scanning	-
`--no-zero-day`	Disable zero-day checks	-

The 8 Agents

Agent 1: Scanner

Scans the complete project structure

Detects framework (Next.js, NestJS, Express, React, Vue, Angular, ...)
Identifies entry points, config files, package manager
Counts files, lines, languages
Detects features (TypeScript, Prisma, GraphQL, WebSockets, Redis, ...)

Agent 2: Analyzer

Deep code analysis of every source file

Parses functions, classes, interfaces, variables
Detects API endpoints (Express, NestJS Decorators)
Detects database operations (Query, Insert, Update, Delete)
Builds dependency graph with circular dependency detection
Detects design patterns (Singleton, Factory, Observer, Middleware, ...)
Calculates cyclomatic complexity
Detects error handling patterns

Agent 3: Strategist

Plans the optimal test strategy

Creates unit test plans per function/class
Plans API integration tests per endpoint
Plans database integration tests
Plans E2E tests per resource
Plans security tests (OWASP-based)
Plans performance tests
Prioritizes by criticality and complexity

Generates test cases for each category: happy-path · edge-case · error-handling · boundary · null-undefined · type-safety · concurrency · security · performance · regression

Agent 4: Writer

Generates tailored test files

Writes unit tests with correct imports and mocks
Writes integration tests with Supertest
Writes E2E tests with Playwright (browser) or Supertest (API)
Writes security tests with real attack payloads
Writes performance tests (latency, concurrency, memory)
Generates runner-specific config (vitest.config.ts, jest.config.ts, playwright.config.ts)
Auto-adapts syntax per runner (vi.fn() vs jest.fn(), imports vs globals)
Creates directory structure automatically

Agent 5: Reviewer

Reviews generated tests for quality

Checks test structure (describe, it, beforeEach)
Counts and evaluates assertions
Checks test isolation (mock reset)
Checks edge case coverage
Checks error handling tests
Checks mock usage and spy assertions
Checks async patterns
Detects placeholder assertions
Calculates quality score (0-100)
Conditional Edge: If score < threshold, pipeline loops back to the Writer

Agent 6: Runner

Executes all tests and collects results

Supports Vitest (default, fastest), Jest, and Node.js built-in test runner
Runs Playwright for E2E browser tests
Runs tests grouped by type (Unit, Integration, E2E, Security, Performance)
Parses JSON output from each runner
Collects errors with test name, message, stack trace
Auto-detects installed test runner from target project
Dry run when runner is not installed
Calculates success rate

Agent 7: Security Auditor

Comprehensive security audit

Vulnerability Checks (100+ Patterns):

Category	Checks
SQL Injection	Template Literals, String Concatenation, Raw Queries
XSS	innerHTML, dangerouslySetInnerHTML, document.write, Reflected Input
Command Injection	exec(), execSync(), spawn() with user input
Path Traversal	readFile/writeFile with user input, path.join manipulation
Prototype Pollution	Object.assign, Spread, proto, constructor
SSRF	fetch/axios/got with user-controlled URL
CSRF	State-changing endpoints without token
ReDoS	User input in RegExp, nested quantifiers
Hardcoded Secrets	Passwords, API keys, private keys, connection strings
Insecure Randomness	Math.random() for tokens/keys
Broken Auth	JWT "none" algorithm, weak hashes, TLS bypass
Information Disclosure	Stack traces in responses, sensitive data in logs
Zero-Day Patterns	process.binding, dynamic Function(), Proxy manipulation, Buffer.allocUnsafe

Additionally:

Dependency vulnerability audit (known CVEs in npm packages)
OWASP Top 10 compliance check (A01-A10:2021)
Code smell detection (TODO, empty catch blocks, console.log in production)
Risk score calculation (0-100)

Agent 8: Reporter

Creates comprehensive documentation

JSON Report — Machine-readable, for CI/CD integration
Markdown Report — Human-readable, for code reviews
HTML Report — Visual report with charts and colors
Agent Log — Chronological log of all agent actions
Production-readiness assessment — YES/NO with reasoning
Recommendations — Concrete steps for improvement

CLI Modes

Mode	Description	Agents
`full`	Complete pipeline	All 8
`analyze`	Analysis only	Scanner → Analyzer
`generate`	Generate tests	Scanner → Analyzer → Strategist → Writer → Reviewer
`run`	Run tests	Scanner → Analyzer → Strategist → Writer → Runner
`security`	Security scan	Scanner → Analyzer → Security → Reporter
`report`	Generate report	Reporter
`interactive`	Interactive menu	Selectable
Custom	Custom combination	Freely selectable (e.g. 1,2,7)

Supported Test Runners

AI Testing Suite auto-detects the test runner from your project. You can also override via CLI or .env.

Runner	Type	Speed	Best For
Vitest (default)	Unit/Integration	Fastest	Modern TypeScript projects, ESM
Jest	Unit/Integration	Fast	Existing Jest projects, CJS
Node.js test runner	Unit	Fast	Zero-dependency, Node >= 18
Playwright (default)	E2E/Browser	Medium	Frontend/Fullstack, cross-browser
Supertest	E2E/API	Fast	Backend API testing

Coverage Tools

Tool	Speed	Integration
v8 (default)	Fastest	Native V8 engine coverage
c8	Fast	V8 coverage via CLI wrapper
istanbul	Standard	Traditional instrumentation

Auto-Detection

The Scanner agent automatically detects your project's existing test tooling:

vitest in package.json    → uses Vitest
jest/ts-jest              → uses Jest
@playwright/test          → uses Playwright for E2E
supertest                 → uses Supertest for E2E
c8                        → uses c8 for coverage
nyc/istanbul              → uses Istanbul for coverage

Override with CLI: npx ai-test --mode full --runner jest --e2e playwright --coverage v8

Security Scanning

OWASP Top 10 (2021) Coverage

#	Category	Status
A01	Broken Access Control	Covered
A02	Cryptographic Failures	Covered
A03	Injection	Covered
A04	Insecure Design	Covered
A05	Security Misconfiguration	Covered
A06	Vulnerable Components	Covered
A07	Auth Failures	Covered
A08	Software Integrity	Covered
A09	Logging Failures	Covered
A10	SSRF	Covered

Zero-Day Pattern Detection

Detection of patterns that may indicate potential zero-day exploits:

process.binding() — Access to internal Node.js APIs
Function() with user input — Dynamic code execution
Proxy with user input — Handler manipulation
Reflect API with user input — Metaprogramming attacks
Dynamic import() with user input — Module injection
constructor bracket notation — Sandbox escape
Buffer.allocUnsafe() — Uninitialized memory (memory leak)
globalThis/global[] — Global scope manipulation

Reports

HTML Report

The HTML report provides a visual overview with:

Overall status (Production-ready YES/NO)
Score cards (Tests, Security, Quality)
Vulnerability table with severity colors
OWASP compliance status
Recommendations

Markdown Report

Contains:

Summary with metrics
Project structure details
Code analysis results
Generated tests by type
Test results with errors
Security report with all vulnerabilities
OWASP status
Recommendations

JSON Report

Complete, machine-readable report for CI/CD integration:

{
  "projectName": "my-app",
  "timestamp": "2025-01-15T10:30:00Z",
  "summary": {
    "totalTests": 142,
    "totalPassed": 138,
    "totalFailed": 4,
    "securityScore": 85,
    "qualityScore": 92,
    "productionReady": false,
    "recommendations": [...]
  },
  "security": {
    "vulnerabilities": [...],
    "owaspChecks": [...],
    "overallRisk": "medium"
  }
}

Configuration

Environment Variables (.env)

# LLM Provider
OPENAI_API_KEY=sk-...           # OpenAI API Key
ANTHROPIC_API_KEY=sk-ant-...    # Or Anthropic API Key
LLM_PROVIDER=openai             # openai | anthropic | openai_compatible | ollama
LLM_MODEL=gpt-4o               # Model name (ollama default: llama3.2)
LLM_TEMPERATURE=0.1            # Low for consistent tests
LLM_MAX_TOKENS=4096

# Paths
TARGET_PROJECT_PATH=./          # Target project
TESTS_OUTPUT_DIR=./testing              # Test output
REPORTS_OUTPUT_DIR=./testing/reports    # Report output

# Test Runner Configuration
TEST_RUNNER=vitest              # vitest (default, fastest) | jest | node
E2E_RUNNER=playwright           # playwright (default) | supertest | none
COVERAGE_TOOL=v8                # v8 (default, native) | c8 | istanbul

# Security
SECURITY_SCAN_DEPTH=deep       # basic | standard | deep
CHECK_ZERO_DAY=true
CHECK_OWASP_TOP_10=true
CHECK_DEPENDENCIES=true

# Agent
MAX_AGENT_ITERATIONS=10
AGENT_VERBOSE=true

Supported Frameworks

Framework	Type	Detection
Next.js	Fullstack	Automatic
NestJS	Backend	Automatic
Express	Backend	Automatic
Fastify	Backend	Automatic
Koa	Backend	Automatic
Hapi	Backend	Automatic
React	Frontend	Automatic
Vue	Frontend	Automatic
Angular	Frontend	Automatic
Svelte	Frontend	Automatic
Nuxt	Fullstack	Automatic
Remix	Fullstack	Automatic
Electron	Fullstack	Automatic
Node.js	Library/CLI	Fallback

Detected Features

TypeScript · Prisma · TypeORM · Mongoose · Sequelize · GraphQL · WebSockets · Redis · Passport · JWT · bcrypt · Helmet · CORS · Joi/Zod/Yup · Winston/Pino · Bull/BullMQ · Swagger/OpenAPI · Vitest · Playwright · Stryker · c8 · ESLint Security · PactumJS

Prerequisites

Node.js >= 18.0.0
npm, yarn, or pnpm
TypeScript (recommended, JavaScript is also supported)
API Key for OpenAI or Anthropic (for LLM-powered features), or Ollama for local inference (no API key needed)

Example Output

======================================================================
  AI TESTING SUITE - FULL MODE
======================================================================

-- Project: /home/user/my-app
-- Pipeline: scanner -> analyzer -> strategist -> writer -> reviewer -> runner -> security -> reporter
-- 8 agents will be executed

>> [SCAN] Agent started...
   [SCAN] 47 files found
   [SCAN] 12 directories found
   [SCAN] Framework detected: Express (backend)
OK [SCAN] Agent completed (0.3s)

>> [ANALYZE] Agent started...
   [ANALYZE] 23 modules analyzed
   [ANALYZE] 45 functions, 8 classes
   [ANALYZE] 12 API endpoints found
OK [ANALYZE] Agent completed (1.2s)

>> [STRATEGY] Agent started...
   Test Type       | Plans | Test Cases
   Unit Tests      |    18 |         72
   Integration     |    14 |         42
   E2E Tests       |     4 |          8
   Security Tests  |     2 |         16
   Performance     |     1 |          3
   TOTAL           |    39 |        141
OK [STRATEGY] Agent completed (0.8s)

>> [WRITE] Agent started...
   [WRITE] 18 unit test files written
   [WRITE] 14 integration test files written
   [WRITE] 4 E2E test files written
OK [WRITE] Agent completed (2.1s)

>> [REVIEW] Agent started...
   36/39 passed (Score: 87.3)
OK [REVIEW] Agent completed (0.5s)

>> [RUN] Agent started...
   PASS auth.test.ts (124ms)
   PASS users.test.ts (89ms)
   FAIL payment.test.ts (203ms)
OK [RUN] Agent completed (12.4s)

>> [SECURITY] Agent started...
   [CRITICAL] hardcoded-credentials: Hardcoded API key (config.ts)
   [HIGH]     sql-injection: Raw SQL with Concatenation (users.service.ts)
   [MEDIUM]   xss: innerHTML assignment (dashboard.tsx)
   Security Score: 72/100
OK [SECURITY] Agent completed (1.8s)

>> [REPORT] Agent started...
   Reports saved to: ./reports
OK [REPORT] Agent completed (0.4s)

======================================================================
  TEST RESULTS
======================================================================
  Project:            my-app
  Framework:          Express (backend)
  Total Tests:        141
  Passed:             137
  Failed:             4
  Security Score:     72/100
  Quality Score:      87/100

!! NOT PRODUCTION-READY - Please review recommendations:
??   -> Fix 4 failing tests
??   -> Address security vulnerabilities
??   -> Fix 1 critical vulnerability immediately!

License

MIT

Built with LangGraph, TypeScript, and the power of 8 AI agents.

Name		Name	Last commit message	Last commit date
Latest commit History 30 Commits
src		src
tests		tests
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
jest.config.ts		jest.config.ts
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json

zurd46/AI-Testing-Suite

Folders and files

Latest commit

History

Repository files navigation

AI Testing Suite

Multi-Agent LangGraph Orchestration for Automated Testing

Overview

Why AI Testing Suite?

Project Structure

Installation

Install via npm (recommended)

Quick Start in 3 Steps

Setup Flags

Alternative: Clone from GitHub

Quick Start

Interactive Mode (recommended)

Direct Modes

CLI Options

The 8 Agents

Agent 1: Scanner

Agent 2: Analyzer

Agent 3: Strategist

Agent 4: Writer

Agent 5: Reviewer

Agent 6: Runner

Agent 7: Security Auditor

Agent 8: Reporter

CLI Modes

Supported Test Runners

Coverage Tools

Auto-Detection

Security Scanning

OWASP Top 10 (2021) Coverage

Zero-Day Pattern Detection

Reports

HTML Report

Markdown Report

JSON Report

Configuration

Environment Variables (.env)

Supported Frameworks

Detected Features

Prerequisites

Example Output

License

About

Topics

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Uh oh!

Contributors 2

Uh oh!

Languages

Packages