Hybrid 916-feature artwork detector with two-stage anime veto

[wfproc]

Negate Artwork Detection — Research & Experiments

negate project — darkshapes
April 6, 2026 (verified end-to-end)

---

How to Read This Document

What experiments were run: Each section documents a specific test with exact datasets, sample sizes, and results. Links to HuggingFace datasets and test scripts are provided throughout.

What solutions we moved toward: We started with 49 handcrafted features (Li & Stamp 2025) and iterated through feature expansion, learned features (ConvNeXt), ensemble methods, threshold calibration, and finally a two-stage architecture with an anime-specific veto model.

Why these directions: The existing negate codebase uses VIT/VAE feature extraction + PCA + XGBoost, achieving ~63% accuracy. We needed to:

1. Improve detection accuracy on artwork specifically (not photos — proved domain transfer fails at 49.7%)
2. Cover modern 2025 generators that didn't exist when the original pipeline was built
3. Minimize false positives (the most dangerous failure mode — accusing a real artist of using AI)
4. Handle anime/illustration content where AI art and real art share very similar visual properties

How this addresses issues in existing code:

• The existing VIT/VAE pipeline requires GPU and downloads multi-GB models. Our 148 handcrafted features are CPU-only.
• The existing pipeline trains on limited generators (SD, SDXL). We train on 15+ sources including Flux, Gemini, Seedream, and CivitAI community models.
• The existing pipeline has no false positive analysis. We tested FP on WikiArt, tellif, anime, and photos with honest reporting.
• The existing pipeline outputs binary GNE/SYN. We output calibrated probability + three-class (GENUINE/UNCERTAIN/SYNTHETIC) with an anime veto stage.

Where does the data come from? All training and test data comes from publicly available HuggingFace datasets or the public CivitAI API. Every source is linked below with its license. Genuine (real art) sources are WikiArt (public domain classical art via ImagiNet, CC BY 4.0), tellif (curated real images, HF public), Hemg (HF public), and latentcat/animesfw (anime illustration archive, HF public). No private or proprietary data is used. CivitAI images are scraped from the public API and not redistributed — users should scrape their own.

---

Two-Stage Architecture

Image → 148 handcrafted features (CPU, ~1 img/s)
      → 768 frozen ConvNeXt-Tiny features (CPU, ~28 img/s)
      → 3-model ensemble (LightGBM 40% + SVM 50% + RF 10%)
      → probability 0-1
      → if prob >= 0.65: SYNTHETIC candidate
          → fine-tuned ConvNeXt anime veto
          → if veto says REAL → override to UNCERTAIN
      → if prob < 0.35: GENUINE
      → else: UNCERTAIN

The anime veto is a ConvNeXt-Tiny with stage 3 fine-tuned on anime real vs AI data. It only acts on images the main ensemble flags as SYNTHETIC — it can downgrade to UNCERTAIN but never upgrade to SYNTHETIC. This eliminates most false positives on anime/illustration content without affecting detection of AI art.

Code: negate/extract/feature_artwork.py (148 HC), negate/extract/feature_learned.py (768 ConvNeXt), models/convnext_anime_finetuned.pt (anime veto)

---

Feature Architecture

Handcrafted Features (148 dimensions)

All features operate on a 255x255 resized image, CPU-only. Grouped by source:

Group	Count	Source / Rationale
Brightness + Color (RGB/HSV stats)	25	Li & Stamp (arXiv:2504.07078, 2025)
Texture — GLCM (single-angle)	4	Li & Stamp
Texture — LBP	2	Li & Stamp
Shape — HOG + Canny edges	6	Li & Stamp
Noise (entropy + SNR)	2	Li & Stamp
Frequency — FFT/DCT spectral	10	Custom (spectral centroid, radial bands, phase)
Enhanced GLCM (multi-angle/distance)	14	Nirob et al. (arXiv:2601.19262, 2026)
Mid-band frequency	4	Inspired by FIRE (CVPR 2025) — diffusion models fail at mid-band
Patch consistency	6	Inspired by CINEMAE (2025) — AI has inconsistent patch-to-context
Multi-scale LBP (3 scales)	8	Nirob et al.
Gabor filter bank (4 freq × 4 orient)	18	Texture at multiple scales/orientations
Wavelet packets (2-level Haar)	12	Subband energy statistics
Edge co-occurrence (direction GLCM)	8	Edge direction relationships
Fractal dimension	2	Box-counting on grayscale + edges
Noise residual autocorrelation	5	Canvas/generator periodicity detection
Stroke edge roughness	4	Art-specific: brush edge irregularity
Color gradient curvature	4	Art-specific: subtractive paint mixing signature
Patch self-similarity	4	AI tiling/repetition detection
Extended HOG (2 scales)	6	Fine/coarse gradient comparison
JPEG ghost detection	4	Recompression response at Q50/70/90
Linework analysis	8	Anime-specific: stroke thickness, edge sharpness, medium consistency

Removed (hurt accuracy): Color coherence vectors (-0.8pp), cross-subband wavelet correlation (-1.0pp).

Learned Features (768 dimensions)

Frozen ConvNeXt-Tiny (ImageNet-22K pretrained, timm library). Penultimate layer = 768-dim embedding.

• Not CLIP (proved +9.1pp bias on CLIP-based generators)
• Not DINOv2 (tested: 77.8%, worse than handcrafted)
• CPU: ~28 img/s

Feature Importance

On 6,344 training images (5-fold CV): 82.6% learned / 17.4% handcrafted

Top features: high_to_mid_ratio (mid-band freq), mslbp_s3_var (coarse LBP), hsv_entropy, jpeg_ghost_q90_rmse, wavelet_hh_energy_ratio, blend_saturation_dip (paint mixing).

Ensemble

• LightGBM (40%): 500 trees, lr 0.05, depth 8, 63 leaves
• SVM RBF (50%): StandardScaler + Platt scaling
• Random Forest (10%): 300 trees, depth 12

Thresholds: GENUINE < 0.35 | UNCERTAIN | SYNTHETIC >= 0.65. Selected from probability distribution analysis — real art max prob = 0.494, safety margin of 0.156 to SYNTHETIC threshold.

---

Verified Results (April 6, 2026)

All numbers from a single end-to-end verification run.

Detection — Large n (OpenFake held-out parquets 5-7)

Generator	n	Detection	Source
HiDream i1	100	90.0%	ComplexDataLab/OpenFake
Ideogram 3.0	100	89.0%	OpenFake
SD 3.5	100	78.0%	OpenFake
Midjourney 6	100	70.0%	OpenFake

Detection — Dedicated Datasets (n=200)

Generator	n	Detection	Source
Recraft v3	200	100.0%	Rapidata/Recraft-v3-24-7-25_t2i_human_preference
Flux 1.1 Pro	200	93.0%	ComplexDataLab/OpenFake

Detection — 2025 Blind Generators (tellif, n=9-20)

Small n — directional only (±33% CI at n=9).

Generator	n	Detection	Source
Imagen 4 Preview	9	100%	tellif/ai_vs_real_image_semantically_similar
Seedream v4	20	95.0%	tellif
Hunyuan v2.1	11	90.9%	tellif
HiDream i1	9	88.9%	tellif
Ideogram v2	9	88.9%	tellif
SD 3.5 Large	9	88.9%	tellif
Qwen Image	12	83.3%	tellif
Gemini 2.5 Flash	18	77.8%	tellif
Recraft v3	9	66.7%	tellif (noisy; n=200 Rapidata = 100%)
Wan v2.2	8	62.5%	tellif
Flux Pro v1.1	9	55.6%	tellif (noisy; n=200 OpenFake = 93%)

False Positive Rates

Content	FP Rate	n	Source	Method
Classical art	0.0%	500	WikiArt via ImagiNet	Main ensemble
High-quality real art	0.0%	122	tellif/...	Two-stage
Anime/illustration	2.0%	100	latentcat/animesfw	Two-stage (anime veto)
Photos	37% (out of scope)	100	OpenFake COCO	Main ensemble

Platform Robustness (simulated)

Platform	Detection	FP
Clean	97.1%	0.0%
Twitter/X (JPEG Q85, max 4096px)	96.1%	0.0%
Discord (JPEG Q80, 4MP limit)	96.1%	0.0%
Tumblr (JPEG Q65, max 1280px)	94.1%	0.0%
Instagram (JPEG Q75, 1080px, sharpen)	93.1%	0.0%

Large-Scale Validation

Test	Accuracy	n
5-fold CV (current config)	94.75% ± 0.18%	6,344
Hemg art-vs-art	92.3% ± 0.6%	10,000

Perturbation Ablation (HC vs ConvNeXt)

Perturbation	HC Only	Learned Only	Combined
Clean	98.5%	95.5%	99.5%
JPEG Q50	97.5%	90.5%	99.0%
Noise σ=10	76.0%	82.5%	88.0%
Blur r=2	87.5%	92.0%	95.0%
Resize 128→orig	66.5%	90.0%	89.5%

HC features fragile under noise/resize. ConvNeXt robust. Combined is best.

UNCERTAIN Calibration

	DETECTED	UNCERTAIN	MISSED
All AI (tellif)	77.2%	22.8%	0.0%
Real art (tellif)	0% FP	1.6%	98.4%

No AI image ever classified as GENUINE.

CLIP Bias (Proven)

Generator	Uses CLIP?	Handcrafted	CLIP	Advantage
SD 2.1	Yes	86.5%	96.1%	+9.6pp
SDXL	Yes	93.5%	99.0%	+5.5pp
SD 3	Yes	85.4%	97.5%	+12.1pp
DALL-E 3	No	98.7%	98.2%	-0.5pp

Source: Rajarshi-Roy-research/Defactify_Image_Dataset (96K images)

Probability Calibration

Range	Actual AI Rate	Label
0-10%	0.4%	Very likely genuine
10-35%	17-25%	Probably genuine
35-65%	37-63%	Uncertain
65-90%	63-91%	Probably AI-generated
90-100%	99.0%	Very likely AI-generated

---

Training Data

Total: 6,344 balanced (3,172 real + 3,172 synthetic)

Real (Genuine) Sources

Source	Count	Type	License	HF Repo
WikiArt	1,500	Classical paintings	CC BY 4.0	delyanboychev/imaginet (extracted wikiart/)
tellif real	122	Curated hard cases	HF public	tellif/ai_vs_real_image_semantically_similar
Hemg real	800	Mixed real art	HF public	Hemg/AI-Generated-vs-Real-Images-Datasets
Anime SFW	2,000	Human anime/illustration	HF public	latentcat/animesfw

All genuine sources are from established public archives. WikiArt contains public domain classical art from museum collections. latentcat/animesfw sources from anime illustration archives with community moderation. No images were generated or fabricated for the genuine class.

Synthetic (AI-Generated) Sources

Source	Count	Generator(s)	License	HF Repo
ImagiNet paintings	500	SDXL, SD, DALL-E 3, MJ, AnimagineXL	CC BY 4.0	delyanboychev/imaginet
Seedream 4.5	200	ByteDance 2025	HF public	ash12321/seedream-4.5-generated-2k
Nano Banana Pro	200	Gemini 2.5 Flash	HF public	exdysa/nano-banana-pro-generated-1k-clone
Flux Dev	300	BFL Flux Dev 2024	Apache 2.0	LukasT9/Flux-1-Dev-Images-1k
Flux Schnell	300	BFL Flux Schnell 2024	Apache 2.0	LukasT9/Flux-1-Schnell-Images-1k
Bitmind Nano Banana	500	Gemini 2.5 Flash	MIT	bitmind/nano-banana
CivitAI Flux	293	Flux (community)	Public API	Scraped, not redistributed
CivitAI SDXL	293	SDXL (community)	Public API	Scraped
CivitAI Pony	290	Pony Diffusion	Public API	Scraped
CivitAI Illustrious	296	Illustrious	Public API	Scraped

CivitAI Scraped Data

Scraped via public CivitAI REST API (/api/v1/images), filtered by tag, sorted newest. Rate-limited. All images publicly posted by users. Not published as a dataset — users should scrape their own or use the HuggingFace datasets above.

Additional scrapes (used for testing, not primary training): sd3 (500), sd35 (500), recraft (500), gemini (500). Local path: .datasets/civitai/.

Test-Only Data

Source	Purpose	HF Repo
tellif (245 images, 11 generators)	Blind 2025 SOTA	tellif/ai_vs_real_image_semantically_similar
OpenFake (963K, 25+ generators)	Large-n validation	ComplexDataLab/OpenFake
Rapidata Recraft (131K)	Recraft validation	Rapidata/Recraft-v3-24-7-25_t2i_human_preference
Defactify (96K)	CLIP bias proof	Rajarshi-Roy-research/Defactify_Image_Dataset
Hemg (152K)	Large-scale art-vs-art	Hemg/AI-Generated-vs-Real-Images-Datasets
latentcat animesfw (3.97M)	Anime FP testing	latentcat/animesfw

---

Dead Ends

Approach	Result	Lesson
EXIF metadata	N/A	Irrelevant for illustrations; stripped by platforms
CLIP embeddings	89-99% biased	Detects CLIP fingerprint, not AI artifacts
DINOv2 embeddings	77.8%	Worse than handcrafted; wrong pretraining objective
One-class anomaly detection	60% F1	Features too sparse for density modeling
EfficientNet-B3 backbone	No improvement	Redundant with ConvNeXt
CivitAI weak-gen training data	Performance WORSE	Generalization trap: learned CivitAI aesthetic
Perturbation-augmented training	±11pp noise	n=9 makes comparison unreliable
Linework features (8)	d=0.72 alone, 0pp in pipeline	Drowned out by 768 ConvNeXt features
Anime oversampling (3x)	Same 7.5% FP	Not a data quantity problem
Anime-specific classifier	33.5% FP	Worse than main classifier
Dual-backbone (frozen+finetuned)	1% anime but 78% classical FP	Fine-tuned features dominated
Color coherence features	-0.8pp	Hurt accuracy
Cross-subband wavelet features	-1.0pp	Hurt accuracy
ConvNeXt perturbation comparison	d=0.78 but inconsistent direction	2025 generators more stable than real art

Anime Side-Quest

The generalization trap: training on WikiArt (classical) as real + CivitAI anime AI as fake taught the model "anime = AI."

Attempt	Anime FP	Method
No anime in training	78.5%	Baseline
800 real anime	9.0%	latentcat/animesfw
2000 real anime	7.5%	More data
3x oversampling	7.5%	Diminishing returns
Linework features	8.0%	Strong alone (d=0.72) but drowned in pipeline
Fine-tuned ConvNeXt alone	1.0% anime but 78% classical FP	Over-specialized
Two-stage veto	2.0%	Main ensemble + fine-tuned veto on SYNTHETIC calls

---

Scope & Limitations

1. Artwork only. 37% FP on photos. Do not use on photographs.
2. Anime FP ~2%. Two-stage veto handles most cases. Some smooth digital art still overlaps with AI in feature space.
3. Heavy perturbation breaks HC features. Noise σ>=15, resize to 64px, blur r>=3.
4. Small tellif n. Per-generator numbers at n=9 have ±33% CI. OpenFake n=100 numbers are more reliable.
5. No adversarial hardening. Training distribution is public.
6. Style vs artifact risk. Smooth digital art may share features with AI art.
7. Moving target. New generators require retraining.
8. Not for accusations. Use for screening/triage. UNCERTAIN class exists for ambiguous cases.

Deployment Notes

• UNCERTAIN handling: 22.8% of AI images go to UNCERTAIN. Production needs human review queue.
• Platform validation: Test on actual platform pipelines before claiming accuracy.
• Retraining: Monthly retraining on fresh generator outputs (CivitAI API or new HF datasets).
• Probability output: Well-calibrated at extremes (0.4% error below 10%, 99% accuracy above 90%). Show percentage to users.

---

Reproduce

uv run python tests/extract_cache.py           # extract features (pause/resume)
uv run python tests/extract_cache.py --train    # train + test from cache
uv run python tests/test_hybrid_modern.py       # full hybrid test
uv run python tests/finetune_anime.py           # train anime veto
uv run python tests/test_adversarial_robustness.py
uv run python tests/test_clip_bias_defactify.py
uv run python tests/test_imaginet_paintings.py

---

Promising Directions & Blockers

What seems promising

1. AnimeDL-2M's DWT+DCT mixed architecture (arXiv:2504.11015) — purpose-built for anime, achieves 99.98%. Would require GPU training but directly solves the anime FP problem.
2. CivitAI scraping pipeline — automated retraining on latest generator outputs. The infrastructure for staying current matters as much as the model.
3. Full backbone fine-tuning — our stage-3-only fine-tune dropped anime FP from 7.5% to 2%. Full fine-tuning on diverse art data could improve further.
4. Reconstruction-based detection (AEROBLADE/RIGID style) — training-free, generalizes to unseen generators. We found ConvNeXt perturbation sensitivity exists (d=0.78) but direction is inconsistent across generators. A better reconstruction metric might work.
5. OpenFake dataset (963K, 25+ generators) — enormous resource for diversifying training. Has SD 3.5, MJ6/7, Ideogram, HiDream, Imagen 4, GPT Image 1.

Current blockers

1. No public datasets for newest generators (Gemini 3 Pro, Nano Banana 2, Midjourney v7/v8, Flux 2). These require API access to generate samples.
2. Anime FP at 2% is a feature-space limit. Smooth anime art and anime-style AI share the same spectral/texture properties. Without full fine-tuning or anime-specific architecture, this is the floor.
3. MJ6 at 70% on held-out data. Midjourney images are diverse and stylistically close to real art. More MJ training data didn't help — the features themselves may not capture MJ-specific artifacts.
4. CPU-only constraint. Full fine-tuning and reconstruction-based methods need GPU training. The current approach is CPU-inference but training was done on CPU too (slow but works).

---

Fuel

During this research (Mar 26-April 6, 2026), the following were consumed:

• 3x Monster Energy Zero Ultra
• 5x Monster Energy Ultra Peachy Keen
• 6x Monster Energy Ultra Paradise

---

References

1. Li & Stamp, "Detecting AI-generated Artwork", arXiv:2504.07078, 2025
2. Nirob et al., "Handcrafted Feature Fusion", arXiv:2601.19262, 2026
3. FIRE, "Frequency-Guided Reconstruction Error", CVPR 2025
4. CINEMAE, "Contextual Anomaly Estimation in MAE", 2025
5. Community Forensics, "Using Thousands of Generators", CVPR 2025
6. AnimeDL-2M, "AI-Generated Anime Detection", arXiv:2504.11015, 2025
7. OpenFake, arXiv:2509.09495, CC-BY-SA-4.0
8. "Organic or Diffused", arXiv:2402.03214, 2024

[exdysa]

Anime Side-Quest

[exdysa]

moved to #57 , previous pruning merge conflicting