Inventory-Accounting-AI

A computer vision + RPA agent that automates invoice creation and record management in legacy accounting software — without needing API access.

Built to solve a real problem: older dealership management systems (like Charter Software's ASPEN) have no APIs. This agent uses Claude Vision to read the screen and PyAutoGUI to control the UI — turning a 20-minute manual process into a 2-minute automated one.

---

The problem

Legacy accounting software from the 2000s-2010s has no REST API. The only way to get data in or out is through the GUI — clicking buttons, filling forms, navigating menus. That's what humans were doing, one invoice at a time, all day.

This agent does it instead.

---

What it does

Task	Time before	Time after
Create invoice from CSV	20 min manual	2 min automated
Search and retrieve records	5 min	30 sec
Create work orders	15 min	90 sec
User management (add/edit)	10 min	60 sec

Four automation modules:

1. Invoice creation — reads a CSV of invoice data, navigates to the correct screen, fills each field, saves
2. Work order creation — creates service work orders linked to customer records
3. Record search — searches the system and extracts structured data from the screen
4. User management — adds and edits user accounts in the system

---

Architecture

┌─────────────────────────────────────────────────────────┐
│                   FastAPI controller                     │
│              Receives automation commands                │
└───────────────────────────┬─────────────────────────────┘
                            │
                ┌───────────┴───────────┐
                ▼                       ▼
        Claude Vision              PyAutoGUI
        (reads the screen,         (clicks, types,
         extracts data,             navigates menus)
         decides next action)
                │                       │
                └───────────┬───────────┘
                            ▼
                    ASPEN DMS (legacy software)
                    Running on the same machine

The agent loop:

1. Capture screenshot
2. Send to Claude Vision with task context
3. Claude decides: "click this button" / "type this text" / "navigate here" / "done"
4. PyAutoGUI executes the action
5. Capture new screenshot → back to step 2
6. Repeat until task complete or error detected

---

Tech stack

Layer	Tech
Vision	Claude Vision (Anthropic SDK)
UI automation	PyAutoGUI
API	FastAPI
Orchestration	Python async
Logging	Python logging + JSON audit trail

---

Quickstart

git clone https://github.com/MuhammadFarid1990/Inventory-Accounting-AI
cd Inventory-Accounting-AI
pip install -r requirements.txt
cp .env.example .env
# Add your ANTHROPIC_API_KEY

# Start the automation server
python -m uvicorn api.main:app --reload

# Run invoice automation from CSV
curl -X POST http://localhost:8000/automate/invoices \
  -F "file=@invoices.csv"

Note: Requires the target software to be running and visible on screen.

---

Project structure

Inventory-Accounting-AI/
├── agents/
│   ├── base_agent.py     # Core vision-action loop
│   ├── invoice_agent.py  # Invoice creation automation
│   ├── workorder_agent.py
│   └── search_agent.py
├── vision/
│   ├── capture.py        # Screenshot capture + preprocessing
│   └── extractor.py      # Claude Vision structured data extraction
├── rpa/
│   ├── controller.py     # PyAutoGUI wrapper with safety checks
│   └── actions.py        # High-level UI action library
├── api/
│   └── main.py           # FastAPI endpoints
├── .env.example
├── requirements.txt
└── README.md

---

Safety design

• Dry-run mode — logs all planned actions without executing (default for testing)
• Human-in-the-loop — optional confirmation step before destructive actions
• Audit trail — every action is logged with timestamp, screenshot, and result
• Bounds checking — PyAutoGUI failsafe (move mouse to corner to abort)
• Rate limiting — configurable delay between actions to avoid overwhelming the UI

---

About the builder

Muhammad Farid — MS Business Analytics & AI @ UT Dallas.

Portfolio · GitHub

Built with Claude.