LLM Agent for Drone Control

Next-generation AI-powered drone control. A fully autonomous drone system integrating LLMs, Computer Vision, and PX4 Autopilot.

🔹 Natural Language Command Execution 🔹 Real-time Sensor Integration 🔹 Simulation & Physical Deployment Support

---

📜 Table of Contents

🔍 Click to Expand

1. 🌟 Project Overview
2. 📐 System Architecture
3. 🛠️ Installation Guide
4. 🚀 Directions to Run the Project
5. 🎯 Expected Results
6. 🔍 Troubleshooting
7. 🎥 Demo Video
8. 👨‍💻 Contributors
9. 📜 License

---

🌟 Project Overview

This project focuses on developing a natural language-controlled drone system that minimizes human intervention. The system allows users to command a PX4 drone via natural language prompts, integrating cutting-edge technologies:

• 🚀 Large Language Models (LLMs) for interpreting user commands.
• 🎮 NVIDIA ISAAC SIM for sensor simulation.
• 🖼️ Computer Vision for object detection and depth estimation.
• 🔗 MAVLink for drone communication.
• ✈️ PX4-Autopilot for controlling drone actions.

The system translates natural language instructions into mission commands that enable autonomous drone operation, offering flexibility and scalability for real-world applications.

---

📐 System Architecture

The system architecture involves several interconnected components working seamlessly together:

• User Input Layer: Receives natural language commands from the user.
• LLM Processing Unit: Interprets and translates commands into actionable tasks.
• Data Integration Module: Integrates sensor and vision data into the command processing pipeline.
• Execution Layer: Converts processed commands into PX4-compatible instructions and executes them.
• Feedback Loop: Collects and processes data from sensors to refine ongoing operations.

graph LR;
  A[User Command] -->|Processed by LLM| B[LLM Module];
  B -->|Generates Tasks| C[Data Integration];
  C -->|Sends Instructions| D[PX4 Controller];
  D -->|Executes Drone Actions| E[Flight Feedback Loop];

Loading

---

🛠️ Installation Guide

1. Clone the Repository

git clone https://github.com/Bapiggott/IsaacDroneControl.git
cd IsaacDroneControl

2. Install Dependencies

Ensure Python is installed, then run:

pip install -r requirements.txt

3. Install Omniverse Launcher

Download the Omniverse Launcher:

wget https://install.launcher.omniverse.nvidia.com/installers/omniverse-launcher-linux.AppImage
chmod +x omniverse-launcher-linux.AppImage
./omniverse-launcher-linux.AppImage

4. Install Required Software

Software	Description	Installation Guide
ROS 2 Humble	🤖 Seamless robot operating system integration.	Install Here
NVIDIA ISAAC SIM	🎮 Simulate realistic drone environments.	Install Here
Pegasus Simulator	✈️ High-fidelity drone flight simulations.	Install Here

4. Verify Setup

Confirm all dependencies and software are correctly installed.

---

🚀 Directions to Run the Project

Command-Line Arguments

The system can be run with the following options to control specific components:

Argument	Short	Description	Default
--llm_model_name	-l	Name of the language model	deepseek-r1:8b
--vlm_model_name	-v	Name of the vision model	llama3.2-vision
--interface_port	-p	Port on which the web interface runs	http://localhost:5000
--vlm_api_url	-va	API endpoint for the VLM	http://localhost:8889
--llm_api_url	-la	API endpoint for the Ollama LLM	http://localhost:8888
--components	-c	Choose which components to start (qgroundcontrol, ollama, image_server, or all)	all

Example Commands

Run IsaacSim Environment with Pegasus

Before starting the main components, ensure the Isaac Sim world is running:

ISAACSIM_PYTHON isaac_sim_world.py

Start All Components

python3 start_system.py -c all

Start Specific Components

To start only specific components like QGroundControl and the Image Server:

python3 start_system.py --components=qgroundcontrol,image_server

Access the User Interface

1. Start the System

   • Navigate to http://localhost:5000 in your browser to interact with the system.

2. Provide Natural Language Commands

   • Use the interface to input commands like:
     • "Survey area"
     • "Capture images of an object"

3. Analyze Mission Data

   • Output data is saved in mission-specific directories within the project folder.

---

🎯 Expected Results

• Depth Data: Displays accurate depth estimations.
• Mission Logs: Visualized using PX4 Log Analysis Tool.
• Object Detection: JSON files contain detection details for mission objects.

---

🔍 Troubleshooting

🛠️ Click for Common Issues & Fixes

• Component Not Starting? Ensure dependencies are installed and verify configurations.
• Simulation Lag? Allocate more system resources or lower rendering settings.
• Interface Not Loading? Confirm --interface_port is correctly set.