Milestone 6

An integrated ROS2 robotics system combining teleoperation, computer vision (YOLO), and AI-powered natural language interaction (LLaMA) for Turtlebot platforms.

Table of Contents

• Overview
• Prerequisites
• Installation
• Building
• Usage
• Resources
• License

Overview

Milestone 6 integrates multiple ROS2 subsystems to create a comprehensive robotics platform with four progressive parts:

• Part 1: Visual Servoing - Object tracking using camera-based feedback
• Part 2: Autonomous Grab & Transport - Complete object manipulation and movement
• Part 3: Voice-Guided Search - Speech interaction with Whisper and Espeak
• Part 4: LLM-Guided Navigation - Natural language commands via LLaMA and Espeak

This project combines:

• Teleoperation (Milestone 3): Command-line interface for robot control
• Computer Vision (Milestone 4): YOLO-based object detection
• LLM Interaction (Milestone 5): LLaMA integration for natural language processing using

The system is designed to run across multiple devices: a remote PC and a Turtlebot with Jetson

Prerequisites

• Operating System: Ubuntu 20.04 LTS
• ROS2: Foxy
• Python: 3.8
• Hardware:
  • Remote PC
  • Turtlebot with a Jetson
  • Secondary Jetson (optional)
• Models:
  • YOLO
  • Llama-2-7B-32K-Instruct-GGUF

Installation

1. Clone the Repository

git clone https://github.com/CS7389K/Milestone-6.git
cd Milestone-6

2. Set Up Ubuntu 20.04

ROS2 Foxy requires Ubuntu 20.04. If you're using Windows, set up WSL2:

wsl --install -d Ubuntu-20.04

Optional: Move WSL to a different drive (e.g., F: drive):

wsl --manage Ubuntu-20.04 --move F:\WSL

3. Install ROS2 Foxy

Run the installation script provided in the repository:

sh ./scripts/install-ros2-foxy-desktop.sh

Alternatively, follow the official ROS2 Foxy installation guide.

Building

Navigate to the project root directory and build the workspace:

colcon build --symlink-install

Usage

After building the project, source the workspace in each terminal session:

source install/setup.sh

Part 1: Visual Servoing for Bottle Tracking

Visual servoing that tracks a bottle by keeping it centered in the camera frame.

ros2 launch milestone6 part1.launch.py

Part 2: Autonomous Grab & Transport

Complete autonomous sequence to grab and transport an object.

ros2 launch milestone6 part2.launch.py

Part 3: Voice-Guided Search

Voice-controlled robot search with speech-to-text and text-to-speech.

# On Remote PC:
ros2 launch milestone6 part3.jetson.launch.py

# On TurtleBot:
ros2 launch milestone6 part3.tb3.launch.py

Note: Remote PC and TurtleBot need to be connected to the same wifi network.

Part 4: LLM-Guided Navigation

Natural language command interface using LLaMA to control the robot.

# On Remote PC:
ros2 launch milestone6 part4.jetson.launch.py

# On TurtleBot:
ros2 launch milestone6 part4.tb3.launch.py

# Send commands using the interactive CLI:
ros2 run milestone6 cli

# Or publish to /user_command topic:
ros2 topic pub /user_command std_msgs/String "data: 'search for the bottle'"

Example Commands (use lowercase):

• "scan the room for the bottle" → Robot rotates 360° looking for bottle
• "pick up the bottle" → Robot executes grab sequence
• "go to the bear" → Robot navigates toward bear object
• "we are done" → Mission complete

Resources

ROS2 Foxy Documentation

• Installation Guide
• Building Packages with Colcon
• ROS2 Actions

Python & Development

• Python 3.8 Documentation
• ROS2 Python Client Library

Troubleshooting

Launch File Not Found Error

Problem: When running ros2 launch milestone6 <launch_file>.launch.py, you get an error like:

File <launch_file>.launch.py was not found in the share directory of the package

Cause: This can occur when an old installation of the package exists in a different location (e.g., milestone6 vs Milestone-6 directory name mismatch).

Solution:

1. Find the old installation location:

   ros2 pkg prefix milestone6

2. Remove the old installation:

   rm -rf <old_installation_path>

3. Rebuild the package:

   colcon build --packages-select milestone6
   source install/setup.bash

4. Verify the new installation location:

   ros2 pkg prefix milestone6

5. Confirm launch files are installed:

   ls -la $(ros2 pkg prefix milestone6)/share/milestone6/launch/

License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Built with ❤️ using ROS2 Foxy and Python 3.8