🎙️ Meet Arty, your Voice-Powered Mobile Assistant

An open-source voice assistant for mobile with real-time API integration. Think "Ollama for mobile + realtime voice."

Connects to your Google Drive, DeepWiki, Hacker News, Daily Hugging Face Top Papers, and the web.

It's currently a thin wrapper around the OpenAI Realtime speech API, however the long term vision is to make it extensible and pluggable, with a fully open source and privacy-first stack. To keep all data within your cloud perimeter, Azure OpenAI Private Endpoint can be configured as an alternative deployment option.

If this sounds interesting, ⭐️ the project on GitHub to help it grow.

🎤 Demo audio - browse hacker news (1 min 30 secs)

hackernews-arty-demo.mp4

Whats's in the demo

• "What are top storiees on hacker news?"
• "What are commetns about montana law story?"
• "SUmmarize new montana law"

🎥 Demo Reel (80 seconds)

Arty_Demo.mp4

or view the full resolution version

📱 Screenshots

Voice chat (home screen)

Text chat

Configure connectors

🎯 Why This Project Was Created

Voice AI is now incredibly powerful when connected to your data, yet current solutions are closed source, compromise your privacy, and are headed toward ads and lock-in.

This project offers a fully open alternative: local execution, no data monetization, and complete control over where your data goes.

▶️ Install it via TestFlight

Test Flight Installation

Security note: TestFlight builds are compiled binaries; do not assume they exactly match this source code. If you require verifiability, build from source and review the code before installing.

Getting Started Instructions

1. Create a new OpenAI API key. Grant the minimum realtime permissions shown below: (Models read, Model capabilities write)
   
2. Grant access to Responses API.
   
3. Paste the key into the onboarding wizard and tap Next.
   
4. Connect Google Drive so Arty can see your files. OAuth tokens stay on-device. See Security + Privacy for details.
   
5. Choose the Google account you want to use.
   
6. Tap “Hide Advanced” and then “Go to vibemachine (unsafe).”
   
7. Review the OAuth scopes that Arty is requesting.
   
8. Confirm the connection. You should see a success screen when Drive is linked.
   
9. Optional: Provide your own Google Drive Client ID for extra control.
   
10. Finish the onboarding wizard.
    
11. Start chatting with Arty.
    

How to get the most out of it

• Personalize Arty: adjust the system prompt, voice, VAD mode, and tool configuration from the Advanced settings sheets to match your workflow.
• Try out text chat mode when you can't use voice. Under settings, configure it to use text chat mode. Note, there's no streaming token support yet, so it feels pretty slow.
• Explore the connectors: Enable DeepWiki for documentation search, Hacker News for tech news, and Daily Hugging Face Top Papers for the latest AI research.

✨ Features

1. Supports several connectors: Google Drive, DeepWiki, Hacker News, Daily Hugging Face Top Papers, and Web Search - Voice assistant that can summarize content in Google Drive, search documentation with DeepWiki, browse Hacker News, discover the latest AI research papers, and search the web
2. Extensible - Adding connectors is fairly easy. File an issue to request the connector you'd want to see.
3. Customizable prompts - Edit system and tool prompts directly from the UI
4. Multi-mode audio - Works with speaker, handset, or Bluetooth headphones
5. Background noise handling - Mute yourself in loud environments
6. Session recording - Optional conversation recording and sharing
7. Voice and text modes - Switch between input methods seamlessly
8. Observability - Optional Logfire integration for debugging (disabled by default)
9. Privacy-focused - Working toward a fully private solution with local execution options

🚧 Limitations

1. Cost - OpenAI API costs can add up with extended usage due to context window management
2. Text Mode is limited - The Text mode does not support streaming tokens yet. It has a very basic and limited UX.
3. Platform - iOS only, no Android support yet due to currently using native WebRTC library, despite using React Native via Expo.
4. UX - No progress indicators during operations
5. Recording - Optional call recording implementation doesn't work very reliably since it regenerates the conversation based on a text transcript

🔐 Security + Privacy

Privacy status: We're actively working toward a fully private, end-to-end local solution. Currently, the app uses OpenAI's API, which means user prompts and connector content are transmitted to OpenAI by design. Your credentials (API keys, OAuth tokens) never leave your device and are stored securely in iOS Keychain. Future updates will add support for self-hosted and fully local execution options.

To keep all data within your cloud perimeter, Azure OpenAI Service with Private Link can be configured to ensure traffic remains within your virtual network infrastructure.

From a security perspective, the main risks are credential leakage or abuse:

1. OpenAI API Key
2. Google Drive Auth Token

Mitigation: All credentials remain on-device, stored only in memory or secure storage (iOS Keychain). Audit the source code to verify that no credentials are transmitted externally.

Security + privacy: storage, scopes, and network flow recap

• All token storage in memory and secure storage happens in lib/secure-storage.ts

• The actual saving/retrieval of tokens is delegated to the Expo library expo-secure-store

• Transport security: All outbound requests to OpenAI, Google, and optional Logfire use HTTPS with TLS handled by each provider. This project does not introduce custom proxies or MITM layers.

• OAuth tokens and API keys are stored via expo-secure-store, which maps to the iOS Keychain using the kSecAttrAccessibleAfterFirstUnlockThisDeviceOnly accessibility level. Tokens are never written to plaintext disk.

• Recording is off by default, and conversation transcripts are not saved. Optional recordings remain on-device and rely on standard iOS filesystem encryption.

• No third-party endpoints beyond OpenAI, Google, and optional Logfire are contacted at runtime. The app does not embed analytics, crash reporting SDKs, or ad networks.

• The Google Drive OAuth scope used by the default Client ID in the TestFlight build is read-only—it can create or edit files that the app created, but cannot edit or delete files that originated elsewhere. For tighter control, register your own Google Drive app, supply its Client ID, and grant the permissions you deem appropriate.

• Assume that connector operations which retrieve file contents may send that content to the LLM for summarization unless you have deliberately disabled that behavior.

Observability logs are disabled by default. Note that these should be automatically scrubbed of API tokens by Logfire itself. Only enable Logfire after you have audited the code and feel comfortable—this is mainly a developer feature and not recommended for casual usage or testing.

Out of scope: This project does not currently defend against (1) on-device compromise, (2) malicious LLM responses executing actions against connected services using delegated tokens, or (3) interception of API traffic by the model provider.

🛠️ Building from source

Installation steps

Clone project and install dependencies

git clone https://github.com/vibemachine-labs/arty.git
cd arty
curl -fsSL https://bun.sh/install | bash
bun install

Create a Google Drive Client ID

When building from source, you will need to provide your own Google Drive Client ID. You can decide the permissions you want to give it, as well as whether you want to go through the verification process.

Google API Instructions

For testing, the following oauth scopes are suggested:

1. See and download your google drive files (included by default)
2. See, edit, create, and delete only the specific Google Drive files you use with this app

Run the app

To run in the iOS simulator:

bunx expo run:ios

⚠️ Audio is flaky on the iOS Simulator. Using a real device is highly recommended.

To run on a physical device:

bunx expo run:ios --device

Editing Swift code in Xcode

Open Xcode project

To open the project in Xcode:

xed ios

In Xcode, the native swift code will be under Pods / Development Pods

Misc Dev Notes

Disable onboarding wizard (optional)

For certain testing scenarios, disable the onboarding wizard by editing app/index.tsx and commenting out the useEffect block that evaluates onboarding status:

useEffect(() => {
  let isActive = true;

  const evaluateOnboardingStatus = async () => {
    try {
      const storedKey = await getApiKey();
      const hasStoredKey = typeof storedKey === "string" && storedKey.trim().length > 0;
      if (!isActive) {
        return;
      }
      setOnboardingVisible(!hasStoredKey);
    } catch (error) {
      if (!isActive) {
        return;
      }
      log.warn("Unable to determine onboarding status from secure storage", error);
      setOnboardingVisible(true);
    }
  };

  if (!apiKeyConfigVisible) {
    void evaluateOnboardingStatus();
  }

  return () => {
    isActive = false;
  };
}, [apiKeyConfigVisible, onboardingCheckToken]);

Development notes

• Project bootstrapped with bunx create-expo-app@latest .
• Refresh dependencies after pulling new changes: bunx expo install
• Install new dependencies: bunx expo install <package-name>
• Allow LAN access once: bunx expo start --lan

Run on iOS device via ad hoc distribution

1. Register device: eas device:create
2. Scan the generated QR code on the device and install the provisioning profile via Settings.
3. Configure build: bunx eas build:configure
4. Build: eas build --platform ios --profile dev_self_contained

Clean build

If pods misbehave, rebuild from scratch:

bunx expo prebuild --clean --platform ios
bunx expo run:ios

⚙️ Technical Details

Architecture overview

Native Swift WebRTC Client

React Native WebRTC libraries did not reliably support speakerphone mode during prototyping. The native Swift implementation resolves this issue but adds complexity and delays Android support.

Connector Architecture

All connectors use statically defined tools with explicit function definitions, providing reliability and predictable behavior. Examples include Google Drive file operations, DeepWiki documentation search, Hacker News browsing, and Daily Hugging Face Top Papers discovery.

MCP Support

Not yet implemented since all tools are currently local. Future versions will add MCP server support via cloud or local tunnel connections.

Web Search

GPT-4 web search serves as a temporary solution. The roadmap includes integrating a dedicated search API (e.g., Brave Search) using user-provided API tokens.

Voice / Text LLM backend

OpenAI is currently the only supported backend. Adding support for multiple providers and self-hosted backends is on the roadmap.

🗺️ Roadmap

1. Address limitations listed above
2. Improve text mode support
3. Investigate async voice processing to reduce cost
4. Add support for alternative voice providers (Unmute.sh, Speaches.ai, self-hosted)
5. Remote MCP integration
6. TypeScript MCP plugin support

💼 Business Model

The app itself will remain completely open source, with no restrictions or limitations.

Business model TBD. Likely a managed backend service using either:

• Azure OpenAI realtime APIs
• Fully open-source stack — possibly Unmute.sh or Speaches.ai

🤝 How You Can Help

• Spread the word - Star github.com/vibemachine-labs/arty, share with friends
• Try it - Run the app and file issues
• Give feedback - Fill out a quick questionnaire (10 questions, 2 mins) or schedule a 15-min user interview
• Contribute ideas - File issues with appropriate labels
• Create pull requests - For larger proposed changes, it's probably better to file an issue first

📬 Contact & Feedback

• Email/Twitter: Email or Twitter/X via my Github profile.
• Issues, Ideas: Submit bugs, feature requests, or connector suggestions on GitHub Issues.
• Discord: A server will be launched if there’s enough interest.
• Responsible disclosure: Report security-relevant issues privately via email using the address listed on my Github profile before any public disclosure.