Blue Guide — Agentic Tour Guide

Individual Apps · Blue Guide — Agentic Tour Guide

iOS · SwiftUI · FastAPI · WebSockets · Multi-agent · Mar 2026 – Present

→ Join the waitlist — get a note when the app ships.

An iOS app that narrates the world around you as you walk or drive. Open it, tap Start, put your phone away — Blue Guide tracks your GPS, researches what's nearby via a multi-agent pipeline, and plays narrations through your headphones. The experience mirrors a museum audio guide, but it works everywhere, continuously, and without pre-planning.

entry_id=1_9zvdz68f widget_id=1_whsln7jd title=Blue Guide Demo Video Mar.24

Production rewrite of the hackathon-era BlueGuide MVP (see Side Projects). Rebuilt around a real-time WebSocket pipeline, a three-agent narration engine, trip journaling, billing, and a self-hosted backend.

What it does

Walk mode — 200 m detection radius, narrates individual buildings, street history, neighborhood culture.
Drive mode — 2 km radius, narrates regions / landscapes / notable exits. Uses heading to say "on your left" or "coming up ahead."
Multimedia suggestion cards — alongside each narration the agent can surface rich cards: YouTube videos, Wikipedia links, image galleries, place markers, music tracks, embedded web pages, quotes, and facts. A card enrichment subagent fires asynchronously to fetch metadata and images.
Real-time audio — narration streams as you move. A collapsible sheet is a music-player-style transcript; tap any line to replay it.
Trip journal — sessions auto-group into city → day buckets (reverse-geocoded, 48-hr window, local-timezone day boundaries). Each day replayable as a map with the traveled polyline and narration pins. Newest days shown first.
Trip share posters — one-tap export of the trip route as a shareable image.
Free + paid tiers — 5 narrations/day free (20 signup bonus; 40 for .edu addresses). Paid options: Pro Monthly (unlimited, $69.99/mo), Day Pass, Weekly Pass, or narration packs (50 or 200).

Screens

The AI pipeline

Turns a GPS coordinate into an audio clip via a three-agent system:

Location update — iOS streams {lat, lng, heading, speed, ts} over WebSocket every 5 s (walk) or 10 s (drive).
Trigger evaluation — backend decides whether to narrate based on (a) distance moved, (b) a new unmentioned POI entering radius, (c) 45 s+ dwell near a landmark, with a 15 s cooldown. GPS confidence gating suppresses jitter.
Geo context assembly — Google Places nearby search + reverse geocoding, filtered against an in-session mentioned_poi_ids set, combined with heading / speed / mode / user interests into a structured context object.
Planner agent (agent.py) — main ReAct loop, default Gemini 3 Flash. Researches via 12 tools (web_search, fetch_url, search_nearby, search_place, get_place_details, get_directions, explore_route, get_notes, suggest, generate_narration, pause, ask_user). Decides when to narrate and what to say. Per-session conversation memory so it never repeats itself.
Narration writer subagent (_execute_narration_subagent) — receives a research brief from the planner, drafts narration JSON; planner reviews/approves. Default Claude Haiku 4.5. Context compression kicks in when prompt tokens exceed 70% of window.
Card enrichment subagent (card_agent.py) — fire-and-forget after each suggest() call. Fetches link metadata, images, writes a "why this matters" body. Default Gemini 2.5 Flash, 12 s hard deadline.
TTS — Gemini gemini-2.5-flash-preview-tts, streamed server-side.
Delivery + storage — narration text and audio pushed to client over WebSocket; every generation persisted to Postgres with {lat, lng, ts, transcript, cost breakdown} for the trip journal.

Architecture

iOS (SwiftUI + MapKit + CoreLocation + AVFoundation)
  ──(REST + WebSocket)──▶
  FastAPI backend (Python 3.11+, ruff, pytest asyncio)
    ├─ auth/      Apple Sign In + Google OAuth + email/password + JWT
    ├─ users/     profile + preferences (voice speed, interests, language)
    ├─ geo/       Google Places, reverse geocoding, weather (Open-Meteo)
    ├─ guide/     planner agent, writer subagent, card subagent, TTS, WS handler
    ├─ trips/     trip / day / generation / bookmark CRUD + path GeoJSON
    ├─ billing/   Stripe subscriptions + Apple IAP + narration quota system
    ├─ admin/     cost analytics + user management dashboard
    └─ shared/    db (asyncpg), cache (Redis), HTTP, cost math, quota enforcement

iOS — ContentView.swift root with NavigationStack + AppDestination enum, ZStack overlays (TalkToGuide, Tutorial, Cards, PinInfo). Features under Features/{Guide,Journal,Auth,Achievements,Map,Settings,Transcript}/. Journal is a first-class feature with TripJournalView, TripReviewMapView, TripDayCard. iOS 26 Liquid Glass design, @Observable + Combine.

Provider layer — pluggable OpenAI-compatible clients support Gemini, Anthropic, OpenAI, Kimi/Moonshot, Zhipu GLM, MiniMax. Each subagent picks its own provider via *_provider / *_model config pairs.

Backend discipline — max ~600 lines per file, split by responsibility, re-export from __init__.py, 294 pytest tests, ruff for lint + format.

Data model

users (with is_student flag), preferences, sessions, trips, trip_days (with path_geojson), generations (timestamped narration segments with lat/lng, transcript, audio URL, per-field cost breakdown), bookmarks, user_narration_balance, user_daily_narrations. Standalone PostgreSQL (not Supabase); migrations in backend/migrations/.

Infrastructure + cost

Hosting — self-hosted VPS (Hetzner, 46.62.224.92). Docker Compose. Serverless ruled out because WebSockets require persistent connections.
DB + cache — standalone Postgres + Redis (POI cache, rate limiting, session state).
Per-narration cost — $0.05–$0.12 typical (planner ~$0.003, writer ~$0.003, Tavily search ~$0.025, Google Places ~$0.055, TTS ~$0.004, infra amortised ~$0.015). Fixed infra ~$45/month.
Billing — narration-count system (not time-based). Atomic quota reservation prevents double-spend. Month-start balance rolled over server-side.

Roadmap

V1.0 (App Store) — auth, walk/drive modes, WS narration pipeline, multimedia cards, TTS, transcript sheet, trip journal + share posters, bookmarks, quota billing, Dynamic Island.
V1.1 — offline narration replay, narration bookmarks, home screen widget (today's distance + count).
V2.0 — conversational mode: two-way voice with Whisper STT, deep-dive "tap a building to ask about it."
V3.0 — camera identification, offline narration packs for popular routes, group tours, multi-language.

Engineering principles

Modular by default — every feature is its own package with router / service / models; max ~600 lines per file
Provider-agnostic AI layer — each of the three agents picks its own model; one-config swap to change provider
Zero-effort UX — after Start, the user should never need to touch the phone again
Non-intrusive pacing — cooldown between narrations, GPS confidence gating, transparent transcript so users can verify the AI's claims

This is the plain-HTML mirror served to crawlers, LLMs, and curl. Humans with a JavaScript-enabled browser see the rich React/XP-themed SPA at the same URL.

All plain pages · Live site · sitemap.xml