NYU Tandon COPHEE Lab · Jan 2026 – Present · with Scarlett Wang · advised by Prof. Dong Whi Yoo
Intro
Benchmarking LLMs against established astrology knowledge bases to probe whether they can deliver the kind of reflective, narrative-driven support that drives astrology's appeal — as a lens for AI-assisted mental-health interventions.
Research goals
Treat astrology as a stress test for whether LLMs can produce structured, personalized reflective narratives.
Explore whether AI-only astrology-style interactions could act as a low-stakes scaffold for emotional reflection in populations that won't engage with clinical tools.
Surface how LLMs handle structured pseudo-knowledge — a useful window into confabulation-with-plausibility.
Research questions
RQ1 — Coverage — does the LLM produce interpretations for every field an established reference covers?
RQ2 — Coherence — are the outputs internally consistent, or self-contradicting across fields?
RQ3 — Reflective quality — do outputs invite open reflection, or collapse into closed predictions?
RQ4 — Safety — does the model refuse outright, over-caveat, or drift into clinical territory it shouldn't?
Plan
A web-scraper pulls canonical interpretations from an established astrology reference; a LiteLLM-based runner queries multiple providers with the same chart inputs; outputs are saved as structured JSON for side-by-side comparison against the reference.
This is the plain-HTML mirror served to crawlers, LLMs, and curl. Humans with a JavaScript-enabled browser see the rich React/XP-themed SPA at the same URL.