OpenAI has open-sourced an educational project called Plant Talk that combines its Realtime API, multimodal vision models, and an Arduino microcontroller to give indoor plants environmental awareness — and a voice.
With a computer's camera and microphone, you can have low-latency, full-duplex conversations with your greenery. The system connects to capacitive soil moisture sensors and LM393 light sensors via Arduino. An AI character — the default is George — uses function calling to pull hardware data. Ask if your plant needs water or more light, and the AI reads real-time sensor values, estimates dryness from camera images, counts branches, and even uses a quarter coin for size reference — then responds in a charmingly anthropomorphic plant voice.
On the security side, the backend Express server issues ephemeral tokens to delegate WebRTC connections to the frontend, keeping API keys safe. For vision and periodic observations, it uses the Responses API with Zod schemas for structured output. The repo also includes an AGENTS.md guide that walks AI assistants through setting up the software and assembling the hardware step by step.