REAPER + AI: Notes on a Data Problem
There's a gap in the standard AI-assisted composition workflow that doesn't get named clearly: the AI is responding to your description of the music, not the music. You paste a screenshot of a piano roll. You type out a chord progression. You explain what you're hearing. The model works from that — and it's useful, but it's one layer removed.
The question worth testing: can an AI read MIDI data directly and reason about it as music? Yes. But format matters more than you'd expect.
What the Test Showed
The same two-measure phrase — a bass line moving from an implied C major cluster to a D major chord — fed to a model three ways: notation screenshot, full MusicXML export, and a compact custom text format derived directly from REAPER's MIDI API. The screenshot produced inconsistent results. The structured data formats produced accurate pitch, rhythm, and voice-leading analysis every time.
Screenshots require visual reasoning — the model has to parse rendered pixels back into musical meaning. Structured text is pattern-matching on note names, durations, and intervals, which is what these models actually do well. Handing a model a notation image and asking for voice-leading feedback is roughly equivalent to describing a film score over the phone and asking someone to notate it.
The Efficiency Problem
MusicXML is the obvious structured format, but it carries significant overhead. A four-measure phrase with two active measures exports at roughly 6,200 characters — mostly layout and positioning tags the model never uses. The same musical information in a purpose-built shorthand runs about 110 characters. API calls are billed by token count. That ratio matters in a working session with frequent data exchanges.
REAPER's ReaScript API can read MIDI note data directly from memory — pitch, velocity, duration, position — without writing any file to disk. A Lua script can extract that data, convert it to a compact readable format, and send it to an AI endpoint in one step, mid-session. No export, no round-trip, no leaving the DAW.
Where This Is Going
The working project name is reaper-ai-bridge — kept platform-agnostic, since the architecture doesn't depend on any specific AI provider. Near-term scope is modest: a dockable panel in REAPER that reads a selected MIDI item, formats it as compact text, and opens a conversation about it. Routing logic handles model selection — a lightweight search model for quick questions, a heavier reasoning model for analysis or code generation.
The technical spec is on GitHub. This post is a working note, not a release. The interesting finding isn't the tooling — it's the framing shift. Describing music to an AI is a workaround. The actual interface is the data.