Coordinated Autonomy for a Jazz Jam Companion

Research question

A human rhythm section shares tempo, form, harmony, and an understanding of who currently has the lead. It does not, however, behave like three copies of one central controller. A pianist may leave space, a bassist may hold the form steady, and a drummer may answer a long rest with a restrained fill. The decisions are separate, but not unbounded.

This project asks whether that division of decision ownership matters in a real-time AI ensemble. The independent variable is not simply which generator produces better notes. It is who owns each musical decision: a fixed arrangement, one central adaptive arranger, or three instrument-specific policies operating under shared constraints.

Can bounded local autonomy make the rhythm section more responsive without making the ensemble less coherent or the human less in control?

The comparison is deliberately narrow. The first study uses one human lead, three MIDI rhythm-section voices, known tempo and harmony, and one fixed form. A system that works in this setting would still not establish general Jazz understanding.

Three matched conditions

B0, B1, and B2 should reuse the same lead sheet, pattern or phrase pool, sound set, clock, scheduler, and latency budget wherever possible. The goal is to compare decision structures, not three unrelated music systems.

Static accompaniment

A fixed rhythm-section arrangement follows the lead sheet but does not react to the sax performance.

What it isolates: Separates a backing track that can play the form from a system that can actually respond.

Centrally scripted adaptation

One arranger maps observable sax features to whole-band states such as low, medium, or high energy.

What it isolates: Measures the value of adaptation when every machine voice still follows the same central decision.

Coordinated autonomy

Piano, bass, and drums choose instrument-specific actions while a coordinator enforces shared state and global guards.

What it isolates: Tests whether bounded local decision ownership feels more like an ensemble without creating more conflict.

Implementation sequence

Implementation starts from timing, form, and logging rather than from a large generative model. Fixed patterns, retrieval, lightweight variation, Wizard-of-Oz material, Notochord, or a Somax2 bridge can all become interchangeable backends later. They should not be allowed to change the research variable while the baselines are being built.

What exists now

Recorded

Pinned implementation reference

The current foundation audit is bound to Somax2 v2.7.0 at commit 3782bbd. Its source, process model, OSC routes, player lifecycle, scheduling path, and Max/Python boundary have been mapped before modifying the system.

Recorded

Partial backend bring-up

A dated local audit records Python compilation and imports, server startup, a central /somax request/response round trip, and creation plus initialization of a separate player-agent process. This is a partial backend check, not proof of the full musical system.

Open

End-to-end music path

The Max interface and DSP path, MIDI/audio rendering, the three-voice B0/B1/B2 prototype, five-minute stability, measured latency, and musician evaluation have not yet been validated end to end.

These bring-up results were recorded in a local architecture and deployment audit on 30 August 2026; they were not rerun as part of this website update.

Shared musical ground
Build a three-track MIDI skeleton in which piano, bass, and drums read the same clock, meter, chord chart, form position, current lead, and event log.
Matched baselines
Implement B0 first, then B1, while keeping the sound set, pattern pool, scheduling rules, and latency budget as comparable as possible.
Local policies and guards
Add B2 policies for energy and density response plus sax-to-piano handoff. Log each proposed action, any rejected conflict, the selected fallback, and the sound that was scheduled.
Live pilot before claims
Use a short 12-bar Jazz-blues loop for engineering smoke tests, then move to one fixed Jazz standard for a musician pilot only after the three conditions are stable and audibly distinguishable.

How it will be evaluated

“It sounded good” is not enough to distinguish an interaction design from a backing track. The study therefore keeps interaction quality, ensemble stability, musical quality, and the musician's experience as separate outcomes.

Responsiveness

Direction and latency of each voice's response to observable changes in sax density, velocity, register, and rests; handoff and return success; missed or excessive responses.

Ensemble coherence

Beat, harmony, and form errors; register or density collisions; simultaneous fills; deadline misses; dropouts; guard interventions; deterministic fallback rate.

Human control

Whether musicians feel that separate voices are listening, whether the result stays predictable without becoming rigid, and whether autonomy distracts or takes away unwanted space.

Two first interaction probes

Energy and density response: the sax becomes denser, louder, sparser, or leaves a long rest; piano, bass, and drums should respond differently but compatibly.
Sax-to-piano handoff and return: the lead moves from the human sax to AI piano and back at a musically legal point, while bass and drums maintain support.

The engineering pilot may use a 12-bar Jazz-blues loop because chorus boundaries are easy to inspect. The intended live pilot uses one fixed Jazz standard selected with musician input. If the project remains limited to one form, the conclusion will remain form-specific.

Scope and claim boundary

In the first study

One human saxophonist or EWI player
AI piano, bass, and drums over MIDI
Known tempo, meter, harmony, clock, and form
B0, B1, and B2 under matched musical conditions
Energy response plus piano handoff and return
Replayable decisions, failures, and fallbacks

Not claimed

A general solution for arbitrary Jazz styles or open forms
A completed or validated public prototype
Perfect real-time transcription of acoustic saxophone
Bass or drum solo autonomy in the first version
Emotion, fatigue, personality, or hidden-intent inference
A “first AI Jazz companion” novelty claim

Relationship to earlier ideas

The earlier public card centered on groove and solo retrieval with bounded generative variation. Those mechanisms remain useful engineering options, but they are no longer the headline question. Retrieval or generation can supply musical candidates; coordinated autonomy decides how decision rights, constraints, and fallbacks are distributed across the ensemble.

Trading fours or twos also remains a useful Jazz interaction and may become a structured stress test. Sparse visual cues may later help with high-level events such as take solo, return, head, or end if music-only handoff remains ambiguous. Neither trading nor vision is required for the main B0/B1/B2 comparison.

This semester research project is also separate from Jammy, an earlier product idea that does not have a prototype or MVP. Research code or findings may inform a future product, but starting this implementation does not make the two projects equivalent.

Coordinated Autonomy for a Jazz Jam Companion

Shared score, local decisions

Human sax

AI piano

AI bass

AI drums