[sdiy] ML / AI for music & synthesis (not coding!)

[brianw]

8-bit is plenty for note number, velocity, timing, or other sheet music level composition values.

Refer to the Synergy album, Computer Experiments Volume One, from 1981. John Simonton of PAiA Electronics wrote the software, which controlled a PAiA 8700 (6503-based) and Synergy Systems interface. The sounds came from a Prophet 5 modified to accept external control (before MIDI). In this example, composition is separate from synthesis.

See the Polyphony magazine article from July/August 1978 (volume 4, number 1) about the "Pink Tunes" software for more details and inspiration.

Of course, random notes within a human-selected scale is much simpler than a neural net trained on your favorite melodies and chords, but the control aspects and interfacing with synthesis should not depend on the way the numbers are generated.


Ever since people began suggesting the use of GPU for audio processing, I have been focused on the 8-bit and 16-bit limitations of GPU pixel processing, versus DSP chips that can easily handle 40-bit and even 80-bit samples. It's difficult to find any experts on GPU architecture, since everyone I've met is basically hand-waving at the high-level and none so far even have a clue about low-level details like register sizes, clock cycles and mathematical operations. Granted, even the 8-bit 6502 can handle 16-bit multiplication via software subroutines, but it's inefficient to run several instructions per operation. DSP chips can perform vast amounts of signal-related calculations in parallel, all in a single cycle. I suggest that synthesis should be realized on DSP, while the note selection, timing, and chording/melody be separated to a GPU or other processor that can implement the trained algorithms.

Brian Willoughby


On Apr 24, 2026, at 11:45 AM, Eric Brombaugh via Synth-diy <synth-diy at synth-diy.org> wrote:
> A more interesting topic to me is the use of machine learning and AI methods for realtime synthesis / effects and music creation. More and more often these days I'm seeing IoT / edge computing hardware that has neural processing accelerators built in. Often these are in quite inexpensive parts - SoCs targeted at networked cameras w/ machine vision features, MCUs with audio processing for "wake up word" recongnition, etc. So the question is how can these resources be turned to interesting synthesis applications. Often these accelerators are merely fast linear algebra coprocessors that do fast dot products so that would easily apply to filtering and convolution, but often they're limited to 8-bit data so unless you're doing chiptunes stuff that's not so useful. Any other cool ways to apply them?