[sdiy] ML / AI for music & synthesis (not coding!)

[Mike Bryant]

"Ever since people began suggesting the use of GPU for audio processing, I have been focused on the 8-bit and 16-bit limitations of GPU pixel processing, versus DSP chips that can easily handle 40-bit and even 80-bit samples."

GPUs can support that as well.  This CUDA instruction for example is as powerful as any DSP :

__ unsigned long long int __umul64hi<https://docs.nvidia.com/cuda/cuda-math-api/cuda_math_api/group__CUDA__MATH__INTRINSIC__INT.html#group__cuda__math__intrinsic__int_1gaefaf2a1bb2ad8986b2c248b74461e12e>(unsigned long long int x, unsigned long long int y)
Calculate the most significant 64 bits of the product of the two 64 unsigned bit integers.

How the GPU implements it shouldn't really be an issue, provided it is fast enough.  But depending when the result is needed it can implement it on a single core or spread it across many cores, of which it has far more than any DSP of course.   And with Mathworks GPUCoder you can go direct from Matlab to the GPU, something still not supported properly on most DSPs.

________________________________
From: Synth-diy <synth-diy-bounces at synth-diy.org> on behalf of brianw <brianw at audiobanshee.com>
Sent: 24 April 2026 20:49
To: Eric Brombaugh <ebrombaugh at gmail.com>
Cc: Synth-diy at synth-diy.org <synth-diy at synth-diy.org>
Subject: Re: [sdiy] ML / AI for music & synthesis (not coding!)

8-bit is plenty for note number, velocity, timing, or other sheet music level composition values.

Refer to the Synergy album, Computer Experiments Volume One, from 1981. John Simonton of PAiA Electronics wrote the software, which controlled a PAiA 8700 (6503-based) and Synergy Systems interface. The sounds came from a Prophet 5 modified to accept external control (before MIDI). In this example, composition is separate from synthesis.

See the Polyphony magazine article from July/August 1978 (volume 4, number 1) about the "Pink Tunes" software for more details and inspiration.

Of course, random notes within a human-selected scale is much simpler than a neural net trained on your favorite melodies and chords, but the control aspects and interfacing with synthesis should not depend on the way the numbers are generated.


Ever since people began suggesting the use of GPU for audio processing, I have been focused on the 8-bit and 16-bit limitations of GPU pixel processing, versus DSP chips that can easily handle 40-bit and even 80-bit samples. It's difficult to find any experts on GPU architecture, since everyone I've met is basically hand-waving at the high-level and none so far even have a clue about low-level details like register sizes, clock cycles and mathematical operations. Granted, even the 8-bit 6502 can handle 16-bit multiplication via software subroutines, but it's inefficient to run several instructions per operation. DSP chips can perform vast amounts of signal-related calculations in parallel, all in a single cycle. I suggest that synthesis should be realized on DSP, while the note selection, timing, and chording/melody be separated to a GPU or other processor that can implement the trained algorithms.

Brian Willoughby


On Apr 24, 2026, at 11:45 AM, Eric Brombaugh via Synth-diy <synth-diy at synth-diy.org> wrote:
> A more interesting topic to me is the use of machine learning and AI methods for realtime synthesis / effects and music creation. More and more often these days I'm seeing IoT / edge computing hardware that has neural processing accelerators built in. Often these are in quite inexpensive parts - SoCs targeted at networked cameras w/ machine vision features, MCUs with audio processing for "wake up word" recongnition, etc. So the question is how can these resources be turned to interesting synthesis applications. Often these accelerators are merely fast linear algebra coprocessors that do fast dot products so that would easily apply to filtering and convolution, but often they're limited to 8-bit data so unless you're doing chiptunes stuff that's not so useful. Any other cool ways to apply them?


________________________________________________________
This is the Synth-diy mailing list
Submit email to: Synth-diy at synth-diy.org
View archive at: https://synth-diy.org/pipermail/synth-diy/
Check your settings at: https://synth-diy.org/mailman/listinfo/synth-diy
Selling or trading? Use marketplace at synth-diy.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://synth-diy.org/pipermail/synth-diy/attachments/20260424/02a219ca/attachment.htm>