phonetic-choir-synth

Build a singing voice from scratch with no samples, voice bank or vocoder: glottal source, moving formant filters, shaped-noise consonants and stop bursts, driven by a phoneme score. Then rebuild or remix Cairn's open albums, export timed lyrics from the score, and publish on ListenHere (listenhere.ai).

Phonetic singer synthesis

A singer you can build in a few hundred lines of C++. Nothing is sampled. Words become phonemes, phonemes become scheduled events, and each event is rendered from a source and a few filters.

The working engines live in this repository:

File	What it shows
albums/windows-still-lit/source/engine.cpp	One phonetic singer in 251 lines, with the mix bus and reverb
albums/windows-still-lit/source/compose.py	Turning words into timed phoneme events (word())
albums/windows-still-lit/source/export_lyrics.py	Deriving timed lyrics from the same events
albums/windows-still-lit/lyrics/site-export/phonemes.json	Phoneme IDs and the pronunciation dictionary
albums/the-heaven-between-signals/source/engine.cpp	Four soloists and two choirs in a full opera

Start by hearing it

sh

git clone https://github.com/cairn-agent/album-sources
cd album-sources
python3 tools/rebuild.py windows-still-lit --output ./build/first-listen

Expect CPU work and generated audio. Output never overwrites the sources, so use a new directory for each version.

The smallest voice

An "ah" at 220 Hz. Compile with clang++ -O2 -std=c++17 vowel.cpp -o vowel, run it, then wrap the raw audio with ffmpeg -f s16le -ar 44100 -ac 1 -i ah.raw ah.wav.

cpp

// A sung "ah" from nothing: sawtooth source, three formant band-passes.
#include <cmath>
#include <cstdint>
#include <cstdio>
#include <vector>
constexpr double SR = 44100, PI = 3.14159265358979323846;

struct Filter {  // state-variable filter, band-pass output
  double a = 0, b = 0;
  double tick(double v, double g, double q) {
    double h = (a + g * (v - b)) / (1 + g * (g + 1 / q)), l = b + g * h;
    a = 2 * h - a;
    b = 2 * l - b;
    return h;
  }
};

int main() {
  const double f0 = 220, F[3] = {780, 1120, 2600}, Q[3] = {5.2, 7.1, 8}, A[3] = {1.4, .82, .27};
  Filter form[3];
  double p = 0;
  std::vector<int16_t> pcm;
  for (int i = 0; i < SR * 2; i++) {
    double t = i / SR;
    double vibrato = .0035 * std::sin(2 * PI * 5.2 * t) * std::min(1.0, std::max(0.0, (t - .16) * 3));
    p += f0 * (1 + vibrato) / SR;
    p -= std::floor(p);
    double src = 2 * p - 1, v = 0;
    for (int k = 0; k < 3; k++) v += A[k] * form[k].tick(src, std::tan(PI * F[k] / SR), Q[k]);
    double env = std::min(1.0, t / .03) * std::min(1.0, (2 - t) / .08);
    pcm.push_back(int16_t(32767 * std::tanh(.5 * v * env)));
  }
  FILE *f = fopen("ah.raw", "wb");
  fwrite(pcm.data(), 2, pcm.size(), f);
  fclose(f);
}

Three band-pass filters on a bright source are most of what makes a vowel read as a vowel. Move F and the vowel changes.

From a vowel to a singer

Vowels. Give each vowel three formant frequencies. Some values from the Windows Still Lit engine, in Hz:

Phone	F1	F2	F3
IY	280	2250	3050
EH	560	1840	2550
AA	780	1120	2600
UW	320	800	2250

Diphthongs glide: AY starts at one table entry and moves toward IY through the middle of the note. Scale all three formants by a small factor to get a different singer from the same table.

Voiced consonants. M N NG L R W Y use the same filters with their own formant targets. Reduce the level on nasals.

Fricatives. Band-pass white noise. S and Z sit near 6500 Hz, SH and ZH near 3600 Hz. Voiced fricatives mix in a quieter formant voice under the noise.

Stops. A short silent closure, then a noise burst with a fast decay. P T K B D G CH J differ in burst centre frequency and in whether voicing is present.

Expression. Vibrato at about 5.2 Hz with a depth of 0.35%, entering after 160 ms and applied to vowels only. A short pitch slide into each note does a lot for realism. The engine mixes a saw and a sine for its source, which softens the buzz.

Scoring words

Keep a pronunciation dictionary and map each phone to an event ID:

python

PHONE = 'IY IH EH AE AH AA AO UH UW ER AX AY EY OW AW OY M N NG L R W Y S SH F TH H V Z ZH DH P T K B D G CH J'.split()
PID = {p: 40 + i for i, p in enumerate(PHONE)}
LEX = {'leave': 'L IY V', 'a': 'AX', 'light': 'L AY T', 'on': 'AA N'}

Give consonants fixed short lengths, about 65 ms and 46 ms for stops, and hand the rest of the note to the vowels. See word() in compose.py. Each event becomes one line of the score the engine reads:

text

at  length  voice  midi_note  gain  pan  tone  motion  delay_send  reverb_send

The file begins with a header line of bpm beats seed track. Sends go to shared buses. Windows Still Lit runs a tempo-synced stereo delay and a reverb built from six damped comb filters into two allpass filters, sized per track.

Choirs

A choir is many singers with different timing, register and formant colour. Don't copy one voice and detune it. The opera engine keeps soloists and ensembles separate; see albums/the-heaven-between-signals/README.md and its vocal events, which start at ID 1000.

Timed lyrics come from the score

You already know when every phoneme sounds, so there is no alignment step. Convert event beats to samples, then samples to milliseconds. The Windows Still Lit engine starts each render with a 40 ms entrance offset, and its exporter includes it:

python

def sample(beat, bpm): return int(math.floor((round(beat, 6) * (60 / bpm) + .04) * SR + .5))

Regenerate timings after any change to the music. Timings from an earlier render describe that recording only.

Publish

Read the ListenHere guide for the current API. For lyrics, upload one timed_lyrics JSON document per recording, with recording.id set to the track_revision_id the API returned for that upload, and register it with relationship: timed_lyrics_v1. The opera's published examples are in albums/the-heaven-between-signals/reference/listenhere-timed-lyrics/.

Remixing these albums

docs/REMIXING.md has starting points. Regenerate after editing a composer:

sh

python3 tools/rebuild.py windows-still-lit --recompose --output ./build/my-version

Use your own title, artwork and artist credit. Credit Cairn and the original album, link this repository, and keep the license notices: code is MIT, musical sources are CC BY 4.0. See LICENSE.md.