Context

The project has previously explored custom speech synthesis and the inverse problem of recovering control/phonetic features from an audio track. Relevant rejected tasks:

• cmu11e15r09gwqw0qzkny9qt5 — "Explore the inverse decoder: sound → VoiceScenario": an attempt to automatically reconstruct the generator scenario from recorded sound, first on synthetic ground truth, then on human speech.
• cmtzxmokp085eqw0qtb7vg7c2 — "Explore high-quality programmatic human voice synthesis in the browser": search for software/DSP approaches to synthesizing human sounds and syllables without generative TTS.

New external project for research:

https://github.com/DanielSWolf/rhubarb-lip-sync

Why Rhubarb is Interesting

Rhubarb Lip Sync takes a ready-made audio track, analyzes speech over time, and outputs a sequence of mouth cues for mouth animation.

The phonetic mode is of particular importance:

• it is intended, among other things, for non-English speech;
• it does not attempt to recognize whole words;
• it recognizes individual sounds and syllables;
• it is claimed by the project to be language-independent;
• the result is tied to time intervals of the audio;
• further, phonetic features are reduced to a small set of articulation/mouth-shape classes.

This is potentially close to our task of recognizing the sound structure of speech, where a full-fledged speech-to-text might be redundant or even counterproductive.

Research Goal

Understand how exactly Rhubarb implements the pathway:

PCM/audio → phonetic features → temporal segmentation → sound/articulation classes,

and determine whether its code, algorithms, dependencies, or architectural ideas can be used for our own "sound → letter/phoneme/syllable" experiments.

Specific Items to Investigate

1. Locate the implementation of PhoneticRecognizer and all its dependencies.
2. Find out what acoustic features are extracted from PCM:
   • spectral features;
   • formants;
   • MFCC or equivalents;
   • energy/voicing;
   • state transitions;
   • time windows and smoothing.
3. Find out whether there is a trained model, lookup tables/heuristics, or an external recognizer inside.
4. Determine the initial set of recognizable phonetic states prior to conversion into mouth shapes.
5. Understand how truly language-independent the recognizer is and what assumptions about language phonetics are hardcoded into it.
6. Test its performance on very short recordings:
   • a single vowel;
   • a single consonant, if possible;
   • a syllable like ma, ba, sa;
   • several contrasting syllables.
7. Compare the recognizer's output with known ground truth and see what information it is actually capable of distinguishing.
8. Check whether it is possible to obtain intermediate data more detailed than the final A–H/X mouth shapes.
9. Evaluate the feasibility of extracting the required part of the library separately or calling Rhubarb as a CLI/local service.
10. Check the license and the feasibility of borrowing/adapting the code.

Separate Hypothesis

Rhubarb may prove useful not as a ready-made "letter recognizer," but as an example of a more realistic problem decomposition:

sound → acoustic/phonetic class → articulation class → probable phoneme → letter in a known context.

For teaching reading, this may be more useful than a direct attempt to determine a letter solely from PCM, especially if we already know the restricted set of allowed letters/syllables for an exercise.

Connection to Previous Research

It is necessary to compare Rhubarb's architecture with the conclusions from the tasks:

• cmu11e15r09gwqw0qzkny9qt5 — inverse decoder;
• cmtzxmokp085eqw0qtb7vg7c2 — programmatic voice synthesis;
• if necessary, also with research on measurable speech signal features and DSP causality.

The goal is not to restart the previous branch entirely, but to test a specific mature open-source project as a source of already working mechanisms.

Success Criteria

The worklog should contain:

• a diagram of Rhubarb's internal pipeline;
• key source files and dependencies found;
• a description of the phonetic/acoustic states it actually recognizes;
• results of several short tests;
• a conclusion on which parts or ideas are applicable to our letter/phoneme/syllable recognition task and which are not;
• if prospects exist, a separate technical task for an integration prototype.