Recorder returns empty buffers when in 'Mixed' mode

[onionhammer]

On my Mac (haven't tried windows yet), when I set my audio engine to 'Mixed' mode instead of 'Record' mode, the data returned in the callback is empty - all 0's - but it works as I would expect if I use Capability.Record.

I have the same issue when trying to use the MicrophoneDataProvider directly instead of the recorder.

// #r "nuget: DotNetEnv"
// #r "nuget: Porcupine"
// #r "nuget: SoundFlow"
using DotNetEnv;
using Pv;
using SoundFlow.Backends.MiniAudio;
using SoundFlow.Components;
using SoundFlow.Enums;

Env.Load(".env");

var PICO_VOICE_KEY = Environment.GetEnvironmentVariable("PICO_VOICE_KEY");

BuiltInKeyword[] keywords = [BuiltInKeyword.BLUEBERRY];

Porcupine porcupine = Porcupine.FromBuiltInKeywords(accessKey: PICO_VOICE_KEY, keywords: keywords);

using var audioEngine = new MiniAudioEngine(porcupine.SampleRate, Capability.Record);

// Buffer to accumulate audio samples for Porcupine processing
Console.WriteLine($"Porcupine frame length: {porcupine.FrameLength} samples");

var frameBuffer = new short[porcupine.FrameLength];

using var recorder = new Recorder(
    (data, c) =>
    {
        // Since we're recording in stereo (2 channels), we need to convert to mono
        // by taking only the left channel (every even index) or averaging both channels
        int monoSampleCount = data.Length / 2;

        for (int i = 0; i < monoSampleCount; i++)
        {
            // Take left channel only (index i * 2), convert float [-1.0, 1.0] to short [-32768, 32767]
            float sample = data[i * 2];
            frameBuffer[i] = (short)(sample * short.MaxValue);
        }

        // Process audio in chunks of the correct frame size
        if (porcupine.Process(frameBuffer) is int keywordIndex and >= 0)
        {
            Console.WriteLine($"Wake word detected: {keywords[keywordIndex]}");
        }
    },
    sampleRate: porcupine.SampleRate,
    sampleFormat: SampleFormat.F32, // Use float format since we're doing the conversion ourselves
    encodingFormat: EncodingFormat.Wav,
    channels: 2
);

recorder.StartRecording();

// Stop playback and capture.
Console.WriteLine("Listening... Press any key to stop.");
Console.Read();

[LSXPrime]

Can you confirm it's the same as miniaudio duplex example?

[onionhammer]

The same in what sense? When I run the sample you link, the vocoder seems to work (I can hear myself from the speakers). Same with https://lsxprime.github.io/soundflow-docs/tutorials-and-examples/#3-microphone-playback-loopbackmonitor

But i'm not trying to directly pipe the microphone to the speaker, I'm trying to just listen for a wakeword, and optionally play some other sound

This removes all the picovoice dependencies so it can be run without a key

using SoundFlow.Backends.MiniAudio;
using SoundFlow.Components;
using SoundFlow.Enums;

using var audioEngine = new MiniAudioEngine(16000, Capability.Mixed);

// Buffer to accumulate audio samples for Porcupine processing
Console.WriteLine($"Porcupine frame length: {512} samples");

var frameBuffer = new short[512];

using var recorder = new Recorder(
    (data, c) =>
    {
        // Since we're recording in stereo (2 channels), we need to convert to mono
        // by taking only the left channel (every even index) or averaging both channels
        int monoSampleCount = data.Length / 2;

        // BREAKPOINT HERE

        for (int i = 0; i < monoSampleCount; i++)
        {
            // Take left channel only (index i * 2), convert float [-1.0, 1.0] to short [-32768, 32767]
            float sample = data[i * 2];
            frameBuffer[i] = (short)(sample * short.MaxValue);
        }

        // // Process audio in chunks of the correct frame size
        // if (porcupine.Process(frameBuffer) is int keywordIndex and >= 0)
        // {
        //     Console.WriteLine($"Wake word detected: {keywords[keywordIndex]}");
        // }
    },
    sampleRate: 16000,
    sampleFormat: SampleFormat.F32, // Use float format since we're doing the conversion ourselves
    encodingFormat: EncodingFormat.Wav,
    channels: 2
);

recorder.StartRecording();

// Stop playback and capture.
Console.WriteLine("Listening... Press any key to stop.");
Console.Read();

[LSXPrime]

There was also a report for an issue with mixed mode on macOS (#67), which I initially thought might be related. However, since Capability.Record is working as expected too, the next release should address your issue.

[LSXPrime]

Please update to V1.2.0 and merge the changes. For Mixed Mode, since Capability.Record is working properly for you, the new duplex device should work fine.

[onionhammer]

Thanks! i'll try it out soon