How to play audio stream ?

Oct 5, 2012 at 10:48 PM

Hi Peekb,

Thank for your last release

i am a beginner,
i succeffuly used the kinect service to access video through the network..

But i am not able to run the audio service on the client.

i have done this:

 SoundPlayer test1 = new SoundPlayer();

AudioClient audioClient = new AudioClient();

audioClient.AudioFrameReady += audioClient_AudioFrameReady;

 i think the probleme is here :

void audioClient_AudioFrameReady(object sender, AudioFrameReadyEventArgs e)
        {          
           if (test1!= null)
            {                        
            test1.Stream.Read(e.AudioFrame.AudioData, 0,0);
            }
        }

Can't find what to write in this event...

Please help

Oct 6, 2012 at 2:12 AM

There isn't a great way to directly play the audio from the Kinect Service on with the Windows client, which I assume you're using.  You need a library that can read the audio format directly, as all you're getting back is an array of bytes of sound data.  Take a look at NAudio or something similar.  Unfortunately I don't have any sample code or examples for you to work from....

Oct 25, 2012 at 9:34 AM
Edited Oct 25, 2012 at 9:37 AM

Thank you for you answer peek. It is working with Naudio and the code that follow :

 

WaveOut wo = new WaveOut();
WaveFormat wf = new WaveFormat(16000,1);
BufferedWaveProvider mybufferwp = null;


 audioClient.AudioFrameReady += audioClient_AudioFrameReady;

void audioClient_AudioFrameReady(object sender, AudioFrameReadyEventArgs e)
        {
            
            if (mybufferwp != null  )
            {
                mybufferwp.AddSamples(e.AudioFrame.AudioData, 0, e.AudioFrame.AudioData.Length);
            }
        }

 private void ecoute_Click(object sender, RoutedEventArgs e)
        {
            mybufferwp = new BufferedWaveProvider(wf);
            mybufferwp.BufferDuration = TimeSpan.FromMinutes(5);
            wo.Init(mybufferwp);
            wo.Play();
        }
        private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs e)
        {
            if (client.isConnected) client.Disconnect(); //Disconnects if the 
            //client is connected, closing the communication thread
            if (wo.PlaybackState == PlaybackState.Playing)
            {
                wo.Stop();
                wo.Dispose();
                wo = null;
              
            }
        }

Oct 25, 2012 at 3:59 PM

Great work!  I'll point people to this thread if they have the same question in the future.  It has come up a few times.  Thanks for sharing your code!

Mar 30, 2013 at 12:44 PM
This is a great work.
However, I found that the response is very slow. Any ideas for the audio compression?

Thanks
Adam
May 16, 2014 at 6:20 PM
Great work! It saved a lot of time for me! Thanks a lot :D
Sep 1, 2014 at 6:11 PM
Edited Sep 1, 2014 at 6:21 PM
Hello,
i had audio compression with nspeex
the audio response is good but i have a probleme with the video...
The video lag a lot and is not synchro with the audio... i don't know why ...i can't a a good video codec usable on c# if someone can help....

here is my code this the audiolistener that i have modified :
// (c) Copyright Microsoft Corporation.
// This source is subject to the Microsoft Public License (Ms-PL).
// Please see http://go.microsoft.com/fwlink/?LinkID=131993 for details.
// All other rights reserved.

using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Kinect;
using NSpeex;
using NAudio.Wave;
using NAudio.CoreAudioApi;
using System.Collections.Generic;

namespace Coding4Fun.Kinect.KinectService.Listeners
{
    public class AudioListener : KinectListener
    {

        
        private bool _running;

        Thread t;

        

        public AudioListener(KinectSensor kinect, int port)
        {
            VerifyConstructorArguments(kinect, port);

            this.Kinect = kinect;
            this.Port = port;
            Kinect.AudioSource.AutomaticGainControlEnabled = false;
            Kinect.AudioSource.BeamAngleMode = BeamAngleMode.Adaptive;
            Kinect.AudioSource.EchoCancellationMode = EchoCancellationMode.CancellationAndSuppression;
            Kinect.AudioSource.NoiseSuppression = true;
            
            t = new Thread(AudioThread) { IsBackground = true };
            t.Start();
        }


        
        public byte[] EncodeAudio(byte[] rawData)
        {
            var encoder = new SpeexEncoder(BandMode.Wide); //input as wideband (16 kHz)
            encoder.Quality = 8;  //Set the encoding quality (0-10)



            var encodedData = new List<byte[]>(); // on cree une list de tableau d octet ( de 0 à 255 )

            var inDataSize = rawData.Length / 2;// variable qui contient la taille des donné a encodé
            inDataSize = inDataSize - inDataSize % encoder.FrameSize; // taille des donnéé a à encodé auquel on enleve un peu 
            var inData = new short[inDataSize]; //tableau d entier signé -32 000 à 32000 de la taille des donné a fournir a la fonction d'encondage ?

            // System.Array.Copy(rawData, inData, rawData.Length); // on copie un tableau qui contient des d'entier de 0 à 255 dans un tableau qui peut contenier des entier de - 32000 à + 32000 sa taille vaut la longueur des donné a encodé  


            for (var index = 0; index < inDataSize; index++)
            {
                inData[index] = BitConverter.ToInt16(rawData, index * 2);
            }
            byte[] encoded = new byte[3204];
            int length = encoder.Encode(inData, 0, inData.Length, encoded, 4, encoded.Length);
            byte[] ien_bytes = BitConverter.GetBytes(length);
            Array.Copy(ien_bytes, encoded, 4);
            return encoded;
        }

        private void AudioThread()
        {
            _running = true;

            byte[] buffer = new byte[3200];
           

            Stream kinectAudioStream = Kinect.AudioSource.Start(); //Stream of audio being captured by the Kinect sensor.

            while (this._running)//  pendant la capture de l'audio
            {
                
                int count = kinectAudioStream.Read(buffer, 0, buffer.Length); // variable du kinect service on affecte a l'entier count le nombre total d'octet lu dans la memoire tampon buffer = audiobuffer
                
               

                    Parallel.For(0, ClientList.Count, index =>
                    {
                        




                        SocketClient sc = ClientList[index];

                        sc.Send(BitConverter.GetBytes(3204));// envois les 
                        sc.Send(EncodeAudio( buffer), 3204);
                        
                    });
                

                RemoveClients();
            }
        }


        public void ShutDownAudio()
        {
            this._running = false;
            if (null !=t)
            {
                t.Join();
                
            }
            if (null != this.Kinect)
            {
                this.Kinect.AudioSource.Stop();
            }
        }
    }
}
Here is the audio client modified with nspêex codec :
// (c) Copyright Microsoft Corporation.
// This source is subject to the Microsoft Public License (Ms-PL).
// Please see http://go.microsoft.com/fwlink/?LinkID=131993 for details.
// All other rights reserved.

using System;
using System.IO;
using System.Net.Sockets;
using Coding4Fun.Kinect.KinectService.Common;
using NSpeex;
using NAudio;
using System.Collections.Generic;



namespace Coding4Fun.Kinect.KinectService.WpfClient
{
    public class AudioClient : KinectServiceClient
    {
        public event EventHandler<AudioFrameReadyEventArgs> AudioFrameReady;
        public AudioFrameData AudioFrame { get; private set; }

        public AudioClient()
        {
            this.ThreadProcessor = AudioThread;
        }
        
        private byte[] DecodeAudio(byte[] encodedData)
        {
            var speexDec = new SpeexDecoder(BandMode.Wide, false);
            var input = encodedData;
            byte[] len_bytes = new byte[4];
            System.Array.Copy(input, len_bytes, 4);

            int dataLength = BitConverter.ToInt32(len_bytes, 0);

            byte[] actual_bytes = new byte[input.Length - 4];
            Buffer.BlockCopy(input, 4, actual_bytes, 0, input.Length - 4);
            var outdata = new byte[3200];
            short[] decoded = new short[3200];

            speexDec.Decode(actual_bytes, 0, dataLength, decoded, 0, false);


            Buffer.BlockCopy(decoded, 0, outdata, 0, 3200);

            return outdata;
        }

        


        private void AudioThread()
        {
            try
            {
                NetworkStream ns = Client.GetStream();
                BinaryReader reader = new BinaryReader(ns);

                while(Client.Connected)
                {
                    int size = reader.ReadInt32(); // les 4 premier octet indique le nombre d'octet suivant a lire qui sont des donnée
                    byte[] bytes = reader.ReadBytes(size); //les données audio sont placé dans bytes elle sont lu jusqua la taille indiqué par count"

                    AudioFrameReadyEventArgs args = new AudioFrameReadyEventArgs();
                    AudioFrameData afd = new AudioFrameData();
                    

                    ;
                    afd.AudioData = DecodeAudio(bytes);
                    args.AudioFrame = afd;

                    Context.Send(delegate
                    {
                        if(AudioFrameReady != null)
                            AudioFrameReady(this, args);
                    }, null);
                }
            }
            catch(IOException)
            {
                Client.Close();
            }
        }
    }
}
Sep 1, 2014 at 6:27 PM
Peekb where are you ?