Subject: WinRT: Recording of streams and getting byte[] from image frames

Jan 8, 2013 at 4:16 PM

Hi,

First of all, thank you for maintaining this project, it's really useful.

I'm working with a Win 8 app and want to be able to save the kinect streams (rgb, depth, skeleton, audio) for later playback.

There is a working solution for this on the non-WinRT platform (kinecttoolbox.codeplex.com), but it obv doesn't run in a Win 8 app project. So I'm porting that code now and I'm stuck.

Here's how recording an image frame is done in the non-winRT project. Irrelevant code omitted. 

void kinect_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
{
    using (var frame = e.OpenColorImageFrame())
    {
        Record(frame);
    }
}
System.IO.BinaryWriter writer;
void Record(ColorImageFrame frame)
{
    byte[] bytes = new byte[frame.PixelDataLength];
    frame.CopyPixelDataTo(bytes);
    writer.Write(bytes);
}

Now, when porting, I need a substitute for the CopyPixelDataTo(byte[]) function, because it looks like it's not supported by your code. Which is where my trouble begins.

Below is your WinRT version of the kinect_ColorFrameReady() function for reference.

private async void kinect_ColorFrameReady(object sender, ColorFrameData e)
{
    InMemoryRandomAccessStream stream = new InMemoryRandomAccessStream();
    await stream.WriteAsync(e.PixelBuffer);
    stream.Seek(0);
    _colorBitmap.SetSource(stream);
    xamlImage.Source = _colorBitmap;
}

I've tried with several workarounds, but nothing works.

1. WriteableBitmapEx (imagetools.codeplex.com): it has a function to convert a WriteableBitmap to byte array which I basically swap with CopyPixelDataTo().

byte[] bytes = new byte[frame.PixelDataLength];
bytes = writeableBitmap.ToByteArray();
writer.Write(bytes);

2. The ColorFrameData has something called PixelBuffer, which I thought was what I was looking for. So I copied it to the bytes array for every frame. Because PixelBuffer has type IBuffer, it has to be converted to byte[]. I used this post for that.

byte[] bytes = new byte[frame.PixelDataLength];
System.Buffer.BlockCopy(frameData.PixelBuffer.ToArray(), 0, bytes, 0, (int)frameData.PixelBuffer.Length);
writer.Write(bytes);

Using this, the program at least didn't crash. But when replaying, I got mostly a dark video with a few top rows with pixel chaos. Reason is PixelBuffer.Length << frame.PixelDataLength, so the first part of the frame is filled with some data but the rest remains 0.

3. I've tried to make use of the IRandomAccessStream which you use in the kinect_ColorFrameReady():

byte[] bytes = new byte[frame.PixelDataLength];
await stream.ReadAsync(bytes.AsBuffer(), (uint)stream.Size, Windows.Storage.Streams.InputStreamOptions.None);
var reader = new DataReader(stream.GetInputStreamAt(0));
bytes = new byte[myMemoryStream.Size];
await reader.LoadAsync((uint)stream.Size);
reader.ReadBytes(bytes);
writer.Write(bytes);

It crashes on replay of the saved file. And it has to do with the size of the byte[] - it's using the stream's size, not PixelDataLength. But when using PixelDataLength, the program crashes on attempt to save the file.

Any suggestions?

Thanks, Anton 

Jan 9, 2013 at 3:33 PM

I found a solution. I extended your client_ColorFrameReady() function (in Coding4Fun.Kinect.KinectService.Samples.WinRTXaml.MainPage.xaml.cs) and used WriteableBitmapEx (imagetools.codeplex.com) for the AsStream() extension.

    private readonly WriteableBitmap _colorBitmap = new WriteableBitmap(1, 1);
    private async void kinect_ColorFrameReady(object sender, ColorFrameData e)
    {
        InMemoryRandomAccessStream stream = new InMemoryRandomAccessStream();
        await stream.WriteAsync(e.PixelBuffer);
        stream.Seek(0);
        _colorBitmap.SetSource(stream);
        xamlImage.Source = _colorBitmap; // <Image x:Name="xamlImage">

        // Added code //

        byte[] bytes = null;
        using (var pixelStream = _colorBitmap.PixelBuffer.AsStream())
        {
            pixelStream.Seek(0, SeekOrigin.Begin);
            bytes = new byte[pixelStream.Length];
            await pixelStream.ReadAsync(bytes, 0, (int)_colorBitmap.PixelBuffer.Length);
        }
        Record(e, bytes);
    }

And the corresponding function for recording

    System.IO.BinaryWriter writer;
    public void Record(ColorFrameData frameData, byte[] bytes)
    {
        // Write header stuff using frameData
        ...
        // Write data
        writer.Write(bytes);
    }

It looks weird to have two streams where one is basically reading from the other (pixelStream through the WriteableBitmap from stream), but I got error when using only stream. I'd love to hear an explanation, but for now I'm happy to have it working. Hope it helps someone out there too.

Coordinator
Jan 10, 2013 at 4:45 AM

Unless I am totally missing what you're attempting to do...this would get the data in the PixelData IBuffer as a byte array, using the ToArray() extension method that lives in the System.Runtime.InteropServices.WindowsRuntime namespace:

 

private void client_ColorFrameReady(object sender, ColorFrameData e)
{
	// get the pixel buffer as an array of bytes
	byte[] bytes = e.PixelBuffer.ToArray();
}

That "bytes" array will contain the image data in whatever format you specified the listener to return...JPG, PNG or raw.  If you want the decoded image data, you could either return the data as raw from the listener, or decode on the client:

private async void client_ColorFrameReady(object sender, ColorFrameData e)
{
	InMemoryRandomAccessStream stream = new InMemoryRandomAccessStream();
	await stream.WriteAsync(e.PixelBuffer);
	stream.Seek(0);

	var decoder = await Windows.Graphics.Imaging.BitmapDecoder.CreateAsync(stream);
	var pixelData = await decoder.GetPixelDataAsync();
	byte[] bytes = pixelData.DetachPixelData();
}

That will decode the image from JPG/PNG to a byte array.