[Pyobjc-dev] Again: QTKit CVImageBufferRef - howto access the data

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi there !

i'm still struggling with my QTKit capture module using pyobjc and to  
retrieve
actual image data (which should finally end up in a numpy array). i'm  
having
trouble to get to the actual data from the CVImageBufferRef.

after the fix from Ronald i'm able to access the CVImageBufferRef as  
hinted
in most of the objective-c examples like:

         @objc.signature('v@:@{__CVBuffer=@}@@')
         def  
captureOutput_didOutputVideoFrame_withSampleBuffer_fromConnection_ 
(self, captureOutput,
			videoFrame, sampleBuffer, connection):
             videoFrame = videoFrame[0]
             #print videoFrame, sampleBuffer
             print  
sampleBuffer.formatDescription().localizedFormatSummary()

             CVPixelBufferRetain(videoFrame)
             buffer_to_release = self.current_buffer
             self.current_buffer = videoFrame
             if buffer_to_release is not None:
                 CVPixelBufferRelease(buffer_to_release)

             print "colorspace:",  
CVImageBufferGetColorSpace(self.current_buffer)
             print "displaysize",  
CVImageBufferGetDisplaySize(self.current_buffer)

             print "width:", CVPixelBufferGetWidth(self.current_buffer)
             print "height:",  
CVPixelBufferGetHeight(self.current_buffer)
             print "pixel_format_type:",  
CVPixelBufferGetPixelFormatType(self.current_buffer)
             print "planar:", CVPixelBufferIsPlanar(self.current_buffer)
             #print "extended_pixels:",  
CVPixelBufferGetExtendedPixels(self.current_buffer)
             print "datasize:",  
CVPixelBufferGetDataSize(self.current_buffer)
             print "bytes_per_row:",  
CVPixelBufferGetBytesPerRow(self.current_buffer)
             print "address:",  
CVPixelBufferGetBaseAddress(self.current_buffer)
             print "attachements:",  
CVBufferGetAttachments(self.current_buffer,  
kCVAttachmentMode_ShouldPropagate)

             #length = CVPixelBufferGetDataSize(self.current_buffer)
             ptr = CVPixelBufferGetBaseAddress(self.current_buffer)

at this point i'm able to access the ptr's data using the __getitem__  
accessor for each
byte one by one - but this is horribly slow and will not work for my  
solution to capture
realtime image streams into numpy arrays for further processing.

the examples suggest to convert the BufferRef to a CIImage but this  
statement:

             # convert the CVImageBuffer into an NSImage
             cimg =  
CIImage.alloc().initWithCVImageBuffer_(self.current_buffer)

fails with:
2009-06-14 17:21:43.587 Python[1044:b803] *** Ignoring exception:  
<type 'exceptions.ValueError'>: depythonifying 'pointer', got  
'CVPixelBufferRef'

which i cannot really interpret nor fix.

i've tried several different ways of passing the ref but none did  
something useful.

on the other hand it would be sufficient for me to get real access to  
the ptr using ctypes and fill the numpy array directly with the
char buffer using some code like this (works fine for libdc1394  
accessed with ctypes):

             #func = ctypes.pythonapi.PyBuffer_FromMemory
             #func.restype = ctypes.py_object

             # XXX howto convert pyobjc varlist to ctypes pointer
             #buffer = func(ptr, length)
             #ret = numpy.frombuffer( buffer, numpy.uint8 )
             #ret = numpy.array(ptr.as_tuple(length))

maybe one of you has a hint how to fix the ValueError or access the  
ptr's data directly using ctypes.

any help would be greatly appreciated - i'm fairly stuck now ;-)

thanks!

Ulrich

Am 22.04.2009 um 07:57 schrieb Ulrich Eck:

> Hi Ronald,
>
> thanks for your immediate action/help !!!
>
> i updated pyobjc-svn today and i can access the CVImageBufferRef now
> (still struggling how to use it) but my initial problem is fixed now.
>
> when i have a working example, i'll post it to the list for the  
> records.
>
> thanks and have a nice day!
>
> Ulrich
>
> Am 21.04.2009 um 17:05 schrieb Ronald Oussoren:
>
>> Ulrich,
>>
>> The CoreVideo bindings in subversion were incorrect until a moment
>> ago. I've just checked in an updated version of the Quartz bindings
>> (r2166), those fix most of the problems with CoreVideo.
>>
>> That said, I haven't used the CoreVideo bindings in real code yet
>> and there aren't any Python examples using CoreVideo either and this
>> means there may be issues left.  Other than creating CVPixel buffers
>> in Python code (CVPixelBufferCreate and friends) the bindings should
>> be complete though.
>>
>> To answer your question:
>>
>> First of all: use "import Quartz" in your module to ensure that the
>> CoreVideo bindings are loaded, otherwise
>>  PyObjC won't load the right framework bindings.
>>
>> PyObjC should then automaticly deduce the correct signature for the
>> delegate method, not doing so would be a bug in the QTKit bindings.
>>
>> BTW. Which version of PyObjC are you using?
>>
>> Ronald
>>
>> On 20 Apr, 2009, at 3:23, Ulrich Eck wrote:
>>
>>> hi,
>>>
>>> i spend a couple of days to find out how to do video capturing
>>> using the current pyobc QTKit framework wrappers and run into a
>>> problem...
>>>
>>> i couldn't find out, how to write the signature for the selector of
>>> my delegate for QTCaptureDecompressedVideoOutput
>>> with the following definition:
>>>
>>> - (void)captureOutput:(QTCaptureOutput *)captureOutput
>>> didOutputVideoFrame:(CVImageBufferRef)videoFrame
>>>      withSampleBuffer:(QTSampleBuffer *)sampleBuffer
>>> fromConnection:(QTCaptureConnection *)connection
>>>
>>> i tried several signature definitions like:
>>>
>>> @objc.signature('v@:@{__CVBuffer=*}@@')
>>> @objc.signature('v@:@{__CVBuffer=@}@@')
>>> @objc.signature('v@:@^{__CVBuffer=}@@')
>>> @objc.signature('v@:@@@@')
>>>
>>> but they all end up in either a PyObcPointer created warning or a
>>> NSCFType for videoStream, which seems not be usable
>>> at all (can i cast to an CVImageBufferRef somehow ?).
>>>
>>> output example without signature:
>>> PyObjCPointer created: at 0x29f4590 of type {__CVBuffer=}@@
>>> got frame: <QTCaptureDecompressedVideoOutput: 0x2981ac0>
>>> <PyObjCPointer object at 0x25c8b78> <QTSampleBuffer: 0x29f3600>
>>> <QTCaptureConnection: 0x2982510 vide 2vuy>
>>>
>>> output example for the "@objc.signature('v@:@@@@')" signature:
>>> got frame: <QTCaptureDecompressedVideoOutput: 0x2982010> <NSCFType:
>>> 0x29f7310> <QTSampleBuffer: 0x29f5bf0> <QTCaptureConnection:
>>> 0x2982a60 vide 2vuy>
>>>
>>> using "@objc.signature('v@:@{__CVBuffer=}@@')" results in an error
>>> message:
>>> <type 'exceptions.RuntimeError'>: Cannot create FFI CIF: 1
>>>
>>>
>>> now i'm lost ... has anyone here on the list an idea how to convice
>>> the pyobjc wrapper to pass me a valid object/reference to the
>>> CVImageBufferRef or what to do with the NSCFType to get to the
>>> buffer -- as shown in the example below .. i want to extract the
>>> image-data
>>> from the captured video (possibly as string-buffer reference for
>>> further use with numpy+ctypes).
>>>
>>> any help is greatly appreciated !!
>>>
>>> thanks for your help
>>>
>>> cheers Ulrich
>>>
>>>
>>> here is a minimal example of my code (created a simple pyobjc app
>>> in xcode and this is the main app-delegate):
>>> i can also provide the whole example for tests if needed.
>>>
>>>
>>> ############################
>>> from Foundation import *
>>> from AppKit import *
>>> from QTKit import *
>>>
>>> k24RGBPixelFormat             = 0x00000018    # 24 bit rgb
>>> k32ARGBPixelFormat            = 0x00000020    # 32 bit argb    (Mac)
>>>
>>> class qtkit_captureAppDelegate(NSObject):
>>>   def applicationDidFinishLaunching_(self, sender):
>>>       NSLog("Application did finish launching.")
>>>
>>>
>>>       session = QTCaptureSession.alloc().init()
>>>
>>>       device =
>>> QTCaptureDevice.defaultInputDeviceWithMediaType_(QTMediaTypeVideo)
>>>       if device is None:
>>>           print "failed to open device."
>>>           return
>>>
>>>       device.open_(None)
>>>
>>>       input = QTCaptureDeviceInput.alloc().initWithDevice_(device)
>>>       session.addInput_error_(input, None)
>>>
>>>       output = QTCaptureDecompressedVideoOutput.alloc().init()
>>>
>>>       settings = dict(kCVPixelBufferWidthKey=640.0,
>>>                       kCVPixelBufferHeightKey=480.0,
>>>
>>> kCVPixelBufferPixelFormatTypeKey=k32ARGBPixelFormat,
>>>                       )
>>>
>>>       output.setPixelBufferAttributes_(settings)
>>>       session.addOutput_error_(output, None)
>>>
>>>       class logger(NSObject):
>>>
>>>           current_buffer = None
>>>
>>>           # Declaration:
>>>           #- (void)captureOutput:(QTCaptureOutput *)captureOutput
>>> didOutputVideoFrame:(CVImageBufferRef)videoFrame
>>>           #       withSampleBuffer:(QTSampleBuffer *)sampleBuffer
>>> fromConnection:(QTCaptureConnection *)connection
>>>
>>>           #@objc.signature('v@:@{__CVBuffer=*}@@')
>>>           #@objc.signature('v@:@{__CVBuffer=@}@@')
>>>           #@objc.signature('v@:@^{__CVBuffer=}@@')
>>>           @objc.signature('v@:@@@@')
>>>           def
>>> captureOutput_didOutputVideoFrame_withSampleBuffer_fromConnection_
>>> (self, captureOutput, videoFrame, sampleBuffer, connection):
>>>               NSLog("got frame: %s %s %s %s" % (captureOutput,
>>> videoFrame, sampleBuffer, connection))
>>>
>>>               # always have only one frame retained
>>>               #CVBufferRetain(videoFrame)
>>>               #buffer_to_release = self.current_buffer
>>>               #self.current_buffer = videoFrame
>>>               #if buffer_to_release is not None:
>>>               #    CVBufferRelease(buffer_to_release)
>>>
>>>               # convert the CVImageBuffer into an NSImage
>>>               #cimg = CIImage.initWithCVImageBuffer_(test)
>>>               #rep = NSCIImageRep.initWithCIImage_(cimg)
>>>               #nimg = NSImage.alloc().initWithSize_(rep.size())
>>>               #nimg.addRepresentation_(rep)
>>>
>>>
>>>       cb = logger.alloc().init()
>>>       output.setDelegate_(cb)
>>>       session.startRunning()
>>>
>>>       self._data = (session, device, input, output, cb)
>>>
>>> ############################
>>>
>>> ------------------------------------------------------------------------------
>>> Stay on top of everything new and different, both inside and
>>> around Java (TM) technology - register by April 22, and save
>>> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
>>> 300 plus technical and hands-on sessions. Register today.
>>> Use priority code J9JMT32. http://p.sf.net/sfu/p_______________________________________________
>>> Pyobjc-dev mailing list
>>> Pyo...@li...
>>> https://lists.sourceforge.net/lists/listinfo/pyobjc-dev
>>
>
>
> ------------------------------------------------------------------------------
> Stay on top of everything new and different, both inside and
> around Java (TM) technology - register by April 22, and save
> $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
> 300 plus technical and hands-on sessions. Register today.
> Use priority code J9JMT32. http://p.sf.net/sfu/p
> _______________________________________________
> Pyobjc-dev mailing list
> Pyo...@li...
> https://lists.sourceforge.net/lists/listinfo/pyobjc-dev