FFMpeg Objects - C++ & JMF bindings / Discussion / user: Performance problems

First of all, I want to thank Jos for his effort in releasing the new version of fobs.
I verified that the problem with path names under Windows (solved by Christian) has been taken care of.

Going back to my original project (doing frame processing on an mpeg2 video stream obtained from video recorded on DVD) I noticed that, when stepping over the video stream one frame at a time (using the skip function of the FramePositioningControl class or interface) the performance in terms of frames per second is very low. I am unable to get beyond 3fps on a 350*576 video stream.
I did some testing with streams that both the standard JMF and, of course, fobs can read, and I noticed that the standard JMF is roughly twice faster (on a fullscreen avi/cinepak video stream).
Of course, my goal would be to be able to step over the video stream at video rate (just as when playing the video with fobs). There is clearly a huge performance hit when asking for frames one at a time compared to asking to just play the movie.

I have written a simple program (an evolution of the SimplePlayer) that tests the performance of playing one frame at a time in terms of frames per second.
It would be interesting to compare the performance in different systems to see how it can be increased.
BTW: my system has a pentium IV 3GHz and 1GB RAM.

The program follows:

import java.awt.*;
import java.awt.event.*;
import java.io.*;

import java.net.*;
import javax.media.*;
import javax.media.format.*;
import javax.media.protocol.*;
import javax.media.control.FrameGrabbingControl;
import javax.media.control.FramePositioningControl;
import javax.media.format.VideoFormat;
import javax.media.util.BufferToImage;

public class JMFSimplePlayer2 extends Frame implements ControllerListener,WindowListener{

Player videoPlayer;
Object waitSync = new Object();
boolean stateTransitionOK = true;
int timeDelay;

DataSource dataSource; //of the capture devices

public JMFSimplePlayer2(String title) {
    super(title);
    addWindowListener(new WindowAdapter() {
      public void windowClosing(WindowEvent we) {
        dispose();
      }
    });
    play();
}

boolean play() {
    try {
      FileDialog fd = new FileDialog(this, "Select File", FileDialog.LOAD);
      fd.show();
      String filename = fd.getDirectory() + fd.getFile();
      System.out.println("Passing MediaLocator the filename: "+filename);
      MediaLocator mediaLocator = new MediaLocator("file://" + filename);

      //videoPlayer = Manager.createPlayer(mediaLocator);
      //DataSource ds = new com.omnividea.media.protocol.file.DataSource(mediaLocator);
      DataSource ds = Manager.createDataSource(mediaLocator);
      videoPlayer = Manager.createPlayer(ds);

      System.out.println("Adding controller listener");
      videoPlayer.addControllerListener(this);
      System.out.println("Starting player ...");

      FramePositioningControl fpc;
      FrameGrabbingControl fgc;


      videoPlayer.realize();
        if (!waitForState(videoPlayer.Realized)) {
            System.out.println("Failed to realize the JMF player.");
            return false;
        }

        // Try to retrieve a FramePositioningControl from the player.
        fpc = (FramePositioningControl)videoPlayer.getControl("javax.media.control.FramePositioningControl");
        if (fpc == null) {
            System.out.println("The player does not support FramePositioningControl.");
            return false;
        }

        // Try to retrieve a FrameGrabbingControl from the player.
        fgc = (FrameGrabbingControl)videoPlayer.getControl("javax.media.control.FrameGrabbingControl");
        if (fgc == null) {
            System.out.println("The player does not support FrameGrabbingControl.");
            return false;
        }


        long totalFrames=0;
        Time duration = videoPlayer.getDuration();
        if (duration != Duration.DURATION_UNKNOWN) {
            System.out.println("Movie duration: " + duration.getSeconds());
            //totalFrames = fpc.mapTimeToFrame(duration)+1;
            totalFrames = (int)(duration.getSeconds()*25.0);
            System.out.println("Movie duration: "+totalFrames+" frames.");
            if (totalFrames==FramePositioningControl.FRAME_UNKNOWN)
                System.out.println("The FramePositioningControl does not support mapTimeToFrame.");
        } else {
            System.out.println("Movie duration: unknown");
        }

        // Prefetch the player.
        videoPlayer.prefetch();
        if (!waitForState(videoPlayer.Prefetched)) {
            System.out.println("Failed to prefetch the player.");
            return false;
        }


        // Display the visual & control component if there's one.
        setLayout(new FlowLayout());
        Component vc;
        if ((vc = videoPlayer.getVisualComponent()) != null) {
            add(vc);
        }


        Buffer frame = fgc.grabFrame();
        VideoFormat format = (VideoFormat)frame.getFormat();
        BufferToImage frameConverter = new BufferToImage(format);
        setVisible(true);
        setSize(300,300);

        long basetime=0;
        double fps;
        int ifps;
        basetime = System.currentTimeMillis();

        int framecount=0;
        timeDelay = 0;
        int currentFrame=0;
        int step = +1;
        System.out.println("Entering loop...");
        while(true){
            //currentFrame = fpc.mapTimeToFrame(videoPlayer.getMediaTime());
            framecount++;
            /*
            //Delay between frames

                try {Thread.sleep(timeDelay); }
                catch (InterruptedException e){}
            */
            //System.out.println((currentFrame+1)+"/"+totalFrames);
            fps =1000.0*(framecount-1)/(double)(System.currentTimeMillis()-basetime);
            ifps = (int)fps;
            //System.out.println(ifps+"fps, delay="+timeDelay+"ms");
            //setTitle(ifps+"fps, delay="+timeDelay+"ms"+"--"+(currentFrame+1)+"/"+totalFrames);
            setTitle(ifps+"fps"+"--"+(currentFrame+1)+"/"+totalFrames);
            fpc.skip(step);
            if(step==+1){
                if(currentFrame == totalFrames-1) step = -1;
            }
            else
                if(currentFrame == 0) step = +1;

            currentFrame +=step;


        }

        //setVisible(false);
        //dispose();


    }
    catch (Exception e) {
      System.out.println(e.toString());
    }

    return true;
}

/** Block until the player has transitioned to the given state. */
    boolean waitForState(int state) {
        synchronized (waitSync) {
            try {
                while (videoPlayer.getState() < state && stateTransitionOK)
                waitSync.wait();
            } catch (Exception e) {}
        }
        return stateTransitionOK;
    }

void stop() {
    if (videoPlayer!=null) {
      videoPlayer.stop();
      videoPlayer.deallocate();
    }
}

/** Controller Listener */
public void controllerUpdate(ControllerEvent evt) {

        if (evt instanceof ConfigureCompleteEvent ||
        evt instanceof RealizeCompleteEvent ||
        evt instanceof PrefetchCompleteEvent) {
            synchronized (waitSync) {
            stateTransitionOK = true;
            waitSync.notifyAll();
        }
        } else if (evt instanceof ResourceUnavailableEvent) {
            synchronized (waitSync) {
                stateTransitionOK = false;
                waitSync.notifyAll();
            }
        } else if (evt instanceof EndOfMediaEvent) {
            videoPlayer.setMediaTime(new Time(0));
            //p.start();
            //p.close();
            //System.exit(0);
        } else if (evt instanceof SizeChangeEvent) {
        }
    }

public void windowActivated(WindowEvent we) {}
public void windowClosed(WindowEvent we) {}
public void windowClosing(WindowEvent we) {}
public void windowDeactivated(WindowEvent we) {}
public void windowDeiconified(WindowEvent we) {}
public void windowIconified(WindowEvent we) {}
public void windowOpened(WindowEvent we) {}

public static void main(String[] argv) {
JMFSimplePlayer2 myFrame = new JMFSimplePlayer2("Java Media Framework Project");
}

}

I also have this issue. In my program, a slider is provided to allow the user to track the frames back and forth. I DO have a codec which is used to access each individual frame which works great when the video is played normally. But when the video is not playing and the user drags the slider (which calls setMediaTime on the processor), there is a significant delay (several seconds) before the frame is displayed, which is not the case without FOBS.

Is there a way to quickly seek to a frame from the native codec, or is this a limitation of the FFMPEG library? I understand the idea of using the avProcess() function directly, but that will just return the next frame sequentially, and not a frame for a given time, right?

Let me finish by saying thanks again, Jos! I know you often only hear the negatives, so I'd like to add that the new release works beautifully and allows me to open files for processing in my program that weren't possible before. You've done an excellent job!

Steve

Performance problems

Forums

Help

Performance problems

Performance problems

Forums

Help

Performance problems document.SUBSCRIPTION_OPTIONS = { "thing": "topic", "subscribed": false, "url": "subscribe", "icon": { "css": "fa fa-envelope-o" } };

Performance problems