Majorca workflow

  • Phil

    I'm back from Majorca & I'm busy rendering the first video I made which looks wonderful. I still have the skipping problem with the new laptop filming at 15fps at 80% quality & 90/48mhz. The colour looks great though & I should get some images & a demo clip up on a page soon.

    I found it hard to focus the camera with it being outside & the laptop inside with the strong sun. I think next time I need to use a black cloth over my head & laptop to see the image clear! 

    If you film more then 18minutes the file is closed but it can be still opened with Virtualdub. so you have to quickly make a new file if the Xdialogue box closes when it reaches 2gb!

    Im exporting all the 2gb clips as RGB raw files to a big formatted hard drive & then I can edit in Vegas5 at 1280x1024 & make a Xvid mpeg4 with it without loss of quality.

    If we can figure out a fix for the skipping then this will be a great camera for movie making!

    • Hi Phil,

      I was very busy with the new model 333 camera (got some Ogg/Theora 1280x1024x30fps clips from it) and did not do much about 313. But still I just made a small upgrade (but tested only on few cameras) - color overflow protection - it seems to fit easily in the 98% used FPGA (I first tried that on 333). So you may download elphel313-6.2.23-flash.tar.bz2 and drop this file on "flashit" icon on Live CD you have. There will be no more green on red lights :-)


      • Thanks Andrey

        Do you think the skipping can be solved via software? If this can be fixed I can really start filming properly with the 313. The image is absolutely stunning with the saturation set at the default! I'm booked in to film a big sport event on the 17th April so I'm begging on hands & knees to anyone for a solution to this or I will have to use the terrible Canon GL2!

        How about Sergeys idea of typing stuff into the root console & then using telnet (whats this)?

        Lance Armstrong will open a cycling exhibit at Madame Tussuad's in New York with our software. It would be great to use the 313 video for this.

        • I found next comands, wich may help to avoid skips with our Live CD
          Switch to superuser: Ctrl+Alt+F2

          #echo 30 5 0 0 5 3000 80 10 0 > /proc/sys/vm/bdflush
          #renice 19 `pidof bdflush`
          #echo 524284 > /proc/sys/net/core/rmem_max

          Return to GUI: Alt+F5

        • Phil,

          the problem seems to be that during writing on disk system gets busy and misses the incoming stream. It is probably possible to solve the problam adjusting system parameters on both sides - network input buffer and some disk operation parameters - I don't know myself aboit the detailes much. And as I understand Sergey is suggesting some particular tuning details. If some will work he will be able to incorporate them in the future releases.

          To enter manually parameters you need the "root shell" - Sergey wrote instructions how to get there and back, or you can open the root shell as a window on the desktop selecting the item from menu under Knoppix (Penguin icon on the teask bar). The "#" prompt indicates that you are in superuser (root) mode and are allowed to issue system-level commands.
          Telnet is not needed for this, it is used to get remote shell, such as issue commands to the GNU/Linux running in the camera, not the Gnu/Linux running on your PC (Knoppix).

          • Phil

            Sergeys post worked!  I made a 1minute long full res 1280x1024 90% 22fps clip with no skipping. even mplayer could show smooth video!

            But the second clip was really bad, it was as if the memory was full. mplayer showed very jerky video & you could see the fps hanging in that Xdialogue box thingy!

            i feel you guys are getting close!


            • Hi Phil.

              It can make sense to look what processes are running and how memory is used. You may try first running it dynamically - just open console window and enter command
              and watch this window while making videos - maybe you'll see something different in the cases when everything is good or bad.
              Then you can put a snapshot of the running processes into a file by the command
              top -b -n 1 > topres
              "topres" will be the result file, then post this file or email it to Sergey.


    • Phil

      Thanks for this. Is there any possibility of a cure for the skipping?  Does the 333 have this also & what kind of quality is this Ogg codec or rather do you think its visually better then the Mjpeg?

    • Phil,

      I do not have a solution for skipping yet - I believe the solution should implement a really big buffer for the incoming Ethernet packets so data will not be lost if the computer will fail to service interrupts in time.

      As for the new model 333 camera - for your application the problem will stay the same :-( (at least now) - The new camera is faster and can handle full 30fps for 1280x1024, but the current implementation of the Theora encoder is designed for the fixed camera where it can benefit from the virtually unchanging background (there is no motion compensation yet). But still encoding of the quantized DCT coefficients (all the losses in quality are before that) is somewhat better in Theora than in baseline JPEG, so the bitrate for the same quality INTRA frames only (similar to MJPEG) of Theora should be lower.


    • Phil

      Just a quick thought before I (attempt) to get that file.  Would more memory solve this? Its the one thing Ive not tried to upgrade, That liveCD must use a lot of memory, I have just 512mb in the laptop.


    • Phil

      Mplayer uses about 40% cpu
      Mencoder uses about 15% cpu
      Other stuff 5%

      Memory usage seems to be less then 10%

      I just did a search on some of the commands from Sergey & it seems others have had problems capturing DV with linux & dropped frames.

      Someone recommended
      echo 0 0 0 0 500 3000 60 0 0 > /proc/sys/vm/bdflush

      I did this & it worked also perfectly for a 2minute skip free video the second was a little Skippy so I started playing with the numbers & this string seems to work for multiple files. (Ive no idea what I am doing!)

      #echo 30 5 0 0 5 3000 60 0 0 > /proc/sys/vm/bdflush

      this line seems to be the one that fixes the skipping. I just need a way to deal with the 2gb limit now. I will start a new thread for this!

      Thanks all Phil

      • This numbers are documented in the kernel source.
        Below fragment from this documentation file "linux/Documentation/sysctl/vm.txt"

        This file controls the operation of the bdflush kernel
        daemon. The source code to this struct can be found in
        fs/buffer.c. It currently contains 9 integer values,
        of which 6 are actually used by the kernel.

        nfract:         The first parameter governs the maximum
                        number of dirty buffers in the buffer
                        cache. Dirty means that the contents of the
                        buffer still have to be written to disk (as
                        opposed to a clean buffer, which can just be
                        forgotten about). Setting this to a high
                        value means that Linux can delay disk writes
                        for a long time, but it also means that it
                        will have to do a lot of I/O at once when
                        memory becomes short. A low value will
                        spread out disk I/O more evenly, at the cost
                        of more frequent I/O operations. The default
                        value is 30%, the minimum is 0%, and the
                        maximum is 100%.

        ndirty:         The second parameter (ndirty) gives the
                        maximum number of dirty buffers that bdflush
                        can write to the disk in one time. A high
                        value will mean delayed, bursty I/O, while a
                        small value can lead to memory shortage when
                        bdflush isn't woken up often enough. The
                        default value is 500 dirty buffers, the
                        minimum is 1, and the maximum is 50000.

        dummy2:         The third parameter is not used.

        dummy3:         The fourth parameter is not used.

        interval:       The fifth parameter, interval, is the minimum
                        rate at which kupdate will wake and flush.
                        The value is in jiffies (clockticks), the
                        number of jiffies per second is normally 100
                        (Alpha is 1024). Thus, x*HZ is x seconds. The
                        default value is 5 seconds, the minimum is 0
                        seconds, and the maximum is 10,000 seconds.

        age_buffer:     The sixth parameter, age_buffer, governs the
                        maximum time Linux waits before writing out a
                        dirty buffer to disk. The value is in jiffies.
                        The default value is 30 seconds, the minimum
                        is 1 second, and the maximum 10,000 seconds.

        sync:           The seventh parameter, nfract_sync, governs
                        the percentage of buffer cache that is dirty
                        before bdflush activates synchronously. This
                        can be viewed as the hard limit before
                        bdflush forces buffers to disk. The default
                        is 60%, the minimum is 0%, and the maximum
                        is 100%.

        stop_bdflush:   The eighth parameter, nfract_stop_bdflush,
                        governs the percentage of buffer cache that
                        is dirty which will stop bdflush. The default
                        is 20%, the miniumum is 0%, and the maxiumum
                        is 100%.

        dummy5:         The ninth parameter is not used.

        So the default is: 30 500 0 0 500 3000 60 20 0   for 100 HZ.

    • Phil

    • Phil

      echo 30 5 0 0 5 3000 60 0 0 > /proc/sys/vm/bdflush

      Seems to be the golden number for my 2.8ghz laptop. Would it be possible to have a future live CD to somehow have this built in?  The less I have to type in when under pressure the less chance of messing up!  Ive made a few files now of upto 5minutes & Ive not seen a single skip!   Great news :-)

      Could the next firmware upgrade have 90/48mhz for the 313? I have to type &fclk=90 etc each time the camera loses power (often when its powered by a car battery & the driver keeps forgetting to leave the engine running! ).

      Thanks all again


      • Hi Phil,

        It is very easy to make clock parameters persistent so you will not need to enter them after the camera looses power.
        open the page (or different IP if you changed it) and go to "Browse file system"->etc->init.d->fpga (or directly\)
        Then change the lines
        fpcf -x 1 36
        fpcf -x 0 75

        fpcf -x 1 48
        fpcf -x 0 90

        These parameters (48MHz, 90MHz) will be used each time the camera will be booted.

    • Phil

      Thanks I did not realize it was so simple!  Is it possible to flip the image also? the new camera mount will have the camera upside down. I can flip the image in editing but If I can do it this way it would be simpler!

      Thanks again Phil

      • Phil,

        you may edit index.html file (and rename the original index.html so it will still be available if you mess up the new one)

        It is better to use ftp and external editor as some browsers can screw up the html files when editing them in the same way as configuration files - with web interface. Get to the /etc/httpd/html - that is the root for the http access (you may also experiment with /var/html (visible as http://<camera_ip>/var ) - it is a RAM-disk so no risk to ruin some functionality.

        OK, in the index html locate
        function setDefaults() {
        // common defaults:
           document.getElementById("idColor").checked= true;
           document.getElementById("idGamma").value= 0.47;
           document.getElementById("idSaturationRed").value= 2.0;
           document.getElementById("idQuality").value= 70;
           document.getElementById("idExposure").value= 2.0;
        (there was a bug earlier - 2 of flipX, mnone of flipY)

        and change them to:


        You may also change some other default parameters.
        Or make your own version of the page that is more convinient for you.


    • Phil

      Its done & working great, Thanks again


      • Hi Phil,

        We are planning to show our camera in Moscow, Russia (April 27-29 at\). Can you send us some samples of your footage to demonstrate there?


    • Phil

      Sure email me your home address & I can send a DVD with some original avi. I have to try to get something for an exhibition (the Lance Armstrong thing) which is in April also. The Majorca video has many skips but I could look for some nice frames? & send what ever I film in the next days if the sun comes out. Otherwise they will use the old DV videos from last year.

      Now the camera is working I just need the rain to stop so I can get some new 'smooth' film. 

      I have only just today noticed that the frame rate is dependant on the light available! Sometimes the camera seems to slow down & go from 22fps to below 15fps. whats going on here? Could it be something to do with the write speed of the hard drive?

      I tried to alter the start up quality setting on index.html & the 313 page became corrupted. So I did a Flash All which reset everything. I now find that I can not flip the image without messing everything up :-I 

      Regards Phil

      • Phil,

        Thank you, and the address is the same as on our web site. Can you put some leaflets about your company/products with the DVD? Maybe some of the visitors who will see the samples will become interested in your products?

        As for the editing the index.html - just copy the original and keep it under different name. A copy of the original on you PC can help too - ftp to the camera works independently of the web server, so if you screwed up the index.html - you can ftp the good one and restart the camera. And you do not need "Flash All" (we probably need to hide this option a little deeper) - it is only for those who have very old firmware (more that 1 1/2 years) - the boot block format  has changed then. The only difference compared to "flashit" is that the boot block is overwritten and it stores the camera serial number/MAC address. It is not a real problem when you have just one camera, but several in the same network will not like it to have the same MAC addresses. It is possible to restore MAC.

        Another note - as camera does not have the battery-powered clock it boots in the year 1970. Only after the first image request (that has &_time=.... in URL) it sets the clock to that of the computer (the first to connect to the camera). But that means, that the index.html is still retrieved from the 1970 and if you modified it and you are not using live CD (or the Live CD was not rebooted) the browser will decide that it has newer copy in the cache and use it instead, not the one that you've just edited.

        The frame rate depending on the light - yes it is possible if the required exposure is longer than the frame . You may verify it even without interruption of the streamer using the "?" link - it will show you the actual rate the sensor is (or was) running.

        If there is not enough light you may increase color gains - they are set near the minimal values by default.


    • Phil

      echo 30 10 0 0 5 3000 60 0 0 > /proc/sys/vm/bdflush 

      Seems more reliable (I increased the second number from 5 to 10)

      I should try this with Sergeys other two commands. It seems after reading about the third command that the buffer memory that receives the 313 signal is set fairly small 64k by default so as not to waste memory.