WAR [Corrupt JPEG data: ..........

Dom
2012-07-18
2013-05-30
  • Dom
    Dom
    2012-07-18

    Firstly thank you for all the effort and work put into this software - its much appreciated.

    I'm testing mjpg-streamer out for network enabling various webcams I have around the house for CCTV purposes and recording the streams via Zoneminder (possibly by Motion if I can't get Zoneminder motion detection working to a decent standard).

    My webcams are Logitch C510 - fantastic little cameras and support MJPEG compression.

    I can capture and stream in both yuv and mjpg formats - obviously the mjpeg if preferred as it has virtually no CPU overhead.

    The problem I have is that when I stream with mjpeg enabled on the camera, my syslog fills up with the following :

    Jul 17 15:35:14 er1402 zmc_m6[4552]: WAR [Corrupt JPEG data: 1 extraneous bytes before marker 0xd6]
    Jul 17 15:35:15 er1402 zmc_m6[4552]: WAR [Corrupt JPEG data: 3 extraneous bytes before marker 0xd1]
    Jul 17 15:35:16 er1402 zmc_m6[4552]: WAR [Corrupt JPEG data: 1 extraneous bytes before marker 0xd1]
    Jul 17 15:35:17 er1402 zmc_m6[4552]: WAR [Corrupt JPEG data: 2 extraneous bytes before marker 0xd0]
    Jul 17 15:35:18 er1402 zmc_m6[4552]: WAR [Corrupt JPEG data: 1 extraneous bytes before marker 0xd6]
    Jul 17 15:35:19 er1402 zmc_m6[4552]: WAR [Corrupt JPEG data: 2 extraneous bytes before marker 0xd7]
    Jul 17 15:35:20 er1402 zmc_dvideo0[1869]: INF [Monitor-2: 36000 - Capturing at 9.90 fps]
    Jul 17 15:35:20 er1402 zmc_m6[4552]: WAR [Corrupt JPEG data: 1 extraneous bytes before marker 0xd0]
    Jul 17 15:35:21 er1402 zma_m2[1873]: INF [Monitor-2: 36000 - Processing at 9.90 fps]
    Jul 17 15:35:21 er1402 zmc_m6[4552]: WAR [Corrupt JPEG data: 2 extraneous bytes before marker 0xd5]
    Jul 17 15:35:22 er1402 zmc_m6[4552]: WAR [Corrupt JPEG data: 2 extraneous bytes before marker 0xd6]
    Jul 17 15:35:23 er1402 zmc_m6[4552]: WAR [Corrupt JPEG data: 2 extraneous bytes before marker 0xd7]/code]
    If I stream by capturing from the camera in yuv mode then it works perfecly (albeit with the extra load on the camera server).
    Any ideas ? Is this the Logitech at fault or something else ? I've tried it with two different C510 webcams.
    Ta in advance
    Dom
    
     
  • Well I have faced with the same issue with a Philips SPC1030NC webcam. In JPG mode the frames came directly from the V4L2 API which relies on the data recieieved from the kernel modules. Unfortunatelly I have no camera where the issue can be reproduced to debug it.

    If you need it I can create a workaround to it.

     
  • Dom
    Dom
    2012-07-18

    I've found a few possible solutions so far …….

    1. Recompile libjpeg and remove the bit where it moans about the corrupt data
    2. Turn off Zoneminder logs
    3. Use yuv and use extra processing power
    4. Use the application "motion" instead of Zoneminder

    I've only had a quick play with motion this morning but it's perfectly happy to accept the mjpg stream from mjgp-streamer and there are no errors in any logs to show its having any issues. Having had a play with motion, I actually prefer it to Zoneminder - it seems to have a better motion detection system, uses less overhead processing power and hasn't argued with me at all yet.

    I may well just ditch ZM and write my own small system using motion as my detection/recording system.

     
  • In our case there were some rubbish bytes after the end of the JPG image. This data came directly from the kernel. It is relatively easy to workaround because the image should alays ends with the EOI (End Of Image) marker (FF D9) f I remember well. In that case if the buffer not ends with these bytes we can seek back for a few bytes to determine the length of the rubbish at the end of the image. Anyway I have used the Qt framework QImage class for reading image, what could decormpress this image without any problem.

     
  • Dom
    Dom
    2012-07-18

    would a dump of the stream using wget be of any use for you ? I've got limited Linux knowledge but I'm quite happy to grab a few mb of stream and host it up for your perusal ?

     
  • Hello,

    Yes please create a dump just to see that your problem is the similar to mine.

    Create a file called nc_request.txt with the following contents:
    GET /?action=stream HTTP/1.0

    then run the following:
    nc context.phys.unideb.hu 8080 < nc_request.txt > nc_out.bin

     
  • And of course please replace the context.phys.unideb.hu with your streamer's IP address.

     
  • Dom
    Dom
    2012-07-19

    Hi - the NC command just creates a zero byte nc_out.bin file unfortunately :(

    I'm using Ubunto 11.04 if that makes any difference.

     
  • Have you replaced the hostname with your server's address?

     
  • Have you replaced the hostname with your server's address?

     
  • Dom
    Dom
    2012-07-19

    Yes - well I think I'm doing it right ……..

    dom@er1402:~$ cat nc_request.txt
    GET /?action=stream HTTP/1.0
    dom@er1402:~$ nc 192.168.1.210 8080 < nc_request.txt > nc_out.bin
    

    nc_out.bin contains zero bytes

    I can access 192.168.1.210 from another machine and the mjpg-streamer is running fine

     
  • Ahh I have forgotten to mention that you should hit enter at the line end in the nc_request file

     
  • Dom
    Dom
    2012-07-19

    Here you go - http://www.picbits.co.uk/nc_out.bin

    Can you let me know when you've got a copy and I'll remove it from the server ? Thanks

     
  • Done. I will try to analyze it asap.

     
  • Dom
    Dom
    2012-07-19

    Thanks - no desperate hurry as I'm quite liking "motion" but obviously if I'm having this problem then others probably will too.

    I've spotted a couple of posts on the Logitech forum from people saying they are having a similar problem with the same warnings although there is no mention of mjpg-streamer so it may just be that the webcams spit out non standard frames in mjpeg mode.

    If you need any other logs or even want to sort out a few hours where I'll open my firewall and let you live stream the images then let me know - its always good when you can replicate a problem.

     
  • WonderKidder
    WonderKidder
    2012-08-10

    logitech quickcam pro 9000 - Corrupt JPEG data:

    Very pleased to have found this thread. I also had the same issue with a Logitech Quickcam Pro 9000 and Zoneminder. Of course the JPEG errors are not occurring when I pass -y for YUV mode, however this does have quite a performance impact even on a quad core server.

    I should be very grateful if somebody could add a feature to mjpg_streamer so that it validates the JPEG images received from kernel/v4l and does not pass on the corrupt ones. It appears that when Zoneminder is fed a corrupted data frame it breaks various parts of the software.

    Is there anything I can do to assist with this development? I would also be happy to provide access to the camera/data files if this would be helpful.

    Kind regards,

     
  • Dom
    Dom
    2012-08-11

    I've been running "motion" with mjpg-streamer for the past couple of weeks without issue now using mjpg compression on my Logitech - I'm actually liking it far more than Zoneminder (and no errors in the logs) - I'll probably ditch ZM now I've found new confidence in motion - it takes a bit of getting used to and setting up but its rock solid software so far.

     
  • bbdom!

    Finally I have hooked up a small C program which checks the SOI and EOI markers in the JPG stream:
    https://dl.dropbox.com/u/4295670/corrupt_marker_finder.c

    I have found 71 images in your file, but there were not any rubbish bytes after the images neither before them.
    Are you sure that the sample stream contains corrupt frames?

    If yes, your situation is different what I had with my Philish webcam.
    In this case further investigation of the corrupted JPG data is needed.

    Thanks,
    Miklós

     
  • Hmm. After looking a bit deeper to the JPG format I have realized that there are many markers in the file. Writing a code from scratch to find them should not worth trying. If we are lucky than the libjpg would have some kind of  image check feature. I will look after it.

     
  • Dom
    Dom
    2012-08-11

    Thanks for looking at it :-)

    It appears to be extremely stable in the configuration I'm using it in at the moment - I'm grabbing in mjpg and analysing it with "Motion" which all works fine so although the problem might still be there, for my application its irrelevent as I've dropped ZM - I didn't get on with its motion detection in the end.

    I'm quite happy to sort out some more recroded files at some point in the future or even put it on a machine and let you drag down live streams but at the moment I've got a couple of weeks of heavy building to do as I'm sorting out a new workshop :)

     
  • WonderKidder
    WonderKidder
    2012-08-14

    Thank you for the corrupt marker finder program. I am running in YUV mode however will switch to MJPEG mode overnight and check the output file for corrupt frames. The issue I forsee is that if the corrupt frames occur several hours into the recording this will be a very large binary file and I have no idea how to truncate it to show only the garbage frames. Perhaps I should write a small script to stop the capture and check the output for bad frames, and if none found overwrite the file, then check again a few minutes later repeat until bad frames found.

    However martonmiklos seems to suggest that it is not worthwhile to write program to check integrity of JPG images (reinvent the wheel?) and instead we should look for function in libjpeg. Does this mean that I should not try to capture the garbage frames as they are not helpful?

    Is this something we should raise with the libjpeg maintainers?

     
  • Dom
    Dom
    2012-08-14

    My setup outputs the corrupt frames almost immediately - I only needed 10 seconds or so before my syslog started filling up with the warnings.