Menu

Incomplete stream recording

Help
Skello
2009-11-26
2013-04-25
1 2 > >> (Page 1 of 2)
  • Skello

    Skello - 2009-11-26

    Hello,

    First I'd like to say that I'm new and RTMP, but I'm pretty techy and after a day of reading documentation and trying I think I've managed to get the hangs of it.

    I discovered Flazr in my quest to find a RTMP client, which would be able to record a particular stream I'm interested in. I've tried rtmpdump and others, but without any success.

    First, I used Wireshark to analyze the traffic and get the necessary parameters. I think I got enough out to at least get started.

    I'm trying to record a stream from a Red5 server. I used the videolectures.groovy script as starting point. I get different behaviors in different Flazr versions:

    **version 0.5** (and the only one I had any success with)

    Handshake succeeds

    Playing starts

    I keep getting stuff like this:
    `00:44:36,156  INFO  - sending bytes read 614455: DirectBuffer  00:44:36,171  INFO  - video write progress: 10 seconds`

    This stops after three or four passes (the most I got were 50 seconds), then I get ping and pongs between the server and client, like this:

    `00:44:57,437  INFO  - server ping: DirectBuffer  00:44:57,437  INFO  - client pong: DirectBuffer`

    This goes on "forever," but nothing more gets recorded. The resulting file has as many seconds as FlvWriter reported.

    **version 0.7-RC1**
    I can't even get the client.bat to parse the script. When I do: client.bat etc\script.groovy (under Windows), I recieve:

    `Exception in thread "main" java.lang.RuntimeException: invalid url: etc\script.groovy 
    at com.flazr.rtmp.client.RtmpClientSession.<init>(RtmpClientSession.java:109) 
    at com.flazr.rtmp.client.RtmpClientSession.main(RtmpClientSession.java:57)`

    **version 0.7-RC2**
    When I run client.bat etc\script.groovy I get:
    `java -Xmx512m -cp conf;lib\commons-cli-1.2.jar;lib\commons-codec-1.2.jar;lib\commons-httpclient-3.1.jar;lib\flazr-0.7-RC2.jar;lib\jcl104-over-slf4j-1.4.2.jar;lib\log4j-1.2.14.jar;lib\netty-3.1.5.GA.jar;lib\slf4j-api-1.4.2.jar;lib\slf4j-log4j12-1.4.2.jar -Dflazr.log.suffix=-client com.flazr.rtmp.client.RtmpClient etc\script.groovy` 
    `01:20:18,515  INFO  - options:  
    01:20:18,953  INFO  - channel opened:  OPEN 
    01:20:19,968  ERROR  - error creating client connection: Connection refused: no further information 
    01:20:19,984  INFO  - channel closed:  CLOSED 
    01:20:19,984  INFO  - exception: Connection refused: no further information`

    I've reached a dead end or the limits of my current RTMP and Flazr knowledge. I'd appreciate any suggestions on how to proceed.

    If you require any more info, please feel free to ask.

    Thank you.

     
  • Peter Thomas

    Peter Thomas - 2009-11-27

    Just 2 things:

    - 0.7 onwards does not use groovy.  take RC-2 and just type "client" to get a help screen / list of command line options
    - read the last section on , Flazr can calculate / generate the hash by itself.  You will need to decompress the SWF and point Flazr to it.

      : http://lkcl.net/rtmp/RTMPE.txt

     
  • Skello

    Skello - 2009-11-27

    Hello Peter and thank you for responding.

    Are you implying that I should use the RTMPE protocol or the SWF verification is required for RTMP too?

    Additionally how do I specify multiple properties with this argument: `-D <property=value>`

    Can you exemplify?

     
  • Skello

    Skello - 2009-11-27

    Ok. Returning with more info from subsequent trials.

    Here is what I determined so far:

    The server only works on RTMP
    `10:33:16,751  WARN  - server does not support rtmpe! falling back to rtmp`

    SWF validation or not, it doesn't seem to even reach that step. Here is where it's stuck:
    `10:37:30,426  INFO  - channel opened:  OPEN 
    10:37:30,464  INFO  - connected, starting handshake 
    10:37:30,480  INFO  - using client version 09007C02 
    10:37:30,893  WARN  - server part 1 validation failed for type 1, will try with type 2 
    10:37:30,899  WARN  - exception: {} 
    java.lang.RuntimeException: server part 1 validation failed even for type: 2 
    at com.flazr.rtmp.RtmpHandshake.decodeServer1(RtmpHandshake.java:377) 
    at com.flazr.rtmp.RtmpHandshake.decodeServerAll(RtmpHandshake.java:324) 
    at com.flazr.rtmp.client.ClientHandshakeHandler.decode(ClientHandshakeHandler.java:60) 
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282) 
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:214) 
    at com.flazr.rtmp.client.ClientHandshakeHandler.handleUpstream(ClientHandshakeHandler.java:78) 
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345) 
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332) 
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323) 
    at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275) 
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196) 
    at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
    at java.lang.Thread.run(Unknown Source) 
    10:37:30,922  INFO  - channel closed:`

    It's intriguing that Flazr 0.5 connects though and the handshake is achieved.

     
  • Skello

    Skello - 2009-11-28

    Me again. I'm still determined to make this work as I really need this stream.

    I think I tracked the problem down, but I need your help with this Peter as I'm not Java coder.

    First of all, I should point out that this only applies to Flazr 0.5, since it is the only version which I succeeded to connect and actually start recording with. I don't know what has been left out or changed in the newer versions, but the handshake fails with the same connection parameters on the same server.

    Ok, so back to Flazr 0.5. Just to reiterate, what's happening is that it starts recording then after 30 - 40 seconds the server stops responding and there's a continuous ping pong exchange between the server and client.

    The only thing that goes wrong during the connection is this:
    `10:39:38,012  WARN  - unhandled server invoke:  
    10:39:38,015  INFO  - ignoring received packet: `

    So the server specifies a bandwidth limit and expects a response from the client. A response that never comes. So it stops sending data.

    This is clearly reflected in the Wireshark traffic, as it continues to send a Server BW (0x05) function call and also a Client BW (0x06) one. According to , 0x05 is a server response and 0x06 is a client request.

    Now this seems to be a Red5 server specific problem to some extent. Another user  on this forum that he experienced the exact same issue. "When i don't limit bandwdith on the server, everything works and it maxes out the connection and downloads the file, however when i limit the bandwidth after a while it goes into a ping loop and stops downloading," he writes. Unfortunately I don't have access to the server and I have to solve this at the client level.

    The onBWDone method seems to be well documented and discussed. Searching for it on Google will reveal many results saying that the client has to respond. An ActionScript programmer  a fix for "missing onBWDone, onFCSubscribe, etc." when writing a FLV player to work with Flash Media Server. This is what he proposes in AS3:

    `public function onFCSubscribe(info:Object):void{ 
        trace(”worked”); 
        } 
        public function play(connectionurl:String):void{ 
        _netConnection = new NetConnection(); 
        _netConnection.client = this; 
        _netConnection.connect(connectionurl); 
        }`
    "That’s it right there. _netConnection.client = this; is the fix. What it tells the server-side actionscript is “when you want to return calls, return them on my client property.” So, since 'client = this' all of the calls from FMS back to my client will run inside of 'this,'" he concludes.

    Even though this is for onFCSubscribe, it applies to onBWDone too and the 'this' seems relevant in the context of Flazr. And here's why.

    If I understood correctly, Flazr resulted from an original concept you posted . Someone using it with a Red5 server stumbled across the same onBWDone issue and  on the Red5dev list.  is what was proposed (follow link for complete explanation): 
    `setServiceProvider(this); 
    callBackProvider = this; 
    connect(host, port, app, callBackProvider, pageUrl, swfUrl, objectEncoding); 
    public void onBWDone(Object obj) 


    public void onBWDone() 

    }`  *Note the **this**

    Now I took a look at the code in RtmpDecoder.java. I believe  is the last one which applies to Flazr 0.5. Here we have:
    `case SERVER_BANDWIDTH: 
    case CLIENT_BANDWIDTH: 
    logger.info("ignoring received packet: " + packet.getHeader()); 
    break; 
    default: 
    throw new RuntimeException("unknown packet type: "  + packet.getHeader());`

    Now if I only knew Java so I could hack the code. But I don't.

    Peter, I really need your help with this. I'm doing my best to figure out what's wrong and give you proper input on the problem. Maybe it will help you fix some scenarios in the newer versions too. Or maybe it was fixed already, but not backported to 0.5, which is the only version working for me.

    Ultimately I need a flazr-0.5.jar that will handle onBWDone correctly, or in a way that the server will continue streaming. I don't mind if it takes a long time to record it. Please help!

      : http://wiki.gnashdev.org/RTMP
      : https://sourceforge.net/projects/flazr/forums/forum/932960/topic/3366006
      : http://www.johncblandii.com/index.php/2007/12/fms-a-quick-fix-for-missing-onbwdone-onfcsubscribe-etc.html
      : http://ptrthomas.wordpress.com/2008/04/19/how-to-record-rtmp-flash-video-streams-using-red5/
      : http://osflash.org/pipermail/red5devs_osflash.org/2008-July/003790.html
      : http://osflash.org/pipermail/red5devs_osflash.org/2008-July/003793.html
      : http://flazr.svn.sourceforge.net/viewvc/flazr/trunk/flazr/src/main/java/com/flazr/RtmpDecoder.java?view=markup&pathrev=17

     
  • Peter Thomas

    Peter Thomas - 2009-11-28

    Okay, I'm still going to insist on using 0.7-RC2.  Sorry :)

    Now this is something I only came to know about only yesterday while experimenting with Red5, try this:

    Pass "-version 00000000" as additional arguments to the client command line and let me know how it goes.

     
  • Skello

    Skello - 2009-11-28

    Thank you for responding. Switching to 0.7-RC2 and made some progress.

    Using your switch the handshake seems to be performed, however another error appears. The log is .

      : http://pastebin.com/m6091e8dd

     
  • Peter Thomas

    Peter Thomas - 2009-11-28

    can you get the log in DEBUG mode ?

     
  • Skello

    Skello - 2009-11-28

    Sure.  it is.

      : http://pastebin.com/m17f32127

     
  • Skello

    Skello - 2009-11-28

    Do you want the 0.5 log also in DEBUG so you can compare what goes on differently?

     
  • Peter Thomas

    Peter Thomas - 2009-11-28

    yeah that may help, really can't figure much from the log

    I've uploaded a new official version RC-2 at the downloads area, try with that although it is doubtful that resolves this problem

    I suggest you try the proxy server (ensure it is in DEBUG log mode) it runs on port 8000 and forwards to port 1935 by default, the advantage of this is you can see each RTMP message on the wire.  just tell the client to connect to port 8000

    if you can get logs comparing what 0.5 sends vs what 0.7 sends that would help

     
  • Skello

    Skello - 2009-11-28

    Ok. Tried to do it on my own, but it looks like I need additional guidance. A few questions:

    1. How do I ensure it is in DEBUG mode? Is it different than setting log4j.rootLogger=DEBUG instead on INFO in log4j.properties?
    2. Do I need to modify anything in flazr.properties? Like proxy.remote.host?
    3. What changes should I make in my client.bat command line so that I force it through the proxy? -host to 127.0.0.1 and -port 8080?
    4. Where should the RTMP messages appear? flazr-client.log or flazr-proxy.log?

    Thanks

     
  • Peter Thomas

    Peter Thomas - 2009-11-28

    1. you have it right
    2. only if somewhere other than localhost
    3. -port 8000 should be fine (assuming all on localhost and proxy port is 8000)
    4. flazr-proxy.log

     
  • Skello

    Skello - 2009-11-28

    Something is not working right.

    are all the configs I used.

    I get "19:56:53,578  DEBUG  - port is null in url, will use default 1935" even though I used -port 8000 in command line (check client.bat line at above link).

      : http://pastebin.com/d1999312e

     
  • Peter Thomas

    Peter Thomas - 2009-11-29

    port is null means that you tried to use something starting with "rtmp://" in the command line?

    here is a YouTube video of a load testing session (with Red5 as the server) using the Flazr client, maybe it will give you some pointers:

      : http://www.youtube.com/watch?v=Wa2bDQLKf8s

     
  • Skello

    Skello - 2009-11-29

    Ok. I got it now, at least for 0.7. I did have to modify proxy.remote.host since the streaming sever is remote and I was indeed using a rmtp:// url in my command line.

    After following your example I got it to work, but I noticed from the log that it sends a tcUrl=rtmp://localhost:8000/oflaDemo to the server. I don't know if that matters at all, but just in case I made an additional log where I manually force the proper tcUrl via the -D argument.

    Here are the logs from the proxy: 
     

    Now. How do I pass the 0.5 traffic through the proxy? Do I use the same proxy started from the 0.7 directory? If I do that, it stops at the same point, when it receives metadata, as flazr's 0.7. Also, the proxy logs looks remarkably similar.

    If I use it normally, without the proxy, flazr 0.5 goes on to receive the metadata, write it to file, then start receiving audio and video data and write that to file. That goes on for an average of 50 seconds. The most I got it to work were 80. After that it goes into a ping pong loop. The resulting file does indeed have some 80 seconds of playable audio and video.

      : http://pastebin.com/m332b38e5
      : http://pastebin.com/ma1fac83

     
  • Peter Thomas

    Peter Thomas - 2009-11-30

    Okay, I think we are close.  It has to be a bug at the point 0.7 decodes the onMetadata coming from Red5 - a bug which 0.5 apparently does not have.  This explains why 0.5 + 0.7 proxy fails at the same point because the proxy decode fails.

    We should be able to get to the bottom of this if you can get me the onMetadata contents.  Since the proxy approach is ruled out, try this:

    - record a Wireshark dump of 0.5 talking to this server
    - to run 0.5 in debug mode, edit log4j.properties - it is inside the flazr-0.5.jar though
    - pre-process the wireshark dump, the  will show you how
    - run Flazr 0.5 against the pre-processed Wireshark dump, if all goes well, you will get a detailed log, the way 0.5 used to do it, but I am hoping for the details of the mysterious "onMetadata" to come through this time

    As to why 0.5 stops after some time, I have no idea, but 0.7 should hopefully handle the server acks better.  It could very well be a bug in Red5 for all you know.  So it may take time, but for e.g. using the proxy to see what goes on when a proper Flash player consumes the stream should lead us to the solution, once we get through the metadata problem.

      : https://sourceforge.net/apps/mediawiki/flazr/index.php?title=Flazr_0.5

     
  • Skello

    Skello - 2009-11-30

    Glad to hear we're getting there with this. The resulting log file in DEBUG mode is 29 MB. I think that's too much for pastebin, so I zipped it and uploaded it .

      : http://www.mediafire.com/?wnjucot3yfh

     
  • Skello

    Skello - 2009-11-30

    PS: I also captured the moment when it goes into ping loop (check end of the log). Maybe that can give you an idea what's going on there too.

     
  • Peter Thomas

    Peter Thomas - 2009-11-30

    Just committed a couple of small changes in trunk.  You are able to check out and run from SVN right ? Let me know.

     
  • Skello

    Skello - 2009-11-30

    I've never did it before. I can probably figure it out, but I'll have to get everything needed to compile etc..

    I think it would be quicker if you compile it and host it somewhere temporarily so I can grab it. If not, I guess I'll have to do it.

     
  • Peter Thomas

    Peter Thomas - 2009-11-30
     
  • Skello

    Skello - 2009-11-30

    It's working!!! At least so far.

    It's recording without stopping. Reached almost 200 seconds and the recording ratio seems to be is 1/1, meaning 10 seconds of video playtime are recorded in 10 seconds.

    I can live with that, but I will have a definitive answer for you regarding any interruptions in 40 mins (that's how long the streamed video lasts).

    Either way, great job and remarkable support. You went to great lengths to get this fixed. I really appreciate it.

     
  • Peter Thomas

    Peter Thomas - 2009-11-30

    That is very good news !  For the benefit of anyone who visits this thread in the future, I'm positive this is a **BUG IN RED5** not Flazr.  Anyway, I changed Flazr code to be lenient when this happens.  Flazr 0.5 was admittedly created by using Red5 as a test subject and at a time when I wasn't really clear about the spec - so that explains that.

    Oh and for the 1:1 download speed, experiment with the -buffer setting on the client, set it to large values, say the anticipated duration (note this is milliseconds!) - it works on other servers, no guarantees for Red5 though :\

     
  • Skello

    Skello - 2009-11-30

    Preliminary report:

    Stream recorded successfully at 1:1 ratio. However, resulting .flv has playing issues. With MPlayer it plays, but seeking is not possible. With VLC it doesn't play. Dunno if that's a flazr-related issue, just thought I should mention it.

    Setting the buffer to the anticipated duration of the stream in milliseconds, generates the previous condition. Recording goes on for about 20 seconds, then it goes into a ping loop. Guess server bandwidth should be respected?

    Will test some more and update.

     
1 2 > >> (Page 1 of 2)

Log in to post a comment.