Nice!

[Nobody/Anonymous]

Hey - well done on the project it works nicely for the most part. However, I
believe you have added a ton of unnecessary complexity. For example, why do
you start 4 threads? You are not actually multithreading, because all the
threads are doing the same job. Also, there is really no need to use the
org.apache package - you can use simply java.net.URL instead. Also I believe
you need to make an update - youtube no longer uses "var swfHTML", try looking
for "videoplayback" instead.

Again, great project - I had wanted to learn you youtube videos could be
downloaded using java and your project showed me the way. Thanks.

[StefanKaerst]

Hi,

For example, why do you start 4 threads?
because I wanted to learn how to do such things with threads. one download
runs "a while" if you select HD .. so why should I wait if java can simply
start more threads doing the same thing?

You are not actually multithreading
why not? You can drop in different URLs and each thread downloads one file.
(If I change it to the new yt source code :)

I decided to use the org.apache package because I wanted to get easy access
to: cookies and the http proxy!
the guys at apache did a great job!

I started developing ytd2 because ytd(1) was/is not open source and did not
start use threads nor can it use a proxy. I wanted to show that this is quite
easy to implement. (with fewer lines of code)

you need to make an update
right! I'll change it ASAP.

thanks for your post!
let me know if I should change something (or if You need help :)

Regards
Stefan

[Nobody/Anonymous]

Hi it's me again

I just checked back on this project and was glad to see you made an update to
account for the youtube source change! Unfortunately, the new version seems to
have a bug which prevents it from working. The HttpGet constructor is throwing
an IllegalArugmentException when you supply it with the video url (extracted
from the youtube html source).

The culprit seems to lie in your savetextdata() method, particularily the line
where you use 'sline = sline.toLowerCase().replaceFirst(".*fmt_url_map\": \"",
"\"");' For some strange reason that I do not understand, after this call
sline contains unicode characters (ie "\u0026" and such). To be honest, I'm
not sure what you're trying to accomplish with the "fmt_url_map" replace. Have
you looked into java.net.URLDecoder to decode the html line? I have found that
to work very nicely.

After seeing your project, I actually went on to make my own youtube
downloader in java. It uses a small 50 line class to extract the video url
from the youtube source (credit goes to you for showing me how =D) and then
uses a download manager (http://www.java-tips.org/java-se-tips/javax.swing
/how-to-create-a-download-manager-in-java.html) to
provided a nice UI for the download, equipped with pause/resume functionality.

Again, thanks a bunch for your code - your project is definitely more useful
than the original ytd. Looking forward to your response

[StefanKaerst]

Hi "somebody"

Unfortunately, the new version seems to have a bug which prevents it from
working

Yes, looks like you found a change of the webpage source code again! (IMHO
this is not really a bug)

To reproduce something I would need the URL you used (u can email me as well)
and the error message. This case I could not get any error output - the thread
just stops working. (which is not good either)

There is a string-replace needed before you can start a download because the
sourcecode is JavaScript - you cannot just copy parts of it and use it without
some kind of interpretation. Or you need a JavaScript engine ..

sline.toLowerCase().replaceFirst(".*fmt_url_map\": \"", "\"");' For some
strange reason that I do not understand

its the same as:

sline = sline.toLowerCase(); sline = sline.replaceFirst(...

but shorter!
You need some understanding of regular expressions.
String.replaceFirst(".*fmt_url_map\": \"", "\"") replaces all including
fmt_url_map": " with "
Which means it just deletes that substring. Only the string that remains is of
importance.
You should/could use Eclipse or some other way to run ytd2 wihtin a debugger
line by line or you change it and insert some calls like System.out.println..
or debugoutput .. ;-)

hth
Thanks for your bug report. I'm looking forward to here from you!

Regards
Stefan

[Nobody/Anonymous]

Hi again

Wow yes I had completely forgotten about regex #-o

I was looking at your code once again and did not really understand why you
were using iteration for your extraction/download process. It seems to be a
naturally linear process to me - first extract the correct video url from the
source code (based on the user's desired video quality) and then download the
binary video file to the user's hard drive. Perhaps I am missing something?

Also, to get back to my earlier question, I still am not sure what benefit the
apache package is providing for the user. If I'm not mistaken (don't feel like
looking it up right now), proxy support is included in the java.net package as
well. And I can't figure out what exactly is the use of cookies in this
particular application.

I made a small test suite for the url extraction. Hopefully I have the right
idea:

public static void main(String[] args) throws Exception {
    // read url from command line
    URL youtubeUrl = new URL(args[0]);
    InputStream htmlStream = youtubeUrl.openStream();
    BufferedReader htmlReader = new BufferedReader(new InputStreamReader(htmlStream));
        while (true) {
        String htmlLine = htmlReader.readLine();
        // look for the line containing the url map
        if (htmlLine.contains("\"fmt_url_map\": \"")) {
            // isolate the map from the rest of the source line using regex
            String urlMapString = htmlLine.replaceFirst(".*\"fmt_url_map\": \"", "").replaceFirst("\".*", "");
            // reformat the line
            urlMapString = urlMapString.replace("\\u0026", "&").replace("\\", "");
                // separate the list key-value pairs into an array
            String[] urlStrings = urlMapString.split(",");
            for (String urlString : urlStrings) {
                // separate the key value pair
                String[] fmtUrlPair = urlString.split("\\|");
                // simply print the info for now
                System.out.println("Fmt: " + fmtUrlPair[0] + ", Url: " + fmtUrlPair[1]);
            }
                break;
        }
    }
    htmlReader.close();
}

Thanks

[Nobody/Anonymous]

^ Eh... the line wrapping and screwy indentation makes the code look all
messed up X-(

[StefanKaerst]

Hi,
You wanna know everything, right? :)

"why you were using iteration for your extraction/download process. It seems
to be a naturally linear process to me"

keep in mind that this project has a "long" history. As I started development
I tried with wget and therefore assumed that cookies have to retransmitted
(which is not the case). I checked what a browser does and saw several get
requests. I decided to use recursion to do more requests (for one yt webpage)
with one thread and the hc.apache.org package because of the easy handling of
cookies, proxy and the http response codes. I did not want to reinvent a http
client.

If you are clever (which you very likely are :) you write your downloader
without any other package and send me the code if you want to. I'd really like
to see your work!
(There are always many ways to accomplish the same task - especially if one
starts from scratch)
One motivation of this project was not to download videos but to learn Java ->
multithreading, gui handling (mouse events) and some other things. I started
other projects in Java (for IBM Lotus Notes) where I used some things I
learned while developing ytd2. (and vice versa) I have at least one Java
project in use that I plan to release as OSS too. Until then the chances are
<1 that I change ytd2 to not to use the hc.apache.org package. ;-)
It works great and it showed me how to handle things with Eclipse (I even had
the sources of the hc package attached to change it ... in particular the part
that parses the cookies :)

There is still work to do. IMHO a software should not stop working without an
(helpfull) error message. I guess google changes the source code from time to
time so ytd2 should handle this.

go ahead!
Regards Stefan

[Nobody/Anonymous]

Haha yes I do want to know everything, thanks for bearing with my questions
;-)

Your last response clarified some of my questions related to this project. I
had not realized that this project was meant to be more of a
experimentation/learning tool than a commercialized product.

You see, I feel that there is a great market for an appropriate youtube
downloader. An enormous number of people around the world enjoy listening to
music. Ever since the invent of the iPod, the demand for music that can be
listened to on the go has risen enormously. However, the acquisition of music
is far from ideal in today's day and age.

The largest percentage of the population gets their music from iTunes -
however, this method costs money and makes it expensive to obtain a large
library of music. For many people who are more technical there are p2p sharing
from software such as Limewire and various torrents - this method, however, is
also unideal because it is unsafe (viruses) and illegal (anybody can get
caught, and those who do get ridiculous fines). Also, p2p sharing is too
technical and inconvenient for the average music listener.

I myself enjoy listening to music on my iPod. When I got my first iPod several
years ago, I used iTunes to get my songs. I did not enjoy paying the money,
and I decided to switch to Limewire. After my computer got infected a couple
times and I learned of people getting fined hundreds of thousands of dollars
after getting caught, I called it quits with Limewire. This is when I came
across youtube downloaders.

Googling "youtube to mp3" can direct you to hundreds of youtube music
downloaders, whether web-based or desktop applications, that extract the audio
from youtube videos and save them to your hard drive. This method is very
convenient and easy to use - all you need is the youtube url - and is entirely
risk-free. Also, youtube now offers high quality audio (up to 128 kbps),
making a youtube downloader an ideal method for obtaining free and high
quality music conveniently and at no risk.

I was looking for a useful project to work on in Java and a youtube audio
downloader seemed like the perfect idea. One current problem with the
commercial youtube downloaders out there today is that they convert the
downloaded video to mp3 format, which results in a loss of quality in the
music. Another problem is that most commercial yt downloaders do not allow for
multiple simultaneous downloads. By overcoming both these shortcomings I
believe one can make a very useful and successful product. Just to clarify, I
am not taking about making money. I am a student and have recently started
learning programming, and I am in search for my first real project.

Phew, that was a lot of writing =D. I hope you now understand where I am
coming from and what my goals are. I suppose this post as much for myself
(just getting my ideas down in writing) as it was for you... Sorry if I bored
you ;). Thanks once more for your responses and for your time, I hope to
continue hearing from you.

[Nobody/Anonymous]

Hi again,

a commercialized product.You see, I feel that there is a great market for an
appropriate youtube downloader.

AFAIK, this is prohibited by the yt terms(!)
we only develop some java software - we don't download v****s! ;-) (only the
yt download clients within TV and HD recorders do that)

however, this method costs money and makes it expensive to obtain a large
library of music.

this touches another (not so easy) topic. people who create music/movies may
work hard to produce something and they try to earn money with that. we are
not in the position to judge their payment (not directly at least)

.. and is entirely risk-free ..

if something is easy to do does not necessarily mean it's conform to the law.
I mean, youtube and the internet is available around the world but there are
national laws the users have to obey (no matter if they know of or not)
let me add something everybody should already know: nothing(!) someone does
using the internet (or any other packet switched network) is really secret.
every active component that transmits packets can potentially write a log.
I dont want to panic you or others but thats just how it works.

Sorry if I bored you

no, you didn't! I had similar ideas. You can still email me if you dont want
to publish all of your ideas here .. ;-)

maybe we get back to the topic!?
I'll check your code snippet above ..

Your last response clarified some of my questions related to this project.

okay, what questions are still open?

I could add something to your question regarding recursion of the threads:
it takes less lines of code if the threads reuse some of its code rather than
write linear calls like:
- create http get request
- read text body
- close stream
- create http get request
- read binary data
- close stream

opening and closing are the same code if you use one object that calls itself.
Unfortunately I did not figure out how to use one streamreader for both text
and binary :(

Regards
Sefan

(excuse my erroneous english)

[Nobody/Anonymous]

Hi

if something is easy to do does not necessarily mean it's conform to the
law. I mean, youtube and the internet is available around the world but there
are national laws the users have to obey (no matter if they know of or not)

I understand what you are saying, but AFAIK downloading youtube videos is not
explicitly prohibited by law. Yes, it is against youtube terms, but you cannot
get federal penalties (i.e. fines and jail time) for downloading youtube
videos. This is evidenced by the enormous number of commercial youtube
downloaders out there, endorsed by well respected downloading sites (cnet, for
example).

people who create music/movies may work hard to produce something and they
try to earn money with that. we are not in the position to judge their payment

This I agree with - stealing is most definitely a bad thing ;-)
However, you have to keep in mind that users do not pay money to listen to
music on youtube. The original uploader of the video is the one responsible
for attaining the rights to the video; once the video is up on youtube it is
open to the public. A youtube downloader is simply for convenience - it allows
users to add youtube music to their iPods instead of maintaining a youtube
playlist.

you write your downloader without any other package and send me the code if
you want to

^ see the patches for the code of the simple test class I made. It is not at
all a complete project - it's meant as a demo

Thanks once again

[Nobody/Anonymous]

sir actually I want java coding of any video converter software, so how can i
get that?