It would be great if you can crawl up until a certain directory deep.
Something like -d 2, would find
http;//www.x.com/
http;//www.x.com/index.html
http;//www.x.com/test/
http;//www.x.com/test/test.html
But not
http;//www.x.com/test/other/
http;//www.x.com/test/other/gua.html