I can give some advice on this as Iv done something similar, I cant post any code tho as my code can be used for hacking and thats not a subject liked much on these forums, also your code to index a site can also be easily abused so be careful. And your on the right track, just tweak your code and work out the bugs.
Google's Spider does the exact same thing, Finds the first Index or Linked page and then parses that for further links which it then follows.
However, it also reads a server file called robots.txt which lists stuff the server admin does not want google to index.
Google's Spider does the exact same thing, Finds the first Index or Linked page and then parses that for further links which it then follows.
However, it also reads a server file called robots.txt which lists stuff the server admin does not want google to index.