I don't think it is possible to download a file listing, you can only search the html pages for links. I do remember creating a search engine crawler once, and if I remember correctly it started from a random page and just followed links from there.
Programming robots/spiders using TCP
Blitz3D Forums/Blitz3D Programming/Programming robots/spiders using TCPThe forum is locked. These are the original blitzbasic.com forums, kept so they can still be read. No new topics, replies or edits can be made here. To join in, head over to the Blitz3D+ forums.