<br/>I want a have spider modified or built which ever is easier. You can use existing opensource libraries or anything, it doesn't matter as long as it acheives the tasks.<br/><br/>I want to be able to run the spider as an applet and from the command line so that i can be execute as a cron job.<br/><br/><br/>The spider must be able to accept command line arguments eg.<br/><br/>main(String args[]) { String var = args[0]}<br/><br/><br/>and the applet should have a simple gui.<br/><br/><br/>The spider should be able to take in the domain name and crawl that domain only unless the option is choosen for the spider to leave the domain. It must have the option to re-index if html page has changed.<br/><br/><br/>It should check header status of a page and does not index unless the page is available, so status 200 etc.<br/><br/><br/>--------------- Specs ---------------------------<br/><br/><br/>Spider gets full html page contents<br/><br/>if the html tag i want to check for (eg &lt;object&gt;&lt;/object&gt;) is found then<br/><br/>Parse all html tags<br/><br/> get : array of tags i specify<br/><br/>example String getTags[]={"title","keyword"}<br/><br/><br/>if keyword is empty/missing and description or title not empty then<br/><br/> split description at every word<br/><br/> return array of keywords limit to 250<br/><br/>else if title empty<br/><br/> attempt to extract keywords from html body up 250 words<br/><br/>if the html tag i checked for is not found then do not parse the page just get all the links from the page and continue crawling.<br/><br/><br/>the crawler need to be able to return the values of html tags and their attributes that i specify.<br/><br/><br/>I'd like the values returned to be in an associative array/map so that<br/><br/><br/>myObject['title'] will contain the title<br/><br/>myObject['keyword'] will contain an array of keywords<br/><br/>myObject['tagName']['Attribute'] will get the attribute value of the html tag example<br/><br/>myObject['embed']['src']<br/><br/><br/>Lastly i want the data to be inserted/indexed in my mySQL database but only if the html tag i checked for was found.<br/><br/><br/><br/>Please make sure you read and udnerstand the reqiurments. This will be integrated into one of my projects and it needs to be built according to my specs.<br/><br/><br/>The spider can be a modification to the one found here<br/><br/>http://www.developer.com/java/other/article.php/1573761/Programming-a-Spider-in-Java.htm<br/><br/><br/>or here<br/><br/>http://www.javaworld.com/javaworld/jw-11-2004/jw-1101-spider.html<br/><br/><br/>or anything from the net<br/><br/>http://www.google.com/search?hl=en&source=hp&q=java +web+spider&aq=0&oq=java+web+spid&aqi=g2<br/><br/><br/>or if you already have a class or library that does this.<br/><br/><br/>It doesn't matter i just want a spider customized to do the above.<br/><br/><br/>Escrow payment only... No automated bids please.<br/><br/>