Northcote Internet are a company which provides data for investors on a wide range of publicly quoted companies and has been used by the stockbroker, The Share Centre to provide these services to its members.
The Northcote Link Spider is based upon an internet robot created by RWAP Services (c) 2003-04, written in Perl 5.
The Link Spider was developed to check the database of links to the company information which is maintained by Northcote Internet. It is easy for this database of links to rapidly become out of date and therefore the Link Spider's job is to check each link within the database and report whether it is a valid link, causes an error or has a redirection set up on the web page.
As well as spotting the standard web page errors and re-directions, the Link Spider is able to list questionable pages, based upon a list of user-defined phrases which may appear in the title or body of a given webpage.
The Northcote Link Spider is designed to follow the standard robots exclusion protocol rules, and is identified by the following user-agent: NorthcoteSpider. If you want to exclude the Link Spider from any part of your website, you will need to follow the robots exclusion protocol and using this identifier to either include a statement in the robots.txt file on your website, or within the meta tag: