No Limit Bot

Our crawlers are designed to fully satisfy the needs of our directory and only the needs of it. The only information that we collect is about the sites we are crawling and is limited to the most important information we need about the site:

• Title
• Keywords used in the site, in order to generate our categories
• Description
• URL

All the information we save is retrieved from the home page of the website, but sometimes we crawl some of the inner pages of a site in order to collect more links. When our crawlers download some inner page of a site the only reason for this is to satisfy the needs of more links in our database.

If for any reason you don't want our crawlers to download and/or index your site you can specify this, according to the robotstxt.org specification. Our crawlers are fully supporting the specification of the robots.txt and meta-robots tag and are identified as nolimitbot. If you'd like to not to crawl your pages using the robots.txt you could simply write the following in your robots.txt: or Note that, we also support sub-directory exclusion as specified by the standard. If you prefer the use of meta-robots write the following tag in your html code: or You can specify noindex or nofollow independently of one another. and the meaning is as follow:

• noindex - we will not include your site in our index. this is only meaningful for your main page of the site, as we do not include anyway the others
• nofollow - no links found on the page will be followed

Also note that, the examples above are just for reference and does not include all of the possibilities available. Also it is important to say that we do not distinguish between uppercase and lower case, so it doesn.t matter in what case the exclusion description is written.