Google Now Open Sources Its Robots.txt Parser Code
The Robot Exclusion Protocol (REP) also known as robot.txt is owned by the search engine giant Google. Robots.txt is used by millions of sites to instruct Google search engine crawlers that which part of their blog content should be crawled or not. For 25 years, the REP was not standardized …