User-agent: Mediapartners-Google Disallow: # /robots.txt file http://www.pighealth.com/robots.txt # you may not have blank lines in an individual record below - # cos they are used to delimit multiple records #A Disallow line must have a User-agent line before it. #As records are delimited by newlines,# there cannot be newlines between the User-agent # and Disallow lines. #Field names are case-sensitive - User-agent and Disallow must have capital initials. # User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /log/ ##########EXAMPLES 2######### # The '*' in the User-agent field is a special value meaning "any robot", you cannot have # lines like "Disallow: /tmp/*" or "Disallow: *.gif" # To exclude all files except one is awkward, as there is no "Allow" field. The easy # way is to put all files to be # disallowed into a separate directory, say "docs", and leave the one file in the level #above this directory # Disallow: /~joe/docs/ ##########EXAMPLES 2######### #User-agent: webcrawler #Disallow: # #User-agent: * #Disallow: /logs # #The first line, starting with '#', specifys a comment #The first paragraph specifies that the robot called 'webcrawler' # has nothing disallowed: it may go anywhere. ## #Two common errors: #Regular expressions are not supported: instead of #'Disallow: /tmp/*' just say 'Disallow: /tmp'. #You shouldn't put more than one path on a Disallow line #(this may change in a future version of the spec)