Viewing source data: RFC9309 indexed from relaton-data-rfcs RFC 9309
docid.type IETF
docid.primary True 10.17487/RFC9309
docid#2.type DOI
docnumber RFC9309
language en
type standard
script Latn
date.value September 2022
date.type published
title.content Robots Exclusion Protocol
title.format text/plain
title.type main
abstract.content <p>This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching.</p>
abstract.format text/html
abstract.script Latn
abstract.language en
revdate September 2022
contributor.role author M. Koster Latn en M. Koster Latn en
contributor#2.role author G. Illyes Latn en G. Illyes Latn en
contributor#3.role author H. Zeller Latn en H. Zeller Latn en
contributor#4.role author L. Sassman Latn en L. Sassman Latn en
series.title.content RFC
series.title.format text/plain
series.number 9309
series.type main
keyword robot
keyword#2 crawler
keyword#3 robots.txt
id RFC9309