X-Git-Url: https://git.r.bdr.sh/rbdr/r.bdr.sh/blobdiff_plain/6bc416664bd5d6c455c3dd38229c3ec9e3a7af0d..b87755aaf01f8ed14235bee6b6e851e2846ff72f:/robots.txt?ds=inline diff --git a/robots.txt b/robots.txt index 6bc6ff8..a52d107 100644 --- a/robots.txt +++ b/robots.txt @@ -21,8 +21,14 @@ User-agent: peer39_crawler User-agent: peer39_crawler/1.0 User-agent: PerplexityBot User-agent: YouBot +User-agent: Meta-ExternalAgent +User-agent: Timpibot +User-agent: OAI-SearchBot +User-agent: DuckAssistBot Disallow: / User-agent: * Allow: / Disallow: /git + +