Where should a robots.txt file exist inside a Payload application?

default discord avatar
mrl711 months ago
2

We need to store a robots.txt on our CMS site but we are not sure exactly where this file should live. Can anyone assist?

  • default discord avatar
    filippomasoni11 months ago

    Following this, I also need to place a robots with disallow: / because I don't want search engines to crawl the backend of the site

  • discord user avatar
    denolfe
    11 months ago

    This can be done via Express. You can add this to your

    server.ts

    :



    app.get('/robots.txt', (_, res) => {
      res.type('text/plain')
      res.send('User-agent: *\nDisallow: /')
    })


    Modify as needed

Star on GitHub

Star

Chat on Discord

Discord

online

Can't find what you're looking for?

Get help straight from the Payload team with an Enterprise License.