Where should a robots.txt file exist inside a Payload application?

default discord avatar
mrl7
3 months ago
2

We need to store a robots.txt on our CMS site but we are not sure exactly where this file should live. Can anyone assist?

  • default discord avatar
    filippomasoni
    2 months ago

    Following this, I also need to place a robots with disallow: / because I don't want search engines to crawl the backend of the site

  • discord user avatar
    denolfe
    Payload Team
    2 months ago

    This can be done via Express. You can add this to your

    server.ts

    :



    app.get('/robots.txt', (_, res) => {
      res.type('text/plain')
      res.send('User-agent: *\nDisallow: /')
    })


    Modify as needed

Open the post
Continue the discussion in Discord
Like what we're doing?
Star us on GitHub!

Star

Connect with the Payload Community on Discord

Discord

online

Can't find what you're looking for?

Get help straight from the Payload team with an Enterprise License.