Where should a robots.txt file exist inside a Payload application?

mrl7

3 months ago

2

• 

  filippomasoni

  2 months ago

  Following this, I also need to place a robots with disallow: / because I don't want search engines to crawl the backend of the site

• 

  denolfe

  Payload Team

  2 months ago

  This can be done via Express. You can add this to your

  server.ts

  :

  
  
  

  app.get('/robots.txt', (_, res) => {
    res.type('text/plain')
    res.send('User-agent: *\nDisallow: /')
  })

  
  

  
  

  Modify as needed