Where should a robots.txt file exist inside a Payload application?

mrl7last year

2

• 

  filippomasoni12 months ago

  Following this, I also need to place a robots with disallow: / because I don't want search engines to crawl the backend of the site

• 

  denolfe

  12 months ago

  This can be done via Express. You can add this to your

  server.ts

  :

  
  
  

  app.get('/robots.txt', (_, res) => {
    res.type('text/plain')
    res.send('User-agent: *\nDisallow: /')
  })

  
  

  
  

  Modify as needed