Payload Cloud 504 Error
I'm trying Payload Cloud with a next-payload app. Sometimes when I try to access the admin panel, I get a 504 error. Any idea how to avoid this ? It happens pretty often.
Hey @skry. - what is your project name? I'll take a look into this
studio222-next-payload-test-2
@skry. This has been happening fairly often for me too. Is it still an issue for you too?
Furthermore, I regularly get Gatsby build failures on Netlify when sourcing content from Payload CMS caused by either 502, 504 or 524 errors.
I think I deleted the project and I'm not using payload cloud for now. But before deleting it I checked and it was still an issue so you might have the same problem.
Thanks for letting me know. This suggests 504 errors have been a thing for a couple of months. I'll follow up with a new thread 👍
Hey @jesschow , did you find anything out about this?
I'm struggling with intermittent 504 errors as well as the occasional 502 and 524 error.
This makes working with Gatsby difficult - it pulls all the data in from the API during the build process. It seems that I can only push the Payload CMS API so far before it gives up and a Gatabys build fails.
I've tried everything I can think of including implementing axios retries, trying to optimise the number of API calls I make, optimizing retry delays...etc. But I am starting to feel that an API should be able to handle a lot of incoming requests at once?
Another idea I had was to tweak the
rateLimitsettings as described in
https://payloadcms.com/docs/production/preventing-abusebut no luck there either. I've tried
a lotof different settings - currently settled on the following:
rateLimit: { max: 9999999999, trustProxy: true, }Is there any way to help reduce timeouts? Is there some sort of throttling going on with Payload Cloud? My project is called
tt.
Is this the best place to get support for Payload Cloud?
I am finding it doesn't take much to trigger 504 timeouts on Payload Cloud. It is temporary but it affects things like:
- Images don't serve if they are being used anywhere in production.
- Netlify/Gatsby can't build sites with
https://github.com/netlify/gatsby-runner(it tries to get a copy of all queried images as part of the build).
- CMS users occaisonally get the 504 timeout and need to wait for it to resolve itself.
We are on the Payload Cloud $35 a month service, and I love the simplicity! However, I would like to understand whether the $35 a month service can't handle a certain amount of traffic or something? Also, we have been using Payload Cloud for a while now but if the 504 timeouts are an issue, I'll need to move the CMS and I can't figure out a way to easily transfer content uploaded to S3 using AWS Cognito.
Hey @capitalt2160 Yes, I can help you out. Our backing provider is DigitalOcean for all compute, so 504 timeouts should not be an issue as we have some very high-throughput services running on the same specs.
Let me ask a few questions to help diagnose:
- Can you provide your Project ID from
Settings -> Billling. I can look into your builds and how gatsby-runner may be having issues.
Images don't serve if they are being used anywhere in production.
Images for Payload Cloud are served via AWS S3 through Cloudflare. "Being used anywhere" is not an issue typically seen with this setup. Is this for
all images? Can you provide a URL, so I can diagnose/troubleshoot?
I can't figure out a way to easily transfer content uploaded to S3 using AWS Cognito.
We're working on the documentation for this. In the meantime, there is a write-up on how to access your S3 assets from local development here:
https://github.com/payloadcms/plugin-cloud/issues/4#issuecomment-1601681789. However, I'm not sure this will meet your needs. I can go more in-depth or provide a script if needed, depending on how you're wanting to do your backups. For future: one-click export of S3 data is in the works.
Hey @denolfe !
My project ID is
64639cddf14c76a2a7d276af.
The intermittent 504 timeouts are an issue for all images - in fact, anything from Payload Cloud whether it's logging into the CMS, using the API or serving images. In summary, anything under
https://cms.teamtailor.app/or
https://tt.payloadcms.app/becomes unavailable for a few minutes every now and then.
We are only using images directly from the CMS on a few pages. e.g.
https://www.teamtailor.com/en/resources/partnerships. Occasionally, all the images on this page will not load. An example image:
https://cms.teamtailor.app/marketing-site-images/muhammad-faiz-zulkeflee-alw-CwGFmwQ-unsplash-1-1220x1525.jpg.
My best guess so far is that I am hitting either a memory limit or API throttling by IP (which is why I have been playing around with the
rateLimitsettings). I guess using Gatsby on Netlify means that the API/images get hit from the same IP address all in one go!
Loving Payload CMS by the way - as are my colleagues. Genuinely the first CMS we love working with, and we have worked with a lot of CMSs'. So much so I have two Payload plugins and a Gatsby plugin that I have been working on for a while that I hope to bring to version 1 soon!
Thank you for the extensive detail 👍 I refreshed that image over the course of a few minutes, and
I wasable to get a single 504. I'm looking into this on our end.
@capitalt2160 I dug into this. It looks like your app is maxing out the RAM of the instance. The baseline for a Payload app is a RAM usage of around 30%. Your app has a baseline of
70% usage on boot, and it looks like it is spiking up to 100% at times - likely causing the 504s you are seeing.
Any idea why your app's baseline RAM usage would be that high? Are you running any RAM-heavy operations?
@denolfe interesting! Thanks for the lead! I'll profile the app locally - it could be the plugins 😅
Possible! Are you running quite a few?
Thanks @denolfe ! Looks like the issue is resolved. I really appreciate the help!
I am not running many plugins, but it seems the plugin I created to sync localized fields with CrowdIn was causing the RAM overload.
I removed it and I'm refactoring it to work differently:
https://github.com/thompsonsj/payload-crowdin-syncI think the issue was passing the CrowdIn API client as middleware into express - which is unnecessary.
Great to hear, glad we were able to get to the bottom of it 👍
Continue the discussion in Discord
Star
Discord
online
Get help straight from the Payload team with an Enterprise License.