I've set up the cloud-storage plugin in my Payload Cloud install with GCS. This works fine locally, but while the crops are working as expected on Cloud, the original files are uploaded "empty".
In the screenshot attached, the first image
arkan-charrogg
was added in my local instance. Both the original jpg and the crop are uploaded fine.
The second one though,
arkan-shrike
, was created on the Payload Cloud instance, and you can see that the original has a size of 0.
The same issue occurs with non-image files like videos, they show up in my GC bucket with a size of 0 (on Payload Cloud, works fine locally).
Given I cannot reproduce the issue anywhere else than in Payload Cloud, I'm a bit stomped on how to investigate this further, so any help would be greatly appreciated! 🙏
So I added some logs to see what was being passed in the beforeChange hook of my media collection (which is how the cloud-storage "intercept" the files:
https://github.com/payloadcms/plugin-cloud-storage/blob/df7499483ac303e585da22f7328a8658808ea023/src/utilities/getIncomingFiles.ts#L5), and found that
req.files
looks like this on Cloud:
file: {
name: 'arkan-torgadoro.jpg',
data: <Buffer >,
size: 374543,
encoding: '7bit',
tempFilePath: '/workspace/tmp/tmp-1-1683612139296',
truncated: false,
mimetype: 'image/jpeg',
md5: 'e3e13c5fa3565aa14ee756a415592d5d',
mv: [Function: mv]
}
Locally, where it works,
req.files.file.data
looks like this:
data: <Buffer ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 ff db 00 43 00 08 06 06 07 06 05 08 07 07 07 09 09 08 0a 0c 14 0d 0c 0b 0b 0c 19 12 13 0f ... 374493 more bytes>,
That explains the empty file in the GCS bucket, now onto figuring out why the req is like this. 🔍
I looked into the
req
object in the
beforeValidate
hook, the buffer's already empty there.
Hmmm so I took a look at how the
plugin-cloud
does things, and there's this part that isn't present in
plugin-cloud-storage
:
const fileBufferOrStream: Buffer | stream.Readable = file.tempFilePath
? fs.createReadStream(file.tempFilePath)
: file.buffer
Could that be it? 🤔
@tinouti Looking into this
Just to clarify, you're wanting to use
your GCSnot Payload Cloud's storage, correct?
If that's the case, you can remove the payloadCloud plugin, as the only thing the plugin does right now is configures Payload Cloud's storage.
We're planning on putting more functionality into
plugin-cloud
in the future (email service). It will allow you do pass
storage: false
to the plugin in order to use your own storage.
Oooh good point. Let me try to remove it for now then and see if that solves it!
But to answer your question, yup, we'd like to use our own already in place CDN with GCS for our assets instead of having to rely on the one built in the Payload Cloud plan. 😊
Okay so removing
payloadCloud()
does fixes the issue. The
Buffer
now has content and the file is properly uploaded to my GCS bucket. 🙌
Ideally we'd love to also keep the Payload Cloud "local" storage to S3 as a backup, but at least we're unblocked for now. Thank you! 🤗
Let me know if you'd like me to test something out for you with my setup. 👍
Awesome, will do. Sounds like we need to shore up our documentation on what the plugin does and how it should be used
Star
Discord
online
Get dedicated engineering support directly from the Payload team..