Payload Cloud and cloud-storage plugin, original file not uploading

default discord avatar
tinoutilast year
12

I've set up the cloud-storage plugin in my Payload Cloud install with GCS. This works fine locally, but while the crops are working as expected on Cloud, the original files are uploaded "empty".



In the screenshot attached, the first image

arkan-charrogg

was added in my local instance. Both the original jpg and the crop are uploaded fine.


The second one though,

arkan-shrike

, was created on the Payload Cloud instance, and you can see that the original has a size of 0.



The same issue occurs with non-image files like videos, they show up in my GC bucket with a size of 0 (on Payload Cloud, works fine locally).



Given I cannot reproduce the issue anywhere else than in Payload Cloud, I'm a bit stomped on how to investigate this further, so any help would be greatly appreciated! 🙏



So I added some logs to see what was being passed in the beforeChange hook of my media collection (which is how the cloud-storage "intercept" the files:

https://github.com/payloadcms/plugin-cloud-storage/blob/df7499483ac303e585da22f7328a8658808ea023/src/utilities/getIncomingFiles.ts#L5

), and found that

req.files

looks like this on Cloud:



file: {
  name: 'arkan-torgadoro.jpg',
  data: <Buffer >,
  size: 374543,
  encoding: '7bit',
  tempFilePath: '/workspace/tmp/tmp-1-1683612139296',
  truncated: false,
  mimetype: 'image/jpeg',
  md5: 'e3e13c5fa3565aa14ee756a415592d5d',
  mv: [Function: mv]
}


Locally, where it works,

req.files.file.data

looks like this:


data: <Buffer ff d8 ff e0 00 10 4a 46 49 46 00 01 01 00 00 01 00 01 00 00 ff db 00 43 00 08 06 06 07 06 05 08 07 07 07 09 09 08 0a 0c 14 0d 0c 0b 0b 0c 19 12 13 0f ... 374493 more bytes>,


That explains the empty file in the GCS bucket, now onto figuring out why the req is like this. 🔍



I looked into the

req

object in the

beforeValidate

hook, the buffer's already empty there.



Hmmm so I took a look at how the

plugin-cloud

does things, and there's this part that isn't present in

plugin-cloud-storage

:



const fileBufferOrStream: Buffer | stream.Readable = file.tempFilePath
          ? fs.createReadStream(file.tempFilePath)
          : file.buffer


Could that be it? 🤔

  • discord user avatar
    denolfe
    last year

    @tinouti Looking into this



    Just to clarify, you're wanting to use

    your GCS

    not Payload Cloud's storage, correct?



    If that's the case, you can remove the payloadCloud plugin, as the only thing the plugin does right now is configures Payload Cloud's storage.



    We're planning on putting more functionality into

    plugin-cloud

    in the future (email service). It will allow you do pass

    storage: false

    to the plugin in order to use your own storage.

  • default discord avatar
    tinoutilast year

    Oooh good point. Let me try to remove it for now then and see if that solves it!



    But to answer your question, yup, we'd like to use our own already in place CDN with GCS for our assets instead of having to rely on the one built in the Payload Cloud plan. 😊



    Okay so removing

    payloadCloud()

    does fixes the issue. The

    Buffer

    now has content and the file is properly uploaded to my GCS bucket. 🙌



    Ideally we'd love to also keep the Payload Cloud "local" storage to S3 as a backup, but at least we're unblocked for now. Thank you! 🤗



    Let me know if you'd like me to test something out for you with my setup. 👍

  • discord user avatar
    denolfe
    last year

    Awesome, will do. Sounds like we need to shore up our documentation on what the plugin does and how it should be used

Star on GitHub

Star

Chat on Discord

Discord

online

Can't find what you're looking for?

Get help straight from the Payload team with an Enterprise License.