regenerateMediaSizes + plugin-cloud-storage

default discord avatar
arielarial
6 months ago
56

Hello everyone! I was hoping any of you had success using James' snippet (

https://github.com/payloadcms/payload/discussions/1834#discussion-4740607

) to regenerate media sizes while using plugin-cloud-storage.


Local API asks for the media filePath to update the media file, but that doesn't exist for cloud media.

  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    You can also pass in the

    file

    directly!



    So you would just need to fetch your file from your own cloud storage, and then pass it in as

    file
  • default discord avatar
    arielarial
    6 months ago

    Thanks a lot @jmikrut! I'll try that

  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    if you get this going, you should share it as a discussion 👍

  • default discord avatar
    arielarial
    6 months ago

    Sure! will do 🙂



    @jmikrut I tried this:



              await fetch(mediaDoc.url)
                .then((response) => response.blob())
                .then(async (blob) => {
                  await payload.update({
                    collection: "media",
                    id: mediaDoc.id,
                    data: mediaDoc,
                    file: blob,
                    overwriteExistingFiles: true,
                  });
    
                  console.log(
                    `Media ${mediaDoc.alt || mediaDoc.id} regenerated successfully`
                  );
                });


    but I'm getting this error:


    `
    TypeError: Cannot read properties of undefined (reading 'split')
        at generateFileData (/[...]/node_modules/payload/dist/uploads/generateFileData.js:87:29)
        at update (/[...]/node_modules/payload/dist/collections/operations/update.js:95:103)
        at processTicksAndRejections (node:internal/process/task_queues:96:5)
        at async /[...]/cms/regenerateMediaSizes.js:28:15
        at async /[...]/cms/regenerateMediaSizes.js:25:11
        at async Promise.all (index 0)
        at async regenerateMediaSizes (/[...]/cms/regenerateMediaSizes.js:22:5)
    Media 640ce5695867e6824e272192 failed to regenerate
    FileUploadError: There was a problem while uploading the file.


    at new ExtendableError (/[...]/node_modules/payload/dist/errors/APIError.js:22:15)
        at new APIError (/[...]/node_modules/payload/dist/errors/APIError.js:38:9)
        at new FileUploadError (/[...]/node_modules/payload/dist/errors/FileUploadError.js:10:9)
        at generateFileData (/[...]/node_modules/payload/dist/uploads/generateFileData.js:121:15)
        at update (/[...]/node_modules/payload/dist/collections/operations/update.js:95:103)
        at processTicksAndRejections (node:internal/process/task_queues:96:5)
        at async /[...]/cms/regenerateMediaSizes.js:28:15
        at async /[...]/cms/regenerateMediaSizes.js:25:11
        at async Promise.all (index 0)
        at async regenerateMediaSizes (/[...]/cms/regenerateMediaSizes.js:22:5) {
      status: 400,
      data: null,
      isPublic: false,
      isOperational: true
    }


    I tried to log blob to check if its loading ok, and it seems fine:


    Blob {
      [Symbol(type)]: 'image/jpeg',
      [Symbol(buffer)]: <Buffer ff d8 ff e1 00 bc 45 78 69 66 00 00 49 49 2a 00 08 00 00 00 06 00 12 01 03 00 01 00 00 00 01 00 00 00 1a 01 05 00 01 00 00 00 56 00 00 00 1b 01 05 00 ... 283730 more bytes>
    }


    @jmikrut can you shed some light on this? What format is expected for

    file

    ?



    I found that the error is because payload is trying to access

    file.name

    . So I tried to send it like this, and did not get any errors, but also no image being updated:



    file: {
      ...blob,
      name: mediaDoc.filename,
    },
  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    you need to pass it corresponding to the type, which is here:



    https://github.com/payloadcms/payload/blob/master/src/uploads/types.ts#L82


    your

    Buffer

    goes on

    file.data
  • default discord avatar
    arielarial
    6 months ago

    Nice, thanks a lot! I'm sending file like specified, getting no errors, but the new media size is still not being generated :/


    I'm not used to work with blobs/buffer, s I'm kinda lost here, sorry



    await fetch(mediaDoc.url)
      .then((response) => response.blob())
      .then(async (blob) => {
        const arrayBuffer = await blob.arrayBuffer();
        const buffer = await Buffer.from(arrayBuffer, "binary");
    
        const file = {
          data: buffer,
          mimetype: blob.type,
          name: mediaDoc.filename,
          size: mediaDoc.filesize,
        };
    
        await payload.update({
          collection: "media",
          id: mediaDoc.id,
          data: mediaDoc,
          file,
          overwriteExistingFiles: true,
          contentType: blob.type,
        });
    
        console.log(
          `Media ${
            mediaDoc.filename || mediaDoc.id
          } regenerated successfully`
        );
      })
  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    is your new file

    bigger

    than your newly added media sizes?



    like, dimensions-wise



    i think you're almost there. honestly this type of thing

    is very confusing

    no matter what



    blob vs buffer, arrayBuffer, etc etc

  • default discord avatar
    arielarial
    6 months ago

    not really, I made a test image size which is 100x100



    the uploaded media is 2560x1440

  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    question - are the

    old

    sizes being re-generated?



    can you tell by looking at modified / created timestamps on the files on disk? I am wondering because maybe your script is using your already-built Payload config, which might not have the new sizes reflected



    and second question - (more like a suggestion)



    if i were you, at this point, i'd try and get set up with a debugger in your IDE so that you can step through what's actually happening inside of both your function, and in Payload's

    update

    operation, to see where the file is failing to be sized to your new sizes

  • default discord avatar
    arielarial
    6 months ago

    hmm, ok. So, about old sizes being re-generated: they are. I've just checked in GCS bucket



    About an already built-in payload config, I don't see how this could be. Because I'm testing this locally, and even restarted manually payload to double check



    But I guess the way to go really is to set a debugger and try to find this out, like you said. Because the way it is now, we are shooting in the dark :/

  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    in your script, are you calling

    payload.init

    ?



    i'd double-check that it's loading your

    /src

    payload config, and not the

    /dist

    one



    (that is, if you're using TS)

  • default discord avatar
    arielarial
    6 months ago

    OH.

  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    😏



    you should definitely make this script into a GH discussion!



    would be super valuable for others i think

  • default discord avatar
    arielarial
    6 months ago

    of course!



    so, I'm using payload.init like this:


    await payload.init({
          secret: PAYLOAD_SECRET,
          mongoURL: MONGODB_URI,
          local: true,
        });


    I'm not specifying the config file ,so it will default to

    /dist

    , thats it?

  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    well that depends



    where is your payload config located?



    and / or, do you have an environment variable set to manually declare

    PAYLOAD_CONFIG_PATH


    it won't default to

    /dist
  • default discord avatar
    arielarial
    6 months ago

    oh, thats right



    oh my...



    "regenerateImages": "cross-env PAYLOAD_CONFIG_PATH=dist/payload.config.js node ./regenerateMediaSizes.js",


    🤡

  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    nailed it



    hey, love it when we get the answers

  • default discord avatar
    arielarial
    6 months ago

    That one took a while!



    Thank you so much, James!

  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    of course



    thanks for staying after it 👍

  • default discord avatar
    arielarial
    6 months ago

    I'll definitely add this to the thread 🙂



    thanks for the patience! 😅



    snippet here:


    https://github.com/payloadcms/payload/discussions/1834#discussioncomment-5327718
  • default discord avatar
    noheadphones
    6 months ago

    nice work, feels like this should be part of the plugin somewhat



    or part of a core, or a combination 😂

Open the post
Continue the discussion in Discord
Like what we're doing?
Star us on GitHub!

Star

Connect with the Payload Community on Discord

Discord

online

Can't find what you're looking for?

Get help straight from the Payload team with an Enterprise License.