I am getting an Error in the console while I try to implement the cloud storage plugin and using the s3 Adapter. File uploading works fine but when I refresh the collection in the admin panel this error is shown.

joshcoady2 years ago
seems like a mismatch somewhere between your cloudstorage plugin config for upload collection vs for the s3 adapter, can you share the config for both?

joelcorey2 years ago

I am getting a similar error when testing using Cloudflare r2 (s3 compatible).

[19:15:21] ERROR (payload): The specified key does not exist.
    err: {
      "type": "NoSuchKey",
      "message": "The specified key does not exist.",
      "stack":
          NoSuchKey: The specified key does not exist.
              at de_NoSuchKeyRes (/home/joel/dev/payload-bigtrashpanda-cms/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:6111:23)
              at de_GetObjectCommandError (/home/joel/dev/payload-bigtrashpanda-cms/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:4316:25)
              at processTicksAndRejections (node:internal/process/task_queues:95:5)
              at /home/joel/dev/payload-bigtrashpanda-cms/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
              at /home/joel/dev/payload-bigtrashpanda-cms/node_modules/@aws-sdk/client-s3/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:30:20
              at /home/joel/dev/payload-bigtrashpanda-cms/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:31:46
              at /home/joel/dev/payload-bigtrashpanda-cms/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/flexibleChecksumsMiddleware.js:63:20
              at /home/joel/dev/payload-bigtrashpanda-cms/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24
              at /home/joel/dev/payload-bigtrashpanda-cms/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20
              at /home/joel/dev/payload-bigtrashpanda-cms/node_modules/@aws-sdk/client-s3/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
      "name": "NoSuchKey",
      "$fault": "client",
      "$metadata": {
        "httpStatusCode": 404,
        "attempts": 1,
        "totalRetryDelay": 0
      },
      "Code": "NoSuchKey"
    }

But everything seems to be working. It uploads the media. I can access it after the fact in both Payload and on Cloudflare's dashboard I see the media. I was getting a region error, but after naming the region to "auto" from the Cloudflare documentation, it worked.

import path from 'path';
import type { CollectionConfig } from 'payload/types';

const Media: CollectionConfig = {
  slug: 'media',
  admin: {
    useAsTitle: 'alt',
    defaultColumns: [
      'id',
      'alt'
    ],
    group: 'Icons'
  },
  upload: {
    staticURL: '/media',
    staticDir: 'media',
    disableLocalStorage: true,
    // Specify the size name that you'd like to use as admin thumbnail
    adminThumbnail: 'thumbnail',
    imageSizes: [
      {
        height: 300,
        width: 400,
        crop: 'center',
        name: 'thumbnail', 
      },
      {
        width: 900,
        height: 450,
        crop: 'center',
        name: 'sixteenByNineMedium',
      },
    ],
  },
  fields: [
    {
      name: 'alt',
      type: 'text',
    }
  ],
};

export default Media;

Here is the collection in question.

joshcoady2 years ago
are you setting the prefix or generateFileURL options in the cloud plugin settings?

thinking you might need prefix set to media to match what statusURL is set to on the collection

joelcorey2 years ago

const adapter = s3Adapter({
  config: {
    endpoint: process.env.S3_ENDPOINT, // Configure for your provider
    credentials: {
      accessKeyId: process.env.S3_ACCESS_KEY_ID,
      secretAccessKey: process.env.S3_SECRET_ACCESS_KEY,
    },
        region: process.env.S3_REGION,
  },
  bucket: process.env.S3_BUCKET,
})

and

plugins: [
        // payloadCloud(),
        cloudStorage({
      collections: {
        'media': {
          adapter
        },
      },
    }),
    ],

I am noticing a lower case "media" when the collection is "Media". Could that be it? I am not familiar with generateFileURL or the statusURL that you mentioned

@562517587376996363

https://github.com/payloadcms/plugin-cloud-storage

After reading the docs there I am still uncertain on how to use that.

My confusion comes down to where do I put prefix?

joshcoady2 years ago
staticURL is in your collection config
```
  upload: {
    staticURL: '/media',
```
the other options are under "Collection-specific options" in
https://github.com/payloadcms/plugin-cloud-storage#plugin-options

I'm not 100% sure, but it seems like there is some sort of mismatch in the paths that it is requesting vs uploading to

I am noticing a lower case "media"
that's fine -- it's matching against the colleciton slug, not the class name
joelcorey2 years ago
I am not sure if it is in fact an issue with Cloudflare R2 bucket storage (
https://developers.cloudflare.com/r2/buckets/
). I don't think folders are supported.

"Buckets do not contain folders that group the individual files, but instead, buckets have a flat structure which simplifies the way you access and retrieve the objects in your bucket."

Which could explain my errors and the fact that it is actually working? Very confusing.

@1087460570749476994
Any progress on your error? I did not intend to steal your help thread.

I am starting to think that the payload cloud plugin and s3 stuff is just a waste of time. The amount of extra pain versus just using the database is beginning to seem to not be worth it.

And I did end up deploying my existing confits/code .. and it just works. It works with the existing errors above. I will have to circle back because today is a head scratcher.

I am willing to bribe someone and pay for a solution here.

I love Payload but I just can't figure this out.
joshcoady2 years ago
if you dont mind sharing, what is the value of process.env.S3_ENDPOINT and process.env.S3_BUCKET and what URL is payload outputting for the file and what URL does cloudflare say is the proper URL for the file?

I dont have experience with cloudflare so I can only make guesses. We have this setup and working with AWS S3 + CF and I dont remember any big probs with it
joelcorey2 years ago
Sure thing. Everything in this project is non sensitive. One sec.

S3_ENDPOINT="https://<cloudflare-account-number-here>.r2.cloudflarestorage.com" S3_BUCKET="bigtrashpanda-cms-media"

Local dev on my machine working. Notice the URL down in the lower left corner.

Deployed code base that is not working correctly.

The url in the actual cloudflare dashboard is very long and will not have anythign to do with the error.

It looks like a CORS issue. I remember the NS BINDINGS error when I first started trying on Payload. I can't remember what fixed that though.

Looks like it cut the screen shots so the URL is not visible. The local machine is localhost:9001/media/meme-example.jpg and deployed is
https://bigtrashpanda-cms.springtree.xyz/media/meme-example.jpg

^ Sorry about that

Also thank you at least taking a look and trying.
joshcoady2 years ago
Some of this is starting to come back to me 🙂

So this default way of handling it, the payload server acts as a proxy, so it seems like payload is not correctly retrieving the file from cloudflare and so it is not able to pass it on

I'm not sure if any of those issues are being logged anywhere, but it may not be passed to the client

Though I wonder if that last entry in the network tab you shared (the 404) has any hints in the response headers or in the body
joelcorey2 years ago
As requested
joshcoady2 years ago
The config we ended up going with does not use Payload as a proxy to serve the files and instead configures payload to directly link to them on our CDN (cloudfront). I dont know if that is an option you have?

I actually wanted the only below that, the 404 🙂

it seems like payload is doing a second try at it internally and getting a 404 that time, so it'd be interesting to see what the result of that was
joelcorey2 years ago
It's a fairly stock config. Only "fancy" thing we have really is the cloud plugin.

https://pastes.io/ak0npisbek

Discord isn't letting me paste directly here because it's to long of a post
joshcoady2 years ago
if you click the preview sizes button, do those show up correctly?
joelcorey2 years ago
No
joshcoady2 years ago
hmm, sorry I havent been much help. Not sure how much more I can do without setting up a cloudflare ccount on my end 🙂

I did want to note that you should check out the yellow box on
https://payloadcms.com/docs/upload/overview#disabling-local-upload-storage

And also in the plugin's example, they dont have either of the static settings on the collection, so maybe try removing those?
https://github.com/payloadcms/plugin-cloud-storage/blob/master/dev/src/collections/Media.ts
joelcorey2 years ago
For your first point, then why does it work on local by not while deployed. And for your second point, I believe those static options are marked as required. Not trying to be difficult, just attempting to give you a snapshot of where my mind is on this issue. Also, never apologize for trying to help. We can schedule a Zoom call if you would like early next week. I would 100% pay you for your time. You have already been helpful. 🙂 Thank you!
varghesetony9012 years ago
hey this is informative. I'm not sure how the deployed version behaves but on development mode everything works fine but the error sucks. No updates till.

I thought I might have end up in configuring the s3 bucket wrong. No idea where the actual problem is.
joelcorey2 years ago
Yeah it really seems to be a wtf situation with the previews working locally but not when deployed. I will probably just have a local Mongo instance running on the same machine and be done with this cloud storage ridiculousness.

I switched back to Digital Ocean Spaces, S3 and R2 equivalent, and it just works.
brianjm2 years ago
I integrated Minio today for local development (Docker). With Minio, I can use payload hooks to modify the prefix (or filename) to force the creation of a directory (prefix/custom-dir/filename.ext). This works with private buckets without a CDN .

I'll try R2 in the coming weeks and let you know how that works.

@423230148721246209
The prefix goes in the build config and it can be adjusted with hooks. This prefix
does
resolve to a "folder" in your bucket.

R2 doesn't have "real folders" but the "folder" is part of the filename/path. Confusing, but essentially folders are created when a folder is defined in the prefix or filename. Folders are "created" when one file exists, and "folders" are removed when no files exist. There no way to create or remove folders otherwise.
joelcorey2 years ago
I ended up switching over to Google Cloud Storage for now. I will consider this resolved in that I just stopped using Cloudflare's R2 product.
brianjm2 years ago
FWIW, I received a similar error when attempting to add an additional folder to the prefix. The error is similar to yours upon retrieval, and it's because the path after the first folder is dropped. Saving is fine.

Fixed it. Uploads are saved into subfolders by modifying the prefix within the beforeChange hook. There is no change to the URL, as the files are still supposed to be unique (within Payload).
joelcorey2 years ago
Do you have an example Collection that I can see?
sandrowegmann2 years ago
This might not be related to the error, but we've been facing similar issues for quite a while now using linodes s3 Buckets. On a fresh deploy, everything works fine, up until a point where we get multiple "no Such Key" Errors. The files in the bucket are fine, but if you try to access them through payload, it just keeps loading indefinitely.

It was never an issue on my local machine, but that might be due to frequent server restarts during development
shamim85372 years ago

Why did this error? I have set my password properly.

brianjm2 years ago

@423230148721246209

Sorry for the delay, but here is the hook I use for

upload enabled collections

. This allows me to create "paths" in s3 the include the

collection name

and

object id

uploadEnabled.ts

import { CollectionBeforeChangeHook } from 'payload/types'

import ObjectID from 'bson-objectid'

export const appendIdToPrefixCollectionBeforeChangeHook: CollectionBeforeChangeHook = async ({
  data, // incoming data to update or create with
  req, // full express request
  operation, // name of the operation ie. 'create', 'update'
  originalDoc, // original document
}) => {
  if (operation !== 'create') return data

  // Create and append an `_id` to the `prefix`
  const objectID = ObjectID().toHexString()
  data._id = objectID
  data.prefix = `${data.prefix}/${objectID}`
  
  // Save the original name, only if a file is provided
  if (req?.files?.file?.name) data.originalName = req.files.file.name

  return data
}

payload.config.ts

export default buildConfig({
  //...
  plugins: [
    //...
    cloudStorage({
      collections: {
        attachments: {
          adapter: s3AdapterConfig,
          disableLocalStorage: true,
          prefix: 'attachments',
        },
        documents: {
          adapter: s3AdapterConfig,
          disableLocalStorage: true,
          prefix: 'documents',
        },
      },
    }),
  ],

Attachments.ts

import { CollectionConfig } from 'payload/types'

import { layout } from './imports/layout'

import { mimeTypes } from '../_config/uploadEnabled'
import { appendIdToPrefixCollectionBeforeChangeHook} from '../../hooks/collection/uploadEnabled'

const Attachments: CollectionConfig = {
  slug: 'attachments',
  admin: {
    useAsTitle: 'filename',
    defaultColumns: ['filename', 'documentName', 'createdAt', 'updatedAt'],
    disableDuplicate: true,
  },
  upload: {
    filesRequiredOnCreate: true,
    mimeTypes: mimeTypes.attachments,
  },
  fields: layout,
  hooks: {
    beforeChange: [
      appendIdToPrefixCollectionBeforeChangeHook,
    ],
  },
}

export default Attachments

marcusr51352 years ago
@1090615594568667286
Big thanks for this - I've been fighting similar all day trying to get "paths" to create on Azure via the upload plugin. Spent a long time trying to pass through a filename with a "prefix/filename" format which is normal for uploads direct to Azure only to figure out eventually that the code in
https://github.com/payloadcms/payload/blob/main/packages/payload/src/uploads/generateFileData.ts
is sanitizing the filename and stripping out the "/" so it will never work. Was looking everywhere for ways round this and your manipulation of the prefix works great. Saved my sanity tonight - thank you!

I think the sanitize-filename package used might be OK for local file storage but doesn't align well to how the cloud providers treat folders/paths which is as part of the filename with a "/". Perhaps something the Payload team might look at in time.......
brianjm2 years ago
@973872398787047444
You're welcome! I'm glad this worked for your use case as well. 🙂
marcusr51352 years ago
In the end I got stuck with Payload wanting globally unique filenames which breaks my use case (my files are in unique folders via the prefix but not uniquely named). MongoDB has a unique index added to the filename which I can't remove and so. I looked through the issues and discussions and added more details to this discussion which I think would resolve our use cases (just allow "/" in the filename!). Feel free to comment there or add any insights you have too.

https://github.com/payloadcms/payload/discussions/2911

Open

Continue the discussion in Discord

Star on GitHub

Star

Chat on Discord

Discord

online

Can't find what you're looking for?

Get dedicated engineering support directly from the Payload team.