Generating pre-signed URLs with payload-cloudstorage

akshay

7 months ago

25

cloudStorage({
      collections: {
        video: {
          adapter: adapter,
          disablePayloadAccessControl: true,
          generateFileURL: function (param) {
            const getCommand = new GetObjectCommand({
              Bucket: process.env.S3_BUCKET,
              Key: param.filename,
            });
            const get = getSignedUrl(s3, getCommand, { expiresIn: 3600 })
            return get;
          }
        },
      },
    }),

WARNING in ./src/payload.config.ts 30:15-23
export 'S3Client' (imported as 'S3Client') was not found in '@aws-sdk/client-s3' (possible exports: HeadObjectCommand, S3)

WARNING in ./src/payload.config.ts 82:33-49
export 'GetObjectCommand' (imported as 'GetObjectCommand') was not found in '@aws-sdk/client-s3' (possible exports: HeadObjectCommand, S3)

webpack compiled with 2 warnings

• 

  jmikrut

  Payload Team

  7 months ago

  it looks like the Webpack alias that we have in place is not working for you

  
  

  
  

  are you doing anything to your Webpack config?

• 

  akshay

  7 months ago

  no, I'm actually not very familiar with how webpack works and how it might be affecting.

  
  

  
  

  Actually it seems like cloudstorage plugin modified the webpack config -

  https://github.com/payloadcms/plugin-cloud-storage/blob/f333ff1c5bad9ef04124b04a9fd19edbdc7c0409/src/webpack.ts#L1
• 

  jmikrut

  Payload Team

  7 months ago

  Yep that's what I was sayin,

  we modify the Webpack config FOR you

  because there are some server-only packages that should be aliased so they don't get included in the browser JS

  
  

  
  

  but for whatever reason, it looks like yours are not working

  
  

  
  

  what OS are you on?

• 

  akshay

  7 months ago

  I'm on macOS Monterey

• 

  jmikrut

  Payload Team

  7 months ago

  ok me too

  
  

  
  

  ohh i see

  
  

  
  

  your

  generateFileURL

  function itself

  
  

  
  

  is using methods that are not mocked appropriately

  
  

  
  

  so if i were you, i'd put that function in another file, and then mock

  that

  function with a Webpack alias

  
  

  
  

  basically exclude it from the admin bundle

  
  

  
  

  because the admin UI doesn't run that function, only the backend. so your goal is to add an alias to just straight up exclude it

  
  

  
  

  https://payloadcms.com/docs/admin/webpack#aliasing-server-only-modules
• 

  akshay

  7 months ago

  thanks, let me give that a try.

  
  

  
  

  Still no luck, followed the example in docs -

  
  
  

  I created s3.js -

  
  
  import { getSignedUrl } from "@aws-sdk/s3-request-presigner"; import { GetObjectCommand, S3Client } from "@aws-sdk/client-s3"; export const s3Config = { endpoint: process.env.S3_ENDPOINT, credentials: { accessKeyId: process.env.S3_ACCESS_KEY_ID, secretAccessKey: process.env.S3_SECRET_ACCESS_KEY, }, region: process.env.REGION } const s3 = new S3Client(s3Config); export function generateFileURL(param) { const getCommand = new GetObjectCommand({ Bucket: process.env.S3_BUCKET, Key: param.filename, }); const get = getSignedUrl(s3, getCommand, { expiresIn: 3600 }); return get; }
  
  

  and mocks/emptyObject.ts

  
  
  export default {};
  
  

  webpack overrides

  
  
  import { s3Config, generateFileURL } from "./utilities/s3"; const s3Path = path.resolve(__dirname, 'utilities/s3.js'); const mockModulePath = path.resolve(__dirname, 'mocks/emptyObject.js'); // config overrides admin: { components: {}, webpack: (config) => ({ ...config, resolve: { ...config.resolve, alias: { ...config.resolve.alias, [s3Path]: mockModulePath, } } }) }
  

  
  

  I get this error from webpack -

  
  
  Watchpack Error (initial scan): Error: ENOTDIR: not a directory, scandir '/Users/akshay/Projects/r/website-cms/src/utilities/s3.js' Actually nvm, I got it working. Had to update my mocks accordingly -

  export const s3Config = {}

  
  

  export const generateFileURL = {};

  
  

  `

  
  

  
  

  Is there a nicer way to do this, rather than having to mock each export?

• 

  Domosaurus

  2 months ago

  Hi, I'm getting the same issue here also:

  
  Watchpack Error (initial scan): Error: ENOTDIR: not a directory, scandir '<xxx>/src/utils/generateFileUrl.ts'

  .

  
  
  

  I have deleted the

  node_modules/.cache/webpack

  as outlined in the documentation

  https://payloadcms.com/docs/admin/webpack#aliasing-server-only-modules

  but unfortunately it does not work.

  
  
  

  In

  src/utils/generateFileUrl.ts

  :

  
  

  import { fromIni } from "@aws-sdk/credential-providers";
  import { Hash } from "@aws-sdk/hash-node";
  import { HttpRequest } from "@aws-sdk/protocol-http";
  import { S3RequestPresigner } from "@aws-sdk/s3-request-presigner";
  import { parseUrl } from "@aws-sdk/url-parser";
  import { formatUrl } from "@aws-sdk/util-format-url";
  import type { GenerateFileURL } from "@payloadcms/plugin-cloud-storage/dist/types";
  import { S3AdapterConfig } from "./s3Adapter";
  
  // Ref: https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html
  const generateFileURL: GenerateFileURL = async param => {
    const url = parseUrl(`https://${S3AdapterConfig.bucket}.s3.${S3AdapterConfig.config.region}.amazonaws.com/${S3AdapterConfig.bucket}/${param.filename}`);
    const presigner = new S3RequestPresigner({
      credentials: fromIni(),
      region: S3AdapterConfig.config.region,
      sha256: Hash.bind(null, "sha256")
    });
    const signedUrlObject = await presigner.presign(new HttpRequest(url), { expiresIn: 604800 });
    const result = formatUrl(signedUrlObject);
    return result;
  };
  
  export default generateFileURL;

  
  
  

  In

  payload.config.ts

  :

  
  
  

  const generateFileUrlPath = path.resolve(__dirname, "utils/generateFileUrl.ts");
  const mockEmptyObjectPath = path.resolve(__dirname, "mocks/emptyObject.ts");
  
  // [...]
  export default buildConfig({
   // [...]
   webpack: config => {
    return {
      ...config,
      resolve: {
        ...config.resolve,
        alias: {
          ...config.resolve.alias,
          [generateFileUrlPath]: mockEmptyObjectPath
        }
      }
    };
   }
   // [...]
  });

  
  
  

  In

  mocks/emptyObject.ts
  

  export default {};

• 

  jmikrut

  Payload Team

  2 months ago

  Just remove the

  .ts

  off

  generateFileUrl.ts

  in your alias

• 

  Domosaurus

  2 months ago

  ah okay, i'll give it a shot tomorrow. Will update!

  
  

  
  

  yeap that solved it thank you!