Generating pre-signed URLs with payload-cloudstorage

default discord avatar
akshay
last year
25

I'm using payload cloudstorage plugin to upload videos and generate pre-signed public URLs. Running into an issue where s3 exports are not being recognized -



Here's what my plugin config looks like -



cloudStorage({
      collections: {
        video: {
          adapter: adapter,
          disablePayloadAccessControl: true,
          generateFileURL: function (param) {
            const getCommand = new GetObjectCommand({
              Bucket: process.env.S3_BUCKET,
              Key: param.filename,
            });
            const get = getSignedUrl(s3, getCommand, { expiresIn: 3600 })
            return get;
          }
        },
      },
    }),


I see the following warning on console, but none of the admin pages render -



WARNING in ./src/payload.config.ts 30:15-23
export 'S3Client' (imported as 'S3Client') was not found in '@aws-sdk/client-s3' (possible exports: HeadObjectCommand, S3)

WARNING in ./src/payload.config.ts 82:33-49
export 'GetObjectCommand' (imported as 'GetObjectCommand') was not found in '@aws-sdk/client-s3' (possible exports: HeadObjectCommand, S3)

webpack compiled with 2 warnings
  • discord user avatar
    jmikrut
    Payload Team
    last year

    it looks like the Webpack alias that we have in place is not working for you



    are you doing anything to your Webpack config?

  • default discord avatar
    akshay
    last year

    no, I'm actually not very familiar with how webpack works and how it might be affecting.



    Actually it seems like cloudstorage plugin modified the webpack config -

    https://github.com/payloadcms/plugin-cloud-storage/blob/f333ff1c5bad9ef04124b04a9fd19edbdc7c0409/src/webpack.ts#L1
  • discord user avatar
    jmikrut
    Payload Team
    last year

    Yep that's what I was sayin,

    we modify the Webpack config FOR you

    because there are some server-only packages that should be aliased so they don't get included in the browser JS



    but for whatever reason, it looks like yours are not working



    what OS are you on?

  • default discord avatar
    akshay
    last year

    I'm on macOS Monterey

  • discord user avatar
    jmikrut
    Payload Team
    last year

    ok me too



    ohh i see



    your

    generateFileURL

    function itself



    is using methods that are not mocked appropriately



    so if i were you, i'd put that function in another file, and then mock

    that

    function with a Webpack alias



    basically exclude it from the admin bundle



    because the admin UI doesn't run that function, only the backend. so your goal is to add an alias to just straight up exclude it



    https://payloadcms.com/docs/admin/webpack#aliasing-server-only-modules
  • default discord avatar
    akshay
    last year

    thanks, let me give that a try.



    Still no luck, followed the example in docs -



    I created s3.js -



    import { getSignedUrl } from "@aws-sdk/s3-request-presigner"; import { GetObjectCommand, S3Client } from "@aws-sdk/client-s3"; export const s3Config = { endpoint: process.env.S3_ENDPOINT, credentials: { accessKeyId: process.env.S3_ACCESS_KEY_ID, secretAccessKey: process.env.S3_SECRET_ACCESS_KEY, }, region: process.env.REGION } const s3 = new S3Client(s3Config); export function generateFileURL(param) { const getCommand = new GetObjectCommand({ Bucket: process.env.S3_BUCKET, Key: param.filename, }); const get = getSignedUrl(s3, getCommand, { expiresIn: 3600 }); return get; }

    and mocks/emptyObject.ts



    export default {};

    webpack overrides



    import { s3Config, generateFileURL } from "./utilities/s3"; const s3Path = path.resolve(__dirname, 'utilities/s3.js'); const mockModulePath = path.resolve(__dirname, 'mocks/emptyObject.js'); // config overrides admin: { components: {}, webpack: (config) => ({ ...config, resolve: { ...config.resolve, alias: { ...config.resolve.alias, [s3Path]: mockModulePath, } } }) }


    I get this error from webpack -



    Watchpack Error (initial scan): Error: ENOTDIR: not a directory, scandir '/Users/akshay/Projects/r/website-cms/src/utilities/s3.js' Actually nvm, I got it working. Had to update my mocks accordingly -

    export const s3Config = {}


    export const generateFileURL = {};


    `



    Is there a nicer way to do this, rather than having to mock each export?

  • default discord avatar
    Domosaurus
    10 months ago

    Hi, I'm getting the same issue here also:


    Watchpack Error (initial scan): Error: ENOTDIR: not a directory, scandir '<xxx>/src/utils/generateFileUrl.ts'

    .



    I have deleted the

    node_modules/.cache/webpack

    as outlined in the documentation

    https://payloadcms.com/docs/admin/webpack#aliasing-server-only-modules

    but unfortunately it does not work.



    In

    src/utils/generateFileUrl.ts

    :


    import { fromIni } from "@aws-sdk/credential-providers";
    import { Hash } from "@aws-sdk/hash-node";
    import { HttpRequest } from "@aws-sdk/protocol-http";
    import { S3RequestPresigner } from "@aws-sdk/s3-request-presigner";
    import { parseUrl } from "@aws-sdk/url-parser";
    import { formatUrl } from "@aws-sdk/util-format-url";
    import type { GenerateFileURL } from "@payloadcms/plugin-cloud-storage/dist/types";
    import { S3AdapterConfig } from "./s3Adapter";
    
    // Ref: https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html
    const generateFileURL: GenerateFileURL = async param => {
      const url = parseUrl(`https://${S3AdapterConfig.bucket}.s3.${S3AdapterConfig.config.region}.amazonaws.com/${S3AdapterConfig.bucket}/${param.filename}`);
      const presigner = new S3RequestPresigner({
        credentials: fromIni(),
        region: S3AdapterConfig.config.region,
        sha256: Hash.bind(null, "sha256")
      });
      const signedUrlObject = await presigner.presign(new HttpRequest(url), { expiresIn: 604800 });
      const result = formatUrl(signedUrlObject);
      return result;
    };
    
    export default generateFileURL;


    In

    payload.config.ts

    :



    const generateFileUrlPath = path.resolve(__dirname, "utils/generateFileUrl.ts");
    const mockEmptyObjectPath = path.resolve(__dirname, "mocks/emptyObject.ts");
    
    // [...]
    export default buildConfig({
     // [...]
     webpack: config => {
      return {
        ...config,
        resolve: {
          ...config.resolve,
          alias: {
            ...config.resolve.alias,
            [generateFileUrlPath]: mockEmptyObjectPath
          }
        }
      };
     }
     // [...]
    });


    In

    mocks/emptyObject.ts
    export default {};
  • discord user avatar
    jmikrut
    Payload Team
    10 months ago

    Just remove the

    .ts

    off

    generateFileUrl.ts

    in your alias

  • default discord avatar
    Domosaurus
    10 months ago

    ah okay, i'll give it a shot tomorrow. Will update!



    yeap that solved it thank you!

Open the post
Continue the discussion in Discord
Like what we're doing?
Star us on GitHub!

Star

Connect with the Payload Community on Discord

Discord

online

Can't find what you're looking for?

Get help straight from the Payload team with an Enterprise License.