Payload Cloud Storage Plugin with google-cloud/storage

default discord avatar
Step
2 months ago
44

I have configured the plugin according to the documentation, but it does not save to the bucket configured in Cloud Storage, maybe I am configuring something wrong. Please help!

  • discord user avatar
    jmikrut
    Payload Team
    2 months ago

    are your credentials correct?



    config looks correct

  • default discord avatar
    Step
    2 months ago

    My credentials are correct, maybe the error is coming from the

    @google-cloud/storage

    library, I'm using version

    6.9.4 (2023-03-02)
  • default discord avatar
    Tinouti
    2 months ago

    I just set this up myself on my local Payload, after some scrambling around to figure out how to get credentials to begin with using gcloud cli and what my bucket name should be, it's now working seamlessly!



    I'm using:


    "@google-cloud/storage": "^6.9.4",
    "@payloadcms/plugin-cloud-storage": "^1.0.14",
    "payload": "^1.6.19",


    What kind of error are you getting @Step ?

  • default discord avatar
    Step
    2 months ago

    I am not getting an error, but it is not saving a GCS bucket either.


    I am using the following versions:


    "@google-cloud/storage": "6.9.4", "@payloadcms/plugin-cloud-storage": "^1.0.14", "payload": "^1.6.22",
  • default discord avatar
    Tinouti
    2 months ago

    Hmm, so let's summarize: you go to add a new item in your

    Media

    collection, select your image, click "save". Then what happens?



    Does the image show up?


    Can you check what url it's trying to display the image from? (that's how I realized that my bucket setting was wrong)


    And you're sure you don't have any error, whether in the browser or in your server logs?

  • default discord avatar
    Step
    2 months ago

    I have this problem, I don't know what the problem is. These are my dependencies:

    "dependencies": { "@google-cloud/storage": "^6.9.4", "@payloadcms/plugin-cloud-storage": "^1.0.14", "dotenv": "^8.2.0", "express": "^4.17.1", "payload": "^1.6.22" },
    Erorr.png
  • default discord avatar
    Tinouti
    2 months ago

    I'm afraid I won't be able to help much more with this, sorry! 🤷‍♂️

  • default discord avatar
    Wandering Nerd
    2 months ago

    I faced that polyfill error once due to declaring global variables and accessing it in a non-appropriate place. Maybe this will give you some hints.

  • default discord avatar
    sam
    2 months ago

    Also have this issue, came here to see if anyone had found a solution.



    I'm on Payload@1.6.28



    I've tried

    resolve.fallback

    for the various modules that it's complaining about. I can get webpack to compile, but then the browser complains about Buffer not being available. From there, I have tried adding the webpack.ProvidePlugin to the webpack config for buffer (as recommended by many people online), however, importing webpack brings in a bunch of other errors

  • discord user avatar
    jmikrut
    Payload Team
    2 months ago

    i can answer here quick



    need a few mins



    OK so here's the deal



    some of the Node packages that the cloud storage plugin requires are ONLY usable within Node, and break the browser (as is shown in the above screenshot)



    so what we suggest to do in order to "get rid" of server-side only code for the admin UI, is to use Webpack

    aliases


    for some background, the docs say a bit more about this here



    https://payloadcms.com/docs/admin/webpack#aliasing-server-only-modules


    BUT - the plugin itself should handle aliasing all of the things

    it

    relys on internally

    for

    you



    so in general, you should not need to do any manual aliasing yourself - the plugin should just do it for you



    and the package that the plugin DOES take care of aliasing for you is the

    @google-cloud/storage

    package



    but i am noticing in the screenshot above that your code is erroring on

    @google-cloud/paginator


    so i suspect that you are using some additional google cloud packages in your own code, and that means you likely also need to manually alias

    those

    packages

  • default discord avatar
    sam
    2 months ago

    For my project, the @google-cloud/storage is the only one from google. I think the paginator one is coming from that dependency, too, based on its package.json:

    https://github.com/googleapis/nodejs-storage/blob/main/package.json
  • default discord avatar
    Alessio 🍣
    2 months ago

    Are you importing anything from

    @google-cloud/storage

    anywhere in your project?



    (so in your own code, not including the payload cloud storage plugin source code)

  • default discord avatar
    sam
    2 months ago

    No, nowhere

  • default discord avatar
    Alessio 🍣
    2 months ago

    hm strange, maybe some google cloud storage update started leaking that package it depends on which it didn't before, and thus isn't covered by the aliases already included in the plugin

  • default discord avatar
    sam
    2 months ago

    It could very well be an issue with my setup, I have payload in an nx monorepo



    Will take a look at all the tsconfigs etc to make sure there's nothing weird

  • default discord avatar
    Alessio 🍣
    2 months ago

    also try implementing it in a test clean installation, so no monorepo. That'd tell if there's a potential with the plugin, or rather with your setup

  • default discord avatar
    sam
    2 months ago

    Clean installation looks good, so yeah, a problem with my setup.



    Out of interest, is there an easy way to get the full sanitised Payload config? I want to compare with the fresh installation, and make sure the aliases are being added to the webpack config



    So I think I have it working. In my project I wanted to allow local storage to be used for local development, and then production deployments to use GCS.



    This is what it looked like before:



    const cloudStorageCollectionOptions: CollectionOptions = {
      adapter:
        process.env.GCLOUD_CREDENTIALS && process.env.GCS_BUCKET
          ? gcsAdapter({
              options: {
                credentials: JSON.parse(process.env.GCLOUD_CREDENTIALS),
              },
              bucket: process.env.GCS_BUCKET,
            })
          : null,
      disableLocalStorage: !!(
        process.env.GCLOUD_CREDENTIALS && process.env.GCS_BUCKET
      ),
      disablePayloadAccessControl: true,
    };


    Then I would do something like



      cloudStorage({
          collections: {
              imageCollection1: cloudStorageCollectionOptions
              ...
              imageCollectionX: cloudStorageCollectionOptions  
          },
      }),


    What seems to work now, is to always set the adapter (and pass in sensible defaults if the env variables aren't present):



    const adapter = gcsAdapter({
      options: {
        credentials: JSON.parse(process.env.GCS_CREDENTIALS || '{}'),
      },
      bucket: process.env.GCS_BUCKET || '',
    });


    And in the plugin config, conditionally pass the adapter and set the localStorage based on the env variables:



    cloudStoage({
        collections: {
            imageCollection1: {
                adapter:
                    process.env.GCS_CREDENTIALS && process.env.GCS_BUCKET
                        ? adapter
                        : undefined,
                disableLocalStorage: !!(
                    process.env.GCS_CREDENTIALS && process.env.GCS_BUCKET
                ),
                disablePayloadAccessControl: true,
            }
        }
    })


    Though I'm not exactly sure why it works 😅



    If I log the webpack config, this one returns



    { resolve: { alias: {} } }



    Whereas the working one has the correct alias present



    Looking at the source code, I see that gcsAdapter is the thing that extends the webpack config.



    My best guess is, since the browser admin panel doesn't receive the process.env variables, it doesn't call gcsAdapter which means the webpack config doesn't get extended with the alias



    The second way works when serving but building the admin panel results in the errors from before. It's related to conditionally setting the adapter, if I set the adapter all the time it will build, however, an error is thrown because the bucket name and credentials are empty.



    Is there a recommended way to allow for the local filesystem to be used if no bucket or credentials are set?



    I believe I've found an OK solution, documented here:

    https://gist.github.com/sdjnes/80248ae5f268f0aba2201f9be5d5edc1

    Using the

    noParse

    configuration in webpack when the local filesystem should be used seems to work

  • discord user avatar
    Jarrod
    Payload Team
    last month

    Payload Cloud Storage Plugin with google-cloud/storage

  • default discord avatar
    jeet_A
    4 weeks ago

    I am facing same issue , I am mainly trying to test gcs is working in local or not . but i have tired with the solution but it is not working I am still getting list of errors .. any other possible way to fix this issue .. ?

  • default discord avatar
    sam
    4 weeks ago

    I think I ended up tweaking this. I will update you (probably tomorrow)



    What I ended up with was the following:



    I have all the image collections in an ImageCollections array, just makes it easier to define the adapter for all of them



    const adapter = gcsAdapter({
      options: {
            credentials: JSON.parse(process.env.GCS_CREDENTIALS || '{}'),
          },
      bucket: process.env.GCS_BUCKET || '',
    });
    
    const USE_LOCAL_STORAGE = !process.env.GCS_BUCKET;
    
    const cloudStorageCollections = ImageCollections.reduce(
      (acc, curr) => ({
        ...acc,
        [curr.slug]: {
          adapter: !USE_LOCAL_STORAGE && adapter,
          disableLocalStorage: !USE_LOCAL_STORAGE,
          disablePayloadAccessControl: true,
        },
      }),
      {}
    );
    
    
    // Then, inside your payload config
    export default buildConfig({
     ...
    webpack: (config) => ({
          ...config,
          resolve: {
            ...config.resolve,
            alias: {
              ...config.resolve?.alias,
              // See the README for why this is important
              '@google-cloud/storage': path.resolve(
                __dirname,
                './mocks/cloud-storage.js'
              ),
            },
          },
        }),
    plugins: [
        cloudStorage({
          collections: cloudStorageCollections,
        }),
      ],
    ...
    })


    The mock is nothing special as it won't actually be called (it's just so the imports don't pull in node API things for the browser bundle):



    exports.Storage = function () {
      return null;
    };
  • default discord avatar
    jeet_A
    4 weeks ago

    Thanks Sam @sam it worked . I was trying lot of things to fix this so things got messed up . tried with fresh project . it worked ..

  • default discord avatar
    sam
    4 weeks ago

    You're welcome 🙂

Open the post
Continue the discussion in Discord
Like what we're doing?
Star us on GitHub!

Star

Connect with the Payload Community on Discord

Discord

online

Can't find what you're looking for?

Get help straight from the Payload team with an Enterprise License.