Payload Cloud Storage Plugin with google-cloud/storage

steprob

6 months ago

44

• 

  jmikrut

  Payload Team

  6 months ago

  are your credentials correct?

  
  

  
  

  config looks correct

• 

  steprob

  6 months ago

  My credentials are correct, maybe the error is coming from the

  @google-cloud/storage

  library, I'm using version

  6.9.4 (2023-03-02)
• 

  tinouti

  6 months ago

  I just set this up myself on my local Payload, after some scrambling around to figure out how to get credentials to begin with using gcloud cli and what my bucket name should be, it's now working seamlessly!

  
  
  

  I'm using:

  
  

  "@google-cloud/storage": "^6.9.4",
  "@payloadcms/plugin-cloud-storage": "^1.0.14",
  "payload": "^1.6.19",

  
  
  

  What kind of error are you getting @steprob ?

• 

  steprob

  6 months ago

  I am not getting an error, but it is not saving a GCS bucket either.

  
  

  I am using the following versions:

  
  "@google-cloud/storage": "6.9.4", "@payloadcms/plugin-cloud-storage": "^1.0.14", "payload": "^1.6.22",
• 

  tinouti

  6 months ago

  Hmm, so let's summarize: you go to add a new item in your

  Media

  collection, select your image, click "save". Then what happens?

  
  
  

  Does the image show up?

  
  

  Can you check what url it's trying to display the image from? (that's how I realized that my bucket setting was wrong)

  
  

  And you're sure you don't have any error, whether in the browser or in your server logs?

• 

  steprob

  6 months ago

  I have this problem, I don't know what the problem is. These are my dependencies:

  "dependencies": { "@google-cloud/storage": "^6.9.4", "@payloadcms/plugin-cloud-storage": "^1.0.14", "dotenv": "^8.2.0", "express": "^4.17.1", "payload": "^1.6.22" },

  
• 

  tinouti

  6 months ago

  I'm afraid I won't be able to help much more with this, sorry! 🤷‍♂️

• 

  iokl

  6 months ago

  I faced that polyfill error once due to declaring global variables and accessing it in a non-appropriate place. Maybe this will give you some hints.

• 

  sam6466

  5 months ago

  Also have this issue, came here to see if anyone had found a solution.

  
  
  

  I'm on Payload@1.6.28

  
  
  

  I've tried

  resolve.fallback

  for the various modules that it's complaining about. I can get webpack to compile, but then the browser complains about Buffer not being available. From there, I have tried adding the webpack.ProvidePlugin to the webpack config for buffer (as recommended by many people online), however, importing webpack brings in a bunch of other errors

• 

  jmikrut

  Payload Team

  5 months ago

  i can answer here quick

  
  

  
  

  need a few mins

  
  

  
  

  OK so here's the deal

  
  

  
  

  some of the Node packages that the cloud storage plugin requires are ONLY usable within Node, and break the browser (as is shown in the above screenshot)

  
  

  
  

  so what we suggest to do in order to "get rid" of server-side only code for the admin UI, is to use Webpack

  aliases
  

  
  

  for some background, the docs say a bit more about this here

  
  

  
  

  https://payloadcms.com/docs/admin/webpack#aliasing-server-only-modules
  

  
  

  BUT - the plugin itself should handle aliasing all of the things

  it

  relys on internally

  for

  you

  
  

  
  

  so in general, you should not need to do any manual aliasing yourself - the plugin should just do it for you

  
  

  
  

  and the package that the plugin DOES take care of aliasing for you is the

  @google-cloud/storage

  package

  
  

  
  

  but i am noticing in the screenshot above that your code is erroring on

  @google-cloud/paginator
  

  
  

  so i suspect that you are using some additional google cloud packages in your own code, and that means you likely also need to manually alias

  those

  packages

• 

  sam6466

  5 months ago

  For my project, the @google-cloud/storage is the only one from google. I think the paginator one is coming from that dependency, too, based on its package.json:

  https://github.com/googleapis/nodejs-storage/blob/main/package.json
• 

  alessiogr

  Payload Team

  5 months ago

  Are you importing anything from

  @google-cloud/storage

  anywhere in your project?

  
  

  
  

  (so in your own code, not including the payload cloud storage plugin source code)

• 

  sam6466

  5 months ago

  No, nowhere

• 

  alessiogr

  Payload Team

  5 months ago

  hm strange, maybe some google cloud storage update started leaking that package it depends on which it didn't before, and thus isn't covered by the aliases already included in the plugin

• 

  sam6466

  5 months ago

  It could very well be an issue with my setup, I have payload in an nx monorepo

  
  

  
  

  Will take a look at all the tsconfigs etc to make sure there's nothing weird

• 

  alessiogr

  Payload Team

  5 months ago

  also try implementing it in a test clean installation, so no monorepo. That'd tell if there's a potential with the plugin, or rather with your setup

• 

  sam6466

  5 months ago

  Clean installation looks good, so yeah, a problem with my setup.

  
  
  

  Out of interest, is there an easy way to get the full sanitised Payload config? I want to compare with the fresh installation, and make sure the aliases are being added to the webpack config

  
  

  
  

  So I think I have it working. In my project I wanted to allow local storage to be used for local development, and then production deployments to use GCS.

  
  
  

  This is what it looked like before:

  
  
  

  const cloudStorageCollectionOptions: CollectionOptions = {
    adapter:
      process.env.GCLOUD_CREDENTIALS && process.env.GCS_BUCKET
        ? gcsAdapter({
            options: {
              credentials: JSON.parse(process.env.GCLOUD_CREDENTIALS),
            },
            bucket: process.env.GCS_BUCKET,
          })
        : null,
    disableLocalStorage: !!(
      process.env.GCLOUD_CREDENTIALS && process.env.GCS_BUCKET
    ),
    disablePayloadAccessControl: true,
  };

  
  
  

  Then I would do something like

  
  
  

    cloudStorage({
        collections: {
            imageCollection1: cloudStorageCollectionOptions
            ...
            imageCollectionX: cloudStorageCollectionOptions  
        },
    }),

  
  

  
  

  What seems to work now, is to always set the adapter (and pass in sensible defaults if the env variables aren't present):

  
  
  

  const adapter = gcsAdapter({
    options: {
      credentials: JSON.parse(process.env.GCS_CREDENTIALS || '{}'),
    },
    bucket: process.env.GCS_BUCKET || '',
  });

  
  
  

  And in the plugin config, conditionally pass the adapter and set the localStorage based on the env variables:

  
  
  

  cloudStoage({
      collections: {
          imageCollection1: {
              adapter:
                  process.env.GCS_CREDENTIALS && process.env.GCS_BUCKET
                      ? adapter
                      : undefined,
              disableLocalStorage: !!(
                  process.env.GCS_CREDENTIALS && process.env.GCS_BUCKET
              ),
              disablePayloadAccessControl: true,
          }
      }
  })

  
  

  
  

  Though I'm not exactly sure why it works 😅

  
  

  
  

  If I log the webpack config, this one returns

  
  
  

  { resolve: { alias: {} } }

  
  
  

  Whereas the working one has the correct alias present

  
  

  
  

  Looking at the source code, I see that gcsAdapter is the thing that extends the webpack config.

  
  
  

  My best guess is, since the browser admin panel doesn't receive the process.env variables, it doesn't call gcsAdapter which means the webpack config doesn't get extended with the alias

  
  

  
  

  The second way works when serving but building the admin panel results in the errors from before. It's related to conditionally setting the adapter, if I set the adapter all the time it will build, however, an error is thrown because the bucket name and credentials are empty.

  
  
  

  Is there a recommended way to allow for the local filesystem to be used if no bucket or credentials are set?

  
  

  
  

  I believe I've found an OK solution, documented here:

  https://gist.github.com/sdjnes/80248ae5f268f0aba2201f9be5d5edc1
  
  

  Using the

  noParse

  configuration in webpack when the local filesystem should be used seems to work

• 

  jarrod69420

  5 months ago

  Payload Cloud Storage Plugin with google-cloud/storage

• 

  jeet_adhikary

  4 months ago

  I am facing same issue , I am mainly trying to test gcs is working in local or not . but i have tired with the solution but it is not working I am still getting list of errors .. any other possible way to fix this issue .. ?

• 

  sam6466

  4 months ago

  I think I ended up tweaking this. I will update you (probably tomorrow)

  
  

  
  

  What I ended up with was the following:

  
  
  

  I have all the image collections in an ImageCollections array, just makes it easier to define the adapter for all of them

  
  
  

  const adapter = gcsAdapter({
    options: {
          credentials: JSON.parse(process.env.GCS_CREDENTIALS || '{}'),
        },
    bucket: process.env.GCS_BUCKET || '',
  });
  
  const USE_LOCAL_STORAGE = !process.env.GCS_BUCKET;
  
  const cloudStorageCollections = ImageCollections.reduce(
    (acc, curr) => ({
      ...acc,
      [curr.slug]: {
        adapter: !USE_LOCAL_STORAGE && adapter,
        disableLocalStorage: !USE_LOCAL_STORAGE,
        disablePayloadAccessControl: true,
      },
    }),
    {}
  );
  
  
  // Then, inside your payload config
  export default buildConfig({
   ...
  webpack: (config) => ({
        ...config,
        resolve: {
          ...config.resolve,
          alias: {
            ...config.resolve?.alias,
            // See the README for why this is important
            '@google-cloud/storage': path.resolve(
              __dirname,
              './mocks/cloud-storage.js'
            ),
          },
        },
      }),
  plugins: [
      cloudStorage({
        collections: cloudStorageCollections,
      }),
    ],
  ...
  })

  
  
  

  The mock is nothing special as it won't actually be called (it's just so the imports don't pull in node API things for the browser bundle):

  
  
  

  exports.Storage = function () {
    return null;
  };

• 

  jeet_adhikary

  4 months ago

  Thanks Sam @sam6466 it worked . I was trying lot of things to fix this so things got messed up . tried with fresh project . it worked ..

• 

  sam6466

  4 months ago

  You're welcome 🙂