Can I replace a S3 bucket with local media storage?

snailedlt2 years ago

18

• 

  tylandavis

  2 years ago

  Hey

  @290479081865084929

  , it would depend on where you deploy your project. The

  next-payload

  repo that this demo is based on is designed to be deployable on Vercel, which is serverless, and therefor doesn't have persistant file storage.

• 

  snailedlt2 years ago

  I see, that clears things up quite a bit, thanks!

  
  
  

  If I were to test locally though, is there any reason why simply removing the cloudstorage plugin wouldn't work?

  
  

  // Can I remove this code to store the images locally?
  cloudStorage({
        collections: {
          'media': {
            adapter,
            disablePayloadAccessControl: true,
          }
        },
      }),

• 

  tylandavis

  2 years ago

  I'm not super familiar with the repo, but I think you might run into issues with the files not saving properly. Because this repo is optimized for serverless, there might be some changes made that prevent that.

• 

  snailedlt2 years ago

  I see, thank you very much for all the help. I'm still quite new to payload so having active support from the devs like this is enormously helpful!

  
  
  @783701636165402624

  I only need the S3 bucket for the Media collection right? Looking at the code (

  https://github.com/search?q=repo%3Apayloadcms%2Fnext-payload-demo%20s3&type=code

  ) it doesn't seem like it's not used for anything else, but just want to make sure there's no magic going on in the background

  
  
  

  I'm mainly wondering because I removed the Media collection and all of the S3 code references, but I'm still getting this error in vercel:

  
  

  Error: The Serverless Function "index" is 63.48mb which exceeds the maximum size limit of 50mb. Learn More: https://vercel.link/serverless-function-size

  
  
  

  Which leads me to believe Payload CMS might be doing some magic with other static files when the cloudStorage plugin is enabled?

• 

  tylandavis

  2 years ago

  Were you getting that error before removing the S3 stuff? It could be related to something else.

• 

  snailedlt2 years ago

  I didn't connect with S3 in the first place since I don't have an amazon account 😓 Do you have any ideas what it could be related to?

  
  
  

  When I build locally everything except the cache comes out to 22.9MB or 23.1MB on disc. The cache alone is 582MB though

• 

  tylandavis

  2 years ago

  "index" in the error message makes me think this is likely a component somewhere causing the issue, but I don't know why one would be 63.48mb

• 

  snailedlt2 years ago

  I was saving static files in the public folder. Gonna try to remove them and see if that was the reason

• 

  tylandavis

  2 years ago

  That could be it. I think if you are importing files into a server component, then the file size might include those imports.

• 

  snailedlt2 years ago

  sadly that wasn't the issue :/

  
  

  It's now 63.49, so .01 higher than before

  
  
  

  It couldn't be this right?

  
  
  

  That's the only component I've added to payload.config.ts

  
  
  
  
  

  Seems like it might be an ongoing issue with next:

  https://github.com/vercel/next.js/issues/42641
• 

  tylandavis

  2 years ago

  Yeah it looks like

  swc/core

  is the common thread here

• 

  snailedlt2 years ago

  yup :/

  
  
  

  According to a lot of comments on the github issue the fix seems to be to add this to next.config.js

  
  

  outputFileTracingExcludes: {
    '*': [
        './node_modules/@swc/core-linux-x64-gnu',
        './node_modules/@swc/core-linux-x64-musl',
    ],
  },

  
  
  

  But that's already added through the withPayloadMethod here:

  
  https://github.com/payloadcms/next-payload/blob/b2c306894eefd21131ff8ca6d4c322ef5d9506f0/withPayloadPlugin.js#L39-L54
  
  

  Which we use like this:

  
  

  // next.config.js
  
  const { withPayload } = require("@payloadcms/next-payload");
  const path = require("path");
  
  /** @type {import('next').NextConfig} */
  const nextConfig = withPayload(
    {
      reactStrictMode: true,
      transpilePackages: ["@payloadcms/plugin-seo", "ui"],
      experimental: {
        serverActions: true,
      },
      images: {
        domains: [
          "localhost",
          "nextjs-vercel.payloadcms.com",
          process.env.NEXT_PUBLIC_APP_URL,
          `${process.env.NEXT_PUBLIC_S3_ENDPOINT}`.replace("https://", ""),
        ],
      },
      eslint: { ignoreDuringBuilds: true },
      typescript: { ignoreBuildErrors: true },
    },
    {
      configPath: path.resolve(__dirname, "./payload/payload.config.ts"),
    }
  );
  
  module.exports = nextConfig;

  
  
  

  Could the implementation of withPayloadMethod be wrong?

  
  
  

  For example I noticed that the withPayloadMethod uses

  
  

  "**/*": [
        "node_modules/@swc/core-linux-x64-gnu",
        "node_modules/@swc/core-linux-x64-musl",
        ...
        ],

  
  

  instead of

  
  
  

    '*': [
        './node_modules/@swc/core-linux-x64-gnu',
        './node_modules/@swc/core-linux-x64-musl',
    ],

  
  
  

  Could the fact that I put the project in a monorepo (turborepo + pnpm-workspaces) have something to do with it perhaps?

  
  
  @364124941832159242

  It seems you were the one that added

  outputFileTracingExcludes

  to withPayload here:

  https://github.com/payloadcms/next-payload/commit/e4d62322f03dfd87fcb84375a0eeee89fb6d9252

  . Do you have any idea why the excluded node_modules still show up in the logs on vercel?

  
  
  

  ref:

  https://discord.com/channels/967097582721572934/1141325269383258152/1141776387540254730
  
  

  Note that I'm using payload inside of a turborepo with pnpm-workspaces, and the payload app is based on the official

  next-payload-demo

  repo

  
  
  

  It works with

  outputFileTracingIgnores.

  Seems like there's a bug currently where it doesn't work with

  outputFileTracingExcludes

  , but it does with

  outputFileTracingIgnores

  .

  
  

  See

  https://github.com/vercel/next.js/issues/54245

  for more info.

  
  
  

  Probably would be good to change the

  next-payload

  code to use

  outputFileTracingIgnores

  until it's fixed

• 

  tylandavis

  2 years ago

  https://github.com/payloadcms/next-payload/blob/b2c306894eefd21131ff8ca6d4c322ef5d9506f0/withPayloadPlugin.js#L65
  
  

  I wonder, should line 65 be

  outputFileTracingExcludes: outputFileTracingExcludes,

  ?

  
  
  

  I'm not set up at the moment to test if that works, but I think we are passing the object to

  experimental

  instead of

  experimental.outputFileTracingExcludes
• 

  snailedlt2 years ago

  Yeah I think you're right

  
  
  

  Unfortunately I don't have much time to contribute to the project, so I hope someone else can fix it if that's the issue 🙂

  
  
  

  I can test the solution in the project I'm working on atm, but cba to setup next-payload locally, so it would have to be deployed somewhere

• 

  tylandavis

  2 years ago

  No worries, I'll see about testing that change and let you know what I find.

  
  
  

  Thanks so much for all the time you've spent looking into this, it's been super helpful!

• 

  snailedlt2 years ago

  Thanks the same! It's been tremendously helpful for us too!

  
  

  Let me know if you need me to test it once changes are implemented 🙂

• 

  jarrod_not_jared

  2 years ago

  If you just pass a variable into an object like that, it will key it with the var name and the contents of the var will be the value of it.

• 

  tylandavis

  2 years ago

  okay, I wasn't sure if that was the case or not