How does one handle syncing uploads between environments?

default discord avatar
Jakey
7 months ago
9

I've noticed that if you just copy an asset into the corresponding upload folder, it doesn't change anything in the CMS to allow you to see it / use it.



This makes me wonder how the use case of syncing your local images up to a remote host should be handled. Is there a "sync-to-remote" functionality in Payload I'm not aware of? Or will I have to write up a script that also merges in the data from local to remote?



Any feedback welcome, thank you.



So far the most common response I'm getting is to utilize a CDN instead of relying on local storage, which is probably why this use case isn't accounted for in Payload?

  • default discord avatar
    thgh
    7 months ago
  • default discord avatar
    see.ess
    7 months ago

    Are you talking uploading media in a local vs remote environment? Or are you asking for a way for Payload to detect you've copied a

    .jpg

    into your uploads folder and create a record for it?

  • default discord avatar
    Jakey
    7 months ago

    at my company, it's common to have a local, qa, and prod environment. local for code changes, qa for review of code changes, then prod is typically left for content management. We never sync data to prod since its data is most up to date. So as you might imagine that's an annoying scenario to deal with at times.



    In this workflow, we'll typically sync our media (images/videos/etc) to the media folder on the other two environments (since typically they don't override existing data, just add on). However, in both Payload and WP, the db state doesn't recognize the new images that were synced up and thus you can't just pick the new items for future content up dates either.



    I'm definitely in favor of instead relying on a CDN solution like that which @thgh shared. I was just curious if Payload had some way of addressing this since it is one of those annoying things that happens in WP for instance.

  • default discord avatar
    see.ess
    7 months ago

    Yeah, kinda figured that's what you're referring to... and I agree it's a pain (especially when building first-class features that depend on the presence of a database record). I've dealt with it in WordPress pretty easily with some abhorrent hacks... not as easy when you're trying to do things the right way. I'm working towards v1 of a site for my client now and am getting close to having to deal with this issue on Payload.



    Not sure if a more general mongo diff/migration tool would help?

    https://github.com/datacoon/diffmongo


    or here's a gist of something relatively simple that you might be able to build on:

    https://gist.github.com/niccottrell/3ff2efc2146090aa8c5b73c1eb75ba2c
  • default discord avatar
    Jakey
    7 months ago

    Interesting…. This seems like it’ll be useful anyway for comparing environment differences. Thanks!!

  • default discord avatar
    see.ess
    7 months ago

    fwiw, I managed to hack together this typescript version of the above gist...

    https://gist.github.com/marpstar/b257ceb628a428ec66aabde72c0cdbc4

    it runs on the entire collection, not just a sample. Also allows you to specify different DB names for each environment. I've included instructions for launching (and debugging) for

    ts-node
  • default discord avatar
    Jakey
    7 months ago

    much appreciated!

Open the post
Continue the discussion in Discord
Like what we're doing?
Star us on GitHub!

Star

Connect with the Payload Community on Discord

Discord

online

Can't find what you're looking for?

Get help straight from the Payload team with an Enterprise License.