Issues viewing a large collection (Over 8500 documents) with production mongo

default discord avatar
David
6 months ago
42

I have a collection with over 8500 documents and I'm getting an error in the payload client when trying to view the list of documents from our production mongo that says:



MongoServerError: PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting.

I've tried setting

indexSortableFields

to true on the payload.config. Does anyone have any ideas on what can be done here? We are on an M5 cluster tier with MongoDB.

  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    ok when you say

    payload client

    , do you mean the List view in the admin UI?



    we need to enable the aggregation property

    allowDiskUse


    I also assume that this is on a collection with versions enabled



    what is happening is that we need to sort a large set of documents, and you have too many which is causing memory issues



    i would suggest setting your versions / drafts config property

    maxPerDoc

    to a low number

  • default discord avatar
    David
    6 months ago

    Yes, I meant the list view in the admin UI.



    And yes, this is also on a collection with versions enabled. We only have 1 version for each doc at the moment but I'll try lowering the amount in the config to see if that helps.



    Is the property of

    allowDiskUse

    something we can enable on our end or would this require a payload update?



    Also with some further testing if I set maxPerDoc to 1 it still has the issue.



    If I disable versions for this collection the documents do render in the list UI however.

    image.png
  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    allowDiskUse needs to be set by Payload but you could directly edit your

    node_modules/payload/dist

    to see if that will fix your issue (i am pretty confident it will)



    file to be changed is here:



    https://github.com/payloadcms/payload/blob/master/src/versions/drafts/queryDrafts.ts#L54


    and here is a screenshot of me editing the

    /dist

    version:





    clearly i am feeling like a 🤠 right now

    Screen_Shot_2023-03-01_at_12.54.44_PM.png
  • default discord avatar
    David
    6 months ago

    Hmm i've just tried editing the file and restarting my local server but still having the same issue:

    image.png
  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    hmm, that seems strange



    same exact error?



    https://www.mongodb.com/docs/atlas/reference/free-shared-limitations/?_ga=2.176267877.1329169847.1677683154-860992573.1647438381#operational-limitations


    looks like M5 clusters don't support allowDiskUse, which is very good to know



    maybe we can figure out a smarter way around this

  • default discord avatar
    David
    6 months ago

    Oh



    ah i see:



    I'm going to try bumping to M10 in the interim but this interesting and also good to know!

    image.png
  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    ok - keep me posted



    i am not sure about how to get around this otherwise



    one other thing to note,

    allowDiskUse

    seems to be

    true

    by default on Mongo 6.0+



    so you might not need to hijack your

    node_modules

    like a maniac (me) if you are using 6.0+

  • default discord avatar
    David
    6 months ago

    We're on Mongo 5.0 at the moment. Will confirm in a couple minutes.



    @jmikrut


    Okay! I can confirm upgrading to an m10 cluster works!

  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    beautiful - did you still have your node_modules fix?



    or should we set this in code? PS - would be a great PR for you to make if you're up for it 👼

  • default discord avatar
    David
    6 months ago

    Just deleting the change to my node_modules...



    It actually seems to work just fine

  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    ok awesome



    sucky that the database has gotta be more expensive - i will keep this on the radar to see if it's possible to do this in a better way

  • default discord avatar
    David
    6 months ago

    Awesome thanks for the support! Yeah this will work fine for us in the interim but would be great for others if a similar issue didn't require the higher dedicated server.



    Sure I could. Would it make sense to hard code allowDiskUse in the query or have this as an optional parameter somewhere? I guess if the m10 cluster seems to solve it at mongo 5.0 though maybe not quite necessary yet?

  • discord user avatar
    jmikrut
    Payload Team
    6 months ago

    hard code makes sense



    i think it's only leveraged if it's necessary



    automatically

  • default discord avatar
    David
    6 months ago
    https://github.com/payloadcms/payload/pull/2216


    PR is here. Feel free to update wording in the note in docs.

Open the post
Continue the discussion in Discord
Like what we're doing?
Star us on GitHub!

Star

Connect with the Payload Community on Discord

Discord

online

Can't find what you're looking for?

Get help straight from the Payload team with an Enterprise License.