Simplify your stack and build anything. Or everything.
Build tomorrow’s web with a modern solution you truly own.
Code-based nature means you can build on top of it to power anything.
It’s time to take back your content infrastructure.

Issues viewing a large collection (Over 8500 documents) with production mongo

default discord avatar
davidko6042 years ago
14

I have a collection with over 8500 documents and I'm getting an error in the payload client when trying to view the list of documents from our production mongo that says:



MongoServerError: PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting.

I've tried setting

indexSortableFields

to true on the payload.config. Does anyone have any ideas on what can be done here? We are on an M5 cluster tier with MongoDB.

  • discord user avatar
    jmikrut
    2 years ago

    ok when you say

    payload client

    , do you mean the List view in the admin UI?



    we need to enable the aggregation property

    allowDiskUse

    I also assume that this is on a collection with versions enabled



    what is happening is that we need to sort a large set of documents, and you have too many which is causing memory issues



    i would suggest setting your versions / drafts config property

    maxPerDoc

    to a low number

  • default discord avatar
    davidko6042 years ago

    Yes, I meant the list view in the admin UI.



    And yes, this is also on a collection with versions enabled. We only have 1 version for each doc at the moment but I'll try lowering the amount in the config to see if that helps.



    Is the property of

    allowDiskUse

    something we can enable on our end or would this require a payload update?



    Also with some further testing if I set maxPerDoc to 1 it still has the issue.



    If I disable versions for this collection the documents do render in the list UI however.

  • discord user avatar
    jmikrut
    2 years ago

    allowDiskUse needs to be set by Payload but you could directly edit your

    node_modules/payload/dist

    to see if that will fix your issue (i am pretty confident it will)



    file to be changed is here:



    https://github.com/payloadcms/payload/blob/master/src/versions/drafts/queryDrafts.ts#L54

    and here is a screenshot of me editing the

    /dist

    version:





    clearly i am feeling like a 🤠 right now

  • default discord avatar
    davidko6042 years ago

    Hmm i've just tried editing the file and restarting my local server but still having the same issue:

  • discord user avatar
    jmikrut
    2 years ago

    hmm, that seems strange



    same exact error?



    https://www.mongodb.com/docs/atlas/reference/free-shared-limitations/?_ga=2.176267877.1329169847.1677683154-860992573.1647438381#operational-limitations

    looks like M5 clusters don't support allowDiskUse, which is very good to know



    maybe we can figure out a smarter way around this

  • default discord avatar
    davidko6042 years ago

    Oh



    ah i see:



    I'm going to try bumping to M10 in the interim but this interesting and also good to know!

  • discord user avatar
    jmikrut
    2 years ago

    ok - keep me posted



    i am not sure about how to get around this otherwise



    one other thing to note,

    allowDiskUse

    seems to be

    true

    by default on Mongo 6.0+



    so you might not need to hijack your

    node_modules

    like a maniac (me) if you are using 6.0+

  • default discord avatar
    davidko6042 years ago

    We're on Mongo 5.0 at the moment. Will confirm in a couple minutes.



    @364124941832159242


    Okay! I can confirm upgrading to an m10 cluster works!

  • discord user avatar
    jmikrut
    2 years ago

    beautiful - did you still have your node_modules fix?



    or should we set this in code? PS - would be a great PR for you to make if you're up for it 👼

  • default discord avatar
    davidko6042 years ago

    Just deleting the change to my node_modules...



    It actually seems to work just fine

  • discord user avatar
    jmikrut
    2 years ago

    ok awesome



    sucky that the database has gotta be more expensive - i will keep this on the radar to see if it's possible to do this in a better way

  • default discord avatar
    davidko6042 years ago

    Awesome thanks for the support! Yeah this will work fine for us in the interim but would be great for others if a similar issue didn't require the higher dedicated server.



    Sure I could. Would it make sense to hard code allowDiskUse in the query or have this as an optional parameter somewhere? I guess if the m10 cluster seems to solve it at mongo 5.0 though maybe not quite necessary yet?

  • discord user avatar
    jmikrut
    2 years ago

    hard code makes sense



    i think it's only leveraged if it's necessary



    automatically

  • default discord avatar
    davidko6042 years ago
    https://github.com/payloadcms/payload/pull/2216

    PR is here. Feel free to update wording in the note in docs.

Star on GitHub

Star

Chat on Discord

Discord

online

Can't find what you're looking for?

Get dedicated engineering support directly from the Payload team.