Issues viewing a large collection (Over 8500 documents) with production mongo
I have a collection with over 8500 documents and I'm getting an error in the payload client when trying to view the list of documents from our production mongo that says:
MongoServerError: PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting.I've tried setting
indexSortableFieldsto true on the payload.config. Does anyone have any ideas on what can be done here? We are on an M5 cluster tier with MongoDB.
ok when you say
payload client, do you mean the List view in the admin UI?
we need to enable the aggregation property
allowDiskUseI also assume that this is on a collection with versions enabled
what is happening is that we need to sort a large set of documents, and you have too many which is causing memory issues
i would suggest setting your versions / drafts config property
maxPerDocto a low number
Yes, I meant the list view in the admin UI.
And yes, this is also on a collection with versions enabled. We only have 1 version for each doc at the moment but I'll try lowering the amount in the config to see if that helps.
Is the property of
allowDiskUsesomething we can enable on our end or would this require a payload update?
Also with some further testing if I set maxPerDoc to 1 it still has the issue.
If I disable versions for this collection the documents do render in the list UI however.
allowDiskUse needs to be set by Payload but you could directly edit your
node_modules/payload/distto see if that will fix your issue (i am pretty confident it will)
file to be changed is here:
https://github.com/payloadcms/payload/blob/master/src/versions/drafts/queryDrafts.ts#L54and here is a screenshot of me editing the
/distversion:
clearly i am feeling like a 🤠right now
Hmm i've just tried editing the file and restarting my local server but still having the same issue:
hmm, that seems strange
same exact error?
https://www.mongodb.com/docs/atlas/reference/free-shared-limitations/?_ga=2.176267877.1329169847.1677683154-860992573.1647438381#operational-limitationslooks like M5 clusters don't support allowDiskUse, which is very good to know
maybe we can figure out a smarter way around this
Oh
ah i see:
I'm going to try bumping to M10 in the interim but this interesting and also good to know!
ok - keep me posted
i am not sure about how to get around this otherwise
one other thing to note,
allowDiskUseseems to be
trueby default on Mongo 6.0+
so you might not need to hijack your
node_moduleslike a maniac (me) if you are using 6.0+
We're on Mongo 5.0 at the moment. Will confirm in a couple minutes.
@jmikrut
Okay! I can confirm upgrading to an m10 cluster works!
beautiful - did you still have your node_modules fix?
or should we set this in code? PS - would be a great PR for you to make if you're up for it 👼
Just deleting the change to my node_modules...
It actually seems to work just fine
ok awesome
sucky that the database has gotta be more expensive - i will keep this on the radar to see if it's possible to do this in a better way
Awesome thanks for the support! Yeah this will work fine for us in the interim but would be great for others if a similar issue didn't require the higher dedicated server.
Sure I could. Would it make sense to hard code allowDiskUse in the query or have this as an optional parameter somewhere? I guess if the m10 cluster seems to solve it at mongo 5.0 though maybe not quite necessary yet?
hard code makes sense
i think it's only leveraged if it's necessary
automatically
- https://github.com/payloadcms/payload/pull/2216
PR is here. Feel free to update wording in the note in docs.
Continue the discussion in Discord
Star
Discord
online
Get help straight from the Payload team with an Enterprise License.