"Payload has transformed the way our clients manage content. It's an indispensable tool for any modern agency."
Get the resources you need to start building today
Microsoft chose Payload to tell the world about AI.
Search doesn't work for large collections
I have a large collection with around 100k documents. If I try to perform a search in the payload admin panel, nothing happens. The filters however seem to work perfectly fine.
Is this a performance issue because of the type of search that is performed?
If you open your network tab, can you provide some details about the request?
Hey Jarrod,
well after analyzing it a bit more, turns out the search is just super slow. It takes an average of 10 seconds for a query. In addition to that, old queries aren't cancelled so if I type in a longer search String that triggers 3-4 search requests, it takes 50 seconds.
It's probably because of the regex search, any way to increase the speed? I've already set the index to true.
Also, it just seems to search the main attribute, and ignores ID and other attributes?
I would imagine there are definitely improvements to be made here
This feedback is awesome. Thank you for investigating it further!
We can:
- further investigate the sloweness
- provide visual feedback when querying
- cancel old requests
You're welcome 🙂
Any suggestions how to sort out this performance issue asap?Search is a key thing for this project, if it takes that long I can't use payload.
I'd be happy to build a workaround and contribute it or turn it into a plugin, I just don't have any experience contributing to open source yet.
I am curious, did you set the index after booting up payload on your live db?
or did you have it set in the config, before ever booting up payload on live?
Good question, I probably added it at some point
Does that make a difference?
I don't think payload touches your indexes after initial creation of a field. Can you verify there is an index in the db for the field you are searching on?
(its size should be fairly large since you have 100k+ docs)
That should be it right? I'm generally confused when it comes to indexes and mongodb Atlas, I've had multiple issues with normal indexes on Atlas in the past. And I should have mentioned I'm using atlas haha
is name the field that is being searched on?
Yeah
Interesting, I can likely throw together a test and compare with your findings
Could it be the large text index that messes things up? I migrated the database with that index
I have not used an index like that larger one you have
Okay yeah I dropped it and it doesn't make any difference
Any way to implement atlas search? That would be pretty awesome
So locally, loading up 100_000 docs, searching on an indexed field takes about 1.2s (with no latency, obviously)
I do wonder about that other index you have
its quite large
Just a thought to input here... would it be worth using something like
https://www.npmjs.com/package/node-lucenein the Payload admin to power the searches there? Might be faster than querying Mongo.
That would be super dope, I always prefer independent solutions rather that relying on atlas search or sth. The package looks pretty abandoned though.
Edit: Maybe implementing sth like flexsearch would be an option?
https://github.com/nextapps-de/flexsearch
@281120856527077378I don't have tons of experience but if I've learned one thing, it's that the UX of an app is heavily determined by the quality of the search. So apart from performance issues, I think it should be discussed how to improve the search functionality in general. As of right now, it feels like it's built for small collections in a CMS where you just want faster access. As soon as you have an app with larger collections, where you need to search through whole docs, do fuzzy matching etc. Payload isn't quite optimal atm
Yeah, I picked lucene as it's a pretty widely known search indexing standard that uses a file-based index, but anything that could operate independently within Express/Payload would work for this.
If it's an abstract implementation it could open the door to allowing plugins to override the implementation to use 3rd party search services too, like Azure cognitive Search, ElasticSearch, Algolia, etc. but I don't think these should be dependencies to use Payload.
It could also open the door to being able to specify search endpoints for collections or even a global search endpoint on Payload's API further down the line.
So true, a global search endpoint would be crazy!
I've checked the forum for some discussions around search, and the current direction seems to be using regex for searching one or multiple fields (which is indeed possible), and as soon as you have more complex requirements integrate algolia etc.
I totally get the point, however I hate to rely on third party services for search, I'd really love to have a plugin for payload
It should be fairly simple to make a plugin that can add hooks to collections & globals to index them as they are added/updated.
The trickier parts would be:
- Verifying/rebuilding the index on a schedule
- Adding new API end points to Payload
Also...
https://github.com/payloadcms/plugin-searchnone of this touches on the in-admin searches though
Oh, I didn't know about that plugin!
If there's a "payload way" to tackle the search, that's obviously even better than a third party library. I wonder if it's the way to go though if the end goal is a search with fuzzy matching etc. Isn't that plugin just another collection that just contains the relevant data for the search?
Yeah, but might be a good one to look at and expand upon
Flexsearch is super sweet and would cover a lot of use-cases, it would be awesome if there was a plugin for it!
For those of us clustering, the in-memory index is unfortunately not an option, but I've used Meilisearch in the past with good results (
https://github.com/meilisearch/meilisearch). It's also a very cool open-source project with a really straightforward JS/TS API client (
https://github.com/meilisearch/meilisearch-js) and it is extremely simple to self-host and scale.
You're right about the in-admin searching and filtering; it would be fantastic if there were hooks to override this functionality. It would be great if it could be supported to have the search functionality return either a list of document ids or a list of complete documents, and to have a toggle for whether or not Payload needs to apply access control logic to the list or if it has already been considered.
I love this discussion! I think this all needs to be on radar. I've opened a discussion on github that makes Payload's built-in search more extensible for the UI and detailed some other ideas from this thread. Putting it on github adds visibility for enhancements that will eventually make it to our roadmap.
https://github.com/payloadcms/payload/discussions/1974We would also be interested in partnering with anyone willing to work on this in an open-source contributing capacity or financially through with an enterprise support agreement.
Star
Discord
online
Get dedicated engineering support directly from the Payload team.