What are the consequences of using autoCreate and autoIndex?

hdodov12 months ago

5 3

mongodb://USER:PASS@host.docker.internal:27017/payload
?tls=true
&tlsCAFile=eu-central-1-bundle.pem
&tlsInsecure=true
&directConnection=true
&retryWrites=false
+ &readPreference=secondaryPreferred

mongodb://USER:PASS@host.docker.internal:27017/payload
?tls=true
&tlsCAFile=eu-central-1-bundle.pem
&tlsInsecure=true
&directConnection=true
&readPreference=secondaryPreferred
&retryWrites=false
+ &autoCreate=true
+ &autoIndex=true

export default buildConfig({
	// ...
	db: mongooseAdapter({
		// Fixes https://github.com/payloadcms/payload/issues/3947
		disableIndexHints: true,
		connectOptions: {
			useFacet: true,
			autoCreate: false,
			autoIndex: false,
		},
		url: process.env.MONGODB_URI,
	}),
	// ...
});

mongodb://USER:PASS@host.docker.internal:27017/payload
?tls=true
&tlsCAFile=eu-central-1-bundle.pem
&tlsInsecure=true
&directConnection=true
&readPreference=secondaryPreferred
&retryWrites=false
- &autoCreate=true
- &autoIndex=true

• Selected Answer
  

  DanRibbens

  12 months ago

  Hey @hdodov,

  I appreciate the detailed explanation on the steps you took and problems you were facing. I'll share my knowledge and hopefully shed some light.

  Add the readPreference option to the connection string

  I haven't tested this but I think there is a difference in how documentDB and mongoDB replicasets are handling traffic. I agree with you on setting the readPreference. You could also experiment with the writeConcern options. I would think you could further distribute the load and remove the disabling of autoCreate/autoIndex though I have not confirmed this.

  connectOptions: {
        writeConcern: {
          w: 'majority',
        }
      },
  

  Did you experiment with that?

  Disable autoCreate and autoIndex in Payload

  If you have autoIndex: false, Payload, calling Mongoose, will not create the indexes that Payload normally makes for you, including the fields you have set in your config. When using this setting you will want to create your indexes inside migrations that can be run once, rather than anytime Payload starts. This is actually a better pattern than the default behavior of having autoIndex: true on in production, but it introduces more complexity in your deployments.

  I have a bigger concern with autoCreate: false, this is the same thing as autoIndex except for adding new collections. You will be able to add new fields to your config, as the mongoose model is not enforced at the ORM level, but Payload itself. Where you will have trouble is if you add a new collection, enable versions or add a plugin that needs a new collection. In these scenarios Payload won't be able to create the new collection since autoCreate is disabled. This can also be solved with migrations, but you'll need to connect and create these collections using Mongoose with new models in a manual way with `payload.db.collections.. It isn't ideal as Payload normally takes care of all that for you.

  You could make a separate DB connection with whatever settings are needed from your migrations if you do continue on this path.

  Does this makes sense? Let us know what you find!
• 

  hdodov12 months ago

  I decided to go with this on our staging server and hit the following error:

  13:26:28 0|server  | MongoTransactionError: Read preference in a transaction must be primary, not: secondaryPreferred
  13:26:28 0|server  |     at executeOperationAsync (/app/node_modules/mongodb/lib/operations/execute_operation.js:74:15)
  13:26:28 0|server  |     at /app/node_modules/mongodb/lib/operations/execute_operation.js:12:45
  13:26:28 0|server  |     at maybeCallback (/app/node_modules/mongodb/lib/utils.js:337:21)
  13:26:28 0|server  |     at executeOperation (/app/node_modules/mongodb/lib/operations/execute_operation.js:12:38)
  13:26:28 0|server  |     at FindCursor._initialize (/app/node_modules/mongodb/lib/cursor/find_cursor.js:54:50)
  13:26:28 0|server  |     at [kInit] (/app/node_modules/mongodb/lib/cursor/abstract_cursor.js:444:14)
  13:26:28 0|server  |     at next (/app/node_modules/mongodb/lib/cursor/abstract_cursor.js:512:22)
  13:26:28 0|server  |     at node:internal/util:441:7
  13:26:28 0|server  |     at new Promise (<anonymous>)
  13:26:28 0|server  |     at next (node:internal/util:427:12) {
  13:26:28 0|server  |   [Symbol(errorLabels)]: Set(0) {}
  13:26:28 0|server  | }
  

  So apparently, Payload can't work with replica instances. Is that right? @DanRibbens could you confirm?

• 

  vibhor-web-headout8 months ago

  I'm trying to get a similar read-replica setup done on our end but I am running into a similar problem.

  You could also experiment with the writeConcern options.

  @DanRibbens, not sure if the above works. I believe writeConcern is associated with the write operation's acknowledgments rather than spreading read queries to replicas.

  you'll need to connect and create these collections using Mongoose with new models in a manual way with `payload.db.collections...

  Unfortunately, this seems prone to error with updates and is not scalable.
  From what I have read, one way is to create two connections for Primary and secondary(s) and pass different configs to them. But this would require payload db packages to internally manage these connections for write/update queries.

  Is there any other workaround for achieving this?