Payloadv2 Custom server : UTF-8 Encoding Issue

baotaohlast year

26

• 

  notchrlast year

  I'm not sure if something specific in Payload caused this however

  
  
  

  Some different threads on SO state this is likely a data issue

  
  
  https://stackoverflow.com/questions/75318626/bsonerror-invalid-utf-8-string-in-bson-document-mongodb
• 

  baotaohlast year

  I thought so too but when testing in development connected to prod db i can save and view all data and in prod i can view previously saved data but with the replacement character and when saving new data with sc(special characters) i get that error.

  
  

  here is the complete error message

  
  "BSONError: Invalid UTF-8 string in BSON document\n at new BSONError (/workspace/node_modules/bson/lib/error.js:23:28)\n at getValidatedString (/workspace/node_modules/bson/lib/parser/deserializer.js:657:27)\n at deserializeObject (/workspace/node_modules/bson/lib/parser/deserializer.js:173:21)\n at deserializeObject (/workspace/node_modules/bson/lib/parser/deserializer.js:233:25)\n at deserializeObject (/workspace/node_modules/bson/lib/parser/deserializer.js:257:21)\n at deserializeObject (/workspace/node_modules/bson/lib/parser/deserializer.js:233:25)\n at deserialize (/workspace/node_modules/bson/lib/parser/deserializer.js:61:12)\n at Object.deserialize (/workspace/node_modules/bson/lib/bson.js:163:43)\n at BinMsg.parse (/workspace/node_modules/mongodb/lib/cmap/commands.js:455:54)\n at Connection.onMessage (/workspace/node_modules/mongodb/lib/cmap/connection.js:206:21)"
• 

  notchrlast year

  Does your hostname, db user, etc, have any special chars?

• 

  baotaohlast year

  No.

• 

  notchrlast year

  hmm

  
  
  https://payloadcms.com/docs/database/mongodb
  
  

  These are the config options for Mongo in payload

  
  
  

  Maybe it requires an option here?

  
  
  

  Also, what is your nodejs version

  
  
  

  is it the same local vs prod

• 

  baotaohlast year

  Am not sure about that. i am using payloadcloud.

  
  
  

  I get that the eror suggests the data i corrupted, but the issue is how the data is getting corrupted. It gets corrupted when saving the document and when fetching the data the proper sc are replaced in production env

  
  
  

  acessing the db with mongodb displays the data correctly and if a new document is added in production the same error occurs in mongoddb compass.

  
  

  this suggests there is a step before inserting and after fetching that i believe is handled by payload that is causing this error.

• 

  denolfe

  last year

  From the error, this isn't something specific with the

  Payload Cloud

  hosting - but something with Payload the framework.

  
  
  

  We'd need to be able to recreate this issue locally in order to look into it. From the error, it seems like

  the actual data

  has some unsupported characters in it.

  
  
  

  Are there any known encoding issues with Payload v2.12.1 when handling special characters?
  

  
  

  No.

  
  
  

  Could the version downgrade or the changes in Next.js configuration be related to this issue?
  

  
  

  I would be surprised if this had anything to do w/ Next.js versioning. Would be worth it for you to figure out your build errors and verify.

  
  
  

  Are there any recommended settings or plugins for proper UTF-8 handling in Payload, especially when using a custom server?
  

  
  

  Nothing should be needed.

  
  
  

  Is there any specific configuration needed for PayloadCloud to handle UTF-8 correctly?
  

  
  

  None have been reported.

  
  
  

  Are you able to consistently recreate the error? What data are you trying to store at the time of the error? What collection, can you provide the config?

• 

  baotaohlast year

  Hi Eliot,

  
  

  Thanks for replying.

  
  

  Yes i can constantly recreate the error, it is not specific to any collection it is across the app anytime i save anything with special characters the data gets corrupted.

  
  
  
  
  

  We'd need to be able to recreate this issue locally in order to look into it. From the error, it seems like the actual data has some unsupported characters in it.
  

  
  

  I havent been able to reproduce this error locally only on the deployed application.

  
  
  

  here is a vid of what happens saving a doc with and without special characters in production.

  
  
  

  and acessing the data in via compass is not possible

  
  
  

  the same occurs if i edit an existing document that already has special characters when saved again it becomes corrupted.
• 

  notchrlast year

  Is the only place you see the error within Compass?

  
  
  

  Oh wow, I see now that you cant save the doc on prod

• 

  denolfe

  last year

  Hmm, very interesting

  
  
  

  What node.js version?

  
  
  

  Pretty sure it had to do w/ that

  
  
  

  just received this when attempting Node 22

• 

  baotaohlast year

  Yeah its node 22

  
  
  

  How do i use a older node version in payload cloud?

• 

  quitecustardlast year

  Did you figure anything out? I'm on 20.1 and getting this error

• 

  baotaohlast year

  No not yet.. going to try to use node 18 on payload cloud.

  
  
  

  Working with node 19

  
  
  

  i added this to my package json.

  
  
  

    "engines": {
      "node": ">=18.12.0 <20.0.0"
    },

  
  
  

  does anybody know the reason this was happening.

• 

  denolfe

  last year

  Nice. I was actually just about to suggest this 🙌

• 

  oldmanutvaralast year

  I'm gettin the same problem on self hosted payloadcms. The response encoding is changing depending on the depth of the request. On depth 0 I'm getting response with latin-1 encoding (headers are ok application/json charset utf-8) and on depth 1 I'm getting proper utf-8. Node version is 20. I'll try to revert the version to 18
• 

  baotaohlast year

  Yes the problem is in node version 20^

  
  

  i checked on a different project and experience the same issue.

  
  
  

  I dont know why it working when you change depth(freakish) but yes try node version < 20.

• 

  oldmanutvaralast year

  I got the BSON error if I try to save anything

  
  
  

  the error is confusing as f, I've spent last two days trying to isolate it

  
  
  

  I'll try with node 18 and report back

• 

  baotaohlast year

  Yea same. downgrading fixed it for me on both projects so highly likely this will fix your issue as well.

  
  

  going to investigate the node repo later when i have time.

• 

  oldmanutvaralast year

  yeah it looks like it is working with node 18

• 

  quitecustardlast year

  We got ours working on 21.7.0, the lts

  
  
  

  We went through and deleted all version bar the last to save (not the failed save with the special characters)

• 

  oldmanutvaralast year

  @202765542145130496

  node 21 or 20? 20 is the LTS version

• 

  quitecustardlast year

  We tried both, it's been left on the 21.7 one

• 

  oldmanutvaralast year

  ok, thanks

• 

  denolfe

  last year

  Looks like this was an issue with Node v22.7 specifically 😬

  

  
  
  https://github.com/nodejs/node/issues/54543
• 

  lzyaclast year

  yikes 😬

• 

  tsemachhlast year

  https://github.com/nodejs/node/releases/tag/v22.8.0

  was released yesterday , fix should be there