LisaList2

General Category => Meta => Topic started by: rayarachelian on February 01, 2020, 11:30:56 am

Title: Unable to post due to database errors is a UTF issue
Post by: rayarachelian on February 01, 2020, 11:30:56 am
I've seen several reports of users unable to post (or reply) and they've gotten back database errors. I believe this is a UTF issue that seems to affect apple devices/browsers,  perhaps it's an incompatibility between SMF and postgres, and I'm not sure what the actual characters involved are.
If you can, please test to help figure out what's happening.
If you're unable to post, please email me at  my first name at symbol my last name dot com.
This is what postgres reports, I'm not sure what that UTF8 sequence is, googling doesn't seem to help find it. I did run across some postgres related stackover questions that might help resolve this, I'll look into that in the background.
Code: [Select]
Feb  1 09:36:41 postgres[999]: 2020-02-01 09:36:41.504 EST [9530] ERROR:  invalid byte sequence for encoding "UTF8": 0xf6 0xf7 0xf8 0x26
Feb  1 09:36:41 postgres[999]: 2020-02-01 09:36:41.505 EST [9530] ERROR:  invalid byte sequence for encoding "UTF8": 0xf6 0xf7 0xf8 0x26
Feb  1 09:37:24 postgres[999]: 2020-02-01 09:37:24.115 EST [9540] ERROR:  invalid byte sequence for encoding "UTF8": 0xf6 0xf7 0xf8 0x26
Feb  1 09:37:24 postgres[999]: 2020-02-01 09:37:24.116 EST [9540] ERROR:  invalid byte sequence for encoding "UTF8": 0xf6 0xf7 0xf8 0x26
Feb  1 09:37:44 postgres[999]: 2020-02-01 09:37:44.056 EST [9546] ERROR:  invalid byte sequence for encoding "UTF8": 0xf6 0xf7 0xf8 0x26
Feb  1 09:37:44 postgres[999]: 2020-02-01 09:37:44.057 EST [9546] ERROR:  invalid byte sequence for encoding "UTF8": 0xf6 0xf7 0xf8 0x26

An example of this is at this message:
https://lisalist2.com/index.php?topic=63.msg291#msg291

Tom tried to reply to that thread, and subsequentially emailed me the failed text, it is embedded in the email. Copying the text from the email into a reply of SMF triggered the db error. I was only able to get a working copy by stripping off all UTF8 by pasting the message into the vi editor, then running cat -v on that file, copying the output of cat -v and pasting here. I'm sure there are other ways to strip off UTF8 chars, but that's what I did to make it work. Perhaps a programmer's editor that has UTF8 disabled will help, but VSCode did not.

A bigger mystery is what does this sequence mean, and why do Apple products embed it? 0xf6 0xf7 0xf8 0x26
Title: Re: Unable to post due to database errors is a UTF issue
Post by: berskyboy on February 07, 2020, 09:30:51 pm
Testing, I was not able to reply to another post, I had tried on my iPhone.  Trying this on my iMac.
Title: Re: Unable to post due to database errors is a UTF issue
Post by: rayarachelian on February 08, 2020, 03:36:32 pm
yea, ipads and iphones seem to have trouble, I'll see if I can switch postgres over to utf8 on the next upgrade.