#1568 Broken shift_jis (japanese) support after 2.5.7-pl1

[Lars B. Jensen]

After upgrading phpMyAdmin from version 2.5.7-pl1 to
any release after this, there is a problem setting and
displaying shift_jis encoded data properly.

I've tried to force shift_jis encoding by modifying the
source files of phpMyAdmin to no use - it seems to
manipulate the data so it shows wrongly in the browser.

Versions :
FreeBSD 5.3
mySQL 4.1.8
PHP 5.0.2

/ Lars

[Alexander M. Turek]

Logged In: YES
user_id=418833

Could you specify the problem a bit more? Could you tell me
how to reproduce it?

Which is the lastest phpMyAdmin version you tested with?

[Ryan Carsten Schmidt]

Logged In: YES
user_id=838923

Could it be that you have the automatic multibyte string override
support enabled in your php.ini? I think we've found that PMA
doesn't work in combination with that option.

[Lars B. Jensen]

Logged In: YES
user_id=583611

The scenario I have is a database server running mySQL
4.1.8 with mixed english and japanese content, where the
japanese content is shift-jis encoded.

Clicking eg "browse" in phpMyAdmin in any table, in 2.5.7 pl1
shows me correct Japanese characters and thus readable
content, whereas if I do the exactly same in 2.6.1 rc2 or
2.6.0-pl3, it comes out scrambled.

I've noticed, in the newer than 2.5.7 pl1 releases of
phpMyAdmin, that I on the frontpage doesnt get "shiftjis" as
encoding method, and even if I modify manually the page
making the browse output and manually type shift-jis, it still
comes out bad. I tried in the browser (IE) to force content
encoding to japanese:autoselect and japanese:shiftjis to no
help.

Upon downgrading to 2.5.7 without any edits, it works like a
charm.

Same machine, same PHP, same everything.

[Ryan Carsten Schmidt]

Logged In: YES
user_id=838923

Do I understand correctly that you have one field in which you
store English text in some records but Japanese text in others?
What is the field's charset? Is it latin1, utf8, sjis? I would think
that the fields should be defined as utf8, with data stored in
utf8, and then converted to and from shift-jis as needed (using
SET NAMES 'sjis'; in your application).

[Lars B. Jensen]

Logged In: YES
user_id=583611

Ryan,

The only application giving me a problem, is phpMyAdmin
whereas the application I use the data for has run flawless
for 5 years.

The charset for the given table is latin1. The application
was designed 5 years ago in a rush, not everything is well
thought of - but somehow phpMyAdmin as the only application
we use, as we upgraded it (and only that) to a newer
version, that it cannot display Japanese characters anymore.

mySQL's own query browser application - which is a great
tool otherwise - doesnt handle Japanese chars at all. So I'm
kinda stuck with a slightly out-of-date phpMyAdmin.

/ Lars

[Alexander M. Turek]

Logged In: YES
user_id=418833

So, you're storing shift_jis characters in a latin1 table?

[Ryan Carsten Schmidt]

Logged In: YES
user_id=838923

For proper MySQL 4.1 support, PMA 2.6.0 has much better
character set support than previous versions, and fixes many
bugs related to this in 2.5.7 and before. PMA 2.6.0 now properly
displays the characters according to their specified character set,
so if you have relied on PMA's previous misbehavior of not doing
that, then you're going to run into problems now.

I'm not very familiar with Shift-JIS. Can it also store English text?
If so, then you could convert this column to Shift-JIS, and
hopefully you would not need to change your application too much.
Instructions for converting the table are here:

http://dev.mysql.com/doc/mysql/en/Charset-conversion.html

Then PMA should show the data correctly. And then you will need
to update your application code. If at all possible, I recommend
converting everything to UTF-8 because I think it's nice to always
use a single character set that can represent any character in any
language. But this will require more changes to your application.

[Lars B. Jensen]

Logged In: YES
user_id=583611

ryan >> that is the next 6months project we're undertaking -
to rewrite the full database application to UTF8 for proper
charactersets. sjis tables can store roman characters too,
yes - it's part of the Japanese 4 written language systems.
The solution is bilingual using same tablesets

rabus >> I am afraid so - very old design mistake