Steel Bank Common Lisp

Sounds like a LOCALE issue.  SBCL -- and probably your (unix/linux?) 
filesystem -- seem to be using UTF-8, but the original filenames were
probably written with a different encoding.

In Lisp, you might have some luck with reading and writing as binary
streams, instead of as characters.  In linux, take a look at "iconv",
which may be able to help you.

        -- Don
_______________________________________________________________________________
Don Geddis                  http://don.geddis.org/               don@...

Tamas K Papp <tkpapp@...> wrote on Sat, 17 Mar 2012:
> Hi,
>
> I recently restored from an old backup and some accented filenames got
> mangled, eg in dired the letter í (latin small letter i with acute)
> shows up as \355.
>
> I thought I would write a small CL script that would traverse the
> directories and replace these characters using a lookup table.  But when
> I try
>
> (fad:list-directory "~/restored-backup/")
>
> I get errors like
>
> c-string decoding error (:external-format :UTF-8):
>   the octet sequence 3 cannot be decoded.
>    [Condition of type SB-INT:C-STRING-DECODING-ERROR]
>
> Restarts:
>  0: [RETRY] Retry SLIME interactive evaluation request.
>  1: [*ABORT] Return to SLIME's top level.
>  2: [TERMINATE-THREAD] Terminate this thread (#<THREAD "worker" RUNNING {100493BDA3}>)
>
> Backtrace:
>   0: (SB-INT:C-STRING-DECODING-ERROR :UTF-8 3)
>   1: (SB-IMPL::READ-FROM-C-STRING/UTF-8 #.(SB-SYS:INT-SAP #X0063EA6B) CHARACTER)
>   2: ((FLET SB-IMPL::ONE-ITER :IN SB-IMPL::CALL-WITH-NATIVE-DIRECTORY-ITERATOR))
>   3: ((FLET SB-IMPL::ITERATE :IN SB-IMPL::MAP-DIRECTORY) #<CLOSURE (FLET
>   SB-IMPL::ONE-ITER :IN SB-IMPL::CALL-WITH-NATIVE-DIRECTORY-ITERATOR)
>   {7FFFF65FDC5B}>)
>
> I wonder if there is a workaround that would allow me to read and
> represent these filenames in SBCL as a string, so that I could perform
> the renaming.
>
> Best,
>
> Tamas

Hi Don,

On Sat 17 Mar 2012 03:52:49 PM CET, Don Geddis wrote:

 > Sounds like a LOCALE issue.  SBCL -- and probably your (unix/linux?) 
 > filesystem -- seem to be using UTF-8, but the original filenames were
 > probably written with a different encoding.

Yes, I am using linux, and that's probably what happened.

 > In Lisp, you might have some luck with reading and writing as binary
 > streams, instead of as characters.  In linux, take a look at "iconv",
 > which may be able to help you.

Your suggestion helped me solve the issue: I used qmv (from renameutils)
to generate a list of filenames, which I read as a binary file in an
SBCL script that did the conversion.  Thanks!

Just curious: is it possible to read the path as some kind of binary
stream directly in SBCL?

Best,

Tamas

On Mar 17, 2012, at 5:33 AM, Tamas K Papp wrote:
> I recently restored from an old backup and some accented filenames got
> mangled, eg in dired the letter í (latin small letter i with acute)
> shows up as \355.
> 
> I thought I would write a small CL script that would traverse the
> directories and replace these characters using a lookup table.  But when
> I try
> 
> (fad:list-directory "~/restored-backup/")
> 
> I get errors like
> 
> c-string decoding error (:external-format :UTF-8):
>  the octet sequence 3 cannot be decoded.
>   [Condition of type SB-INT:C-STRING-DECODING-ERROR]

Clearly you should have specified :external-format :WTF-8
... oh wait, still a couple of weeks until April Fools'.

		Peace
			--Devon

PS: This is an oft-seen train-wreck!

Where are the char-encoding-dwim restarts?
Where are the char-binary-agnostic streams? 

Let's be liberal in what we accept
and conservative in what we send.

Devon Sean McCullough <SBCL-user@...> writes:

> PS: This is an oft-seen train-wreck!
>
> Where are the char-encoding-dwim restarts?
> Where are the char-binary-agnostic streams? 

A utf-8b external format would be a fairly sensible addition to our
external format support (along with all the other missing features, such
as the oft-requested crlf line termination).  Bivalent streams are
supported; see section 10.1 of the manual.  You can define your own
"DWIM" character encoding handlers for stream output, where passing a
suitable argument to USE-VALUE allows you to encode what you mean.

> Let's be liberal in what we accept and conservative in what we send.

This approach is directly against the principles documented in the
PRINCIPLES file of the SBCL source tree.  You're of course free to
advocate what you want, and people may even listen to you, but you might
want to consider your audience.

Best,

Christophe

On Mar 17, 2012, at 2:28 PM, Christophe Rhodes wrote:
>> PS: This is an oft-seen train-wreck!
>> 
>> Where are the char-encoding-dwim restarts?
>> Where are the char-binary-agnostic streams? 
> 
> A utf-8b external format would be a fairly sensible addition to our
> external format support (along with all the other missing features, such
> as the oft-requested crlf line termination).  Bivalent streams are
> supported; see section 10.1 of the manual.  You can define your own
> "DWIM" character encoding handlers for stream output, where passing a
> suitable argument to USE-VALUE allows you to encode what you mean.

Thanks, UTF-8B bivalence should be mighty easy to implement
and would be the sanest default in general.

>> Let's be liberal in what we accept and conservative in what we send.
> 
> This approach is directly against the principles documented in the
> PRINCIPLES file of the SBCL source tree.  You're of course free to
> advocate what you want, and people may even listen to you, but you might
> want to consider your audience.

Which survived the test of time, x.25/ISO or TCP/IP?
Shorp tools are brittle and prone to break.
Think of restarts as eye protection.
When the host OS hoses the users this often and
a restart fixes 99% of the lossage, it's just
good manners to provide it.  Failing that,
augment the error message.  As a last resort
make sure the cure appears high in search results
for the cryptic error message, if it must remain so.

		Peace
			--Devon

PS: I see now why I worked so long to stay with CMUCL but
I'm determined (holding nose) to give SBCL a fair shake
by trying it in at least one real commercial project.