Fedora Commons

Hi Chris,

I was able to reproduce the problem using your file. =20
On the console (STDOUT), I got the following additional information:

   Failure reading [...] Invalid byte 2 of 4-byte UTF-8 sequence.

It looks like the actual character encoding used in your file isn't what =
it
says in the xml declaration (UTF-8).  You can either correct the =
declaration
to whatever the actual encoding is used before you ingest it, or change =
the
actual encoding of the characters to UTF-8. =20

Fedora will accept any character encoding on ingest (as long as the JRE
supports it), and then will normalize to UTF-8 internally.

- Chris

-----Original Message-----
From: fedora-users-bounces@... on behalf of Chris Tomlinson
Sent: Mon 3/28/2005 10:34 PM
To: fedora-users@...
Subject: [Fedora-users] odd ingestion error
=20
Hello,

In Fedora 2.0 on Redhat linux: Linux 2.4.21-27.0.2.ELsmp #1 SMP Wed Jan =
12
23:35:44 EST 2005 i686 i686 i386 GNU/Linux with java version "1.4.2_07",
we're getting the seemingly random occurrence of:


  <ingested file=3D"works/W18034A1.xml" fType=3D"O" =
targetPID=3D"tbrc:W18034A1" />
  <failed file=3D"works/W19711.xml" fType=3D"O">
    Failure reading
file:///usr/local/fedora-2.0/server/jakarta-tomcat-5.0.28/temp/fedora-ing=
est-
temp39696.xml
  </failed>
  <ingested file=3D"works/W16658A1.xml" fType=3D"O" =
targetPID=3D"tbrc:W16658A1" />


The objects seem rather unremarkable and are well-formed. They are =
generated
by the same tool at the other objects that seem to be ingested just =
fine.

Any ideas?

I've attached the tbrc:W19711 file in case something is amiss there.

Thanks for your help,
Chris

Chris,

> a previous effort failed du to out of memory.

To clarify: This was on the client, right?  Your workaround sounds fine =
for
now but I think we can reduce memory utilization for the next release, =
too...
it shouldn't require that much.

- Chris


-----Original Message-----
From: fedora-users-bounces@... on behalf of Chris Tomlinson
Sent: Mon 3/28/2005 11:08 PM
To: Neb Radivojevic; fedora-users@...
Subject: Re: [Fedora-users] odd ingestion error
=20
Dear Neb,

I'm using "fedora-ingest d .....". There are no other users of Fedora =
during
the ingestion. It appears to be "random" in that there doesn't appear to =
be a
pattern such as ever 10th object or anything like that. There are a =
total of
49,888 objects in the directory and this example was taken after about =
18,000
objects had been ingested. I've also set the memory on the jvm via =
-Xmx1000m
since a previous effort failed du to out of memory.

Thanks,
Chris


>hello Chris
>
>did you try to insert this using single object or using batch
>insert? is it any object or just randomly? what is frequency of
>this problem? as anyone else using fedora / to access a data?
>
>regards,
>
>-neb
>
>
>
>
>--- Chris Tomlinson <ct@...> wrote:
>> Hello,
>>=20
>> In Fedora 2.0 on Redhat linux: Linux 2.4.21-27.0.2.ELsmp #1
>> SMP Wed Jan 12 23:35:44 EST 2005 i686 i686 i386 GNU/Linux with
>> java version "1.4.2_07", we're getting the seemingly random
>> occurrence of:
>>=20
>>=20
>>   <ingested file=3D"works/W18034A1.xml" fType=3D"O"
>> targetPID=3D"tbrc:W18034A1" />
>>   <failed file=3D"works/W19711.xml" fType=3D"O">
>>     Failure reading
>>
>file:///usr/local/fedora-2.0/server/jakarta-tomcat-5.0.28/temp/fedora-in=
gest
-temp39696.xml
>>   </failed>
>>   <ingested file=3D"works/W16658A1.xml" fType=3D"O"
>> targetPID=3D"tbrc:W16658A1" />
>>=20
>>=20
>> The objects seem rather unremarkable and are well-formed. They
>> are generated by the same tool at the other objects that seem
>> to be ingested just fine.
>>=20
>> Any ideas?
>>=20
>> I've attached the tbrc:W19711 file in case something is amiss
>> there.
>>=20
>> Thanks for your help,
>> Chris> _______________________________________________
>> Fedora-users mailing list
>> Fedora-users@...
>> http://comm.nsdl.org/mailman/listinfo/fedora-users
>>=20

_______________________________________________
Fedora-users mailing list
Fedora-users@...
http://comm.nsdl.org/mailman/listinfo/fedora-users

Dear Chris,

Thanks for the help. I've just discovered the problem also. I thought that =
all the original source data was ascii but there is this single character t=
hat is causing the problem.

Thanks,
Chris

>Hi Chris,
>
>I was able to reproduce the problem using your file. =20
>On the console (STDOUT), I got the following additional information:
>
>   Failure reading [...] Invalid byte 2 of 4-byte UTF-8 sequence.
>
>It looks like the actual character encoding used in your file isn't what i=
t
>says in the xml declaration (UTF-8).  You can either correct the declarati=
on
>to whatever the actual encoding is used before you ingest it, or change th=
e
>actual encoding of the characters to UTF-8. =20
>
>Fedora will accept any character encoding on ingest (as long as the JRE
>supports it), and then will normalize to UTF-8 internally.
>
>- Chris
>
>-----Original Message-----
>From: fedora-users-bounces@... on behalf of Chris Tomlinson
>Sent: Mon 3/28/2005 10:34 PM
>To: fedora-users@...
>Subject: [Fedora-users] odd ingestion error
>=20
>Hello,
>
>In Fedora 2.0 on Redhat linux: Linux 2.4.21-27.0.2.ELsmp #1 SMP Wed Jan 12
>23:35:44 EST 2005 i686 i686 i386 GNU/Linux with java version "1.4.2_07",
>we're getting the seemingly random occurrence of:
>
>
>  <ingested file=3D"works/W18034A1.xml" fType=3D"O" targetPID=3D"tbrc:W180=
34A1" />
>  <failed file=3D"works/W19711.xml" fType=3D"O">
>    Failure reading
>file:///usr/local/fedora-2.0/server/jakarta-tomcat-5.0.28/temp/fedora-inge=
st-
>temp39696.xml
>  </failed>
>  <ingested file=3D"works/W16658A1.xml" fType=3D"O" targetPID=3D"tbrc:W166=
58A1" />
>
>
>The objects seem rather unremarkable and are well-formed. They are generat=
ed
>by the same tool at the other objects that seem to be ingested just fine.
>
>Any ideas?
>
>I've attached the tbrc:W19711 file in case something is amiss there.
>
>Thanks for your help,
>Chris
>

Dear Chris,

No the out of memory was on the server:

=C2=A0 <failed file=3D"works/W14359.xml" fType=3D"O">
=C2=A0=C2=A0=C2=A0 java.lang.reflect.InvocationTargetException
=C2=A0 </failed>
=C2=A0 <ingested file=3D"works/W25.xml" fType=3D"O" targetPID=3D"tbrc:W25" =
/>
=C2=A0 <ingested file=3D"works/W14073.xml" fType=3D"O" targetPID=3D"tbrc:W1=
4073" />
=C2=A0 <ingested file=3D"works/W3226.xml" fType=3D"O" targetPID=3D"tbrc:W32=
26" />
=C2=A0 <ingested file=3D"works/W9478A1.xml" fType=3D"O" targetPID=3D"tbrc:W=
9478A1" />
=C2=A0 <ingested file=3D"works/W1436A1.xml" fType=3D"O" targetPID=3D"tbrc:W=
1436A1" />
=C2=A0 <ingested file=3D"works/W22620A1.xml" fType=3D"O" targetPID=3D"tbrc:=
W22620A1" />
=C2=A0 <failed file=3D"works/W13157.xml" fType=3D"O">
=C2=A0=C2=A0=C2=A0 java.lang.reflect.InvocationTargetException
=C2=A0 </failed>
=C2=A0 <ingested file=3D"works/W6730.xml" fType=3D"O" targetPID=3D"tbrc:W67=
30" />
=C2=A0 <ingested file=3D"works/W9739.xml" fType=3D"O" targetPID=3D"tbrc:W97=
39" />
=C2=A0 <ingested file=3D"works/W19975A2.xml" fType=3D"O" targetPID=3D"tbrc:=
W19975A2" />
=C2=A0 <ingested file=3D"works/W27473.xml" fType=3D"O" targetPID=3D"tbrc:W2=
7473" />
=C2=A0 <ingested file=3D"works/W2607.xml" fType=3D"O" targetPID=3D"tbrc:W26=
07" />
=C2=A0 <ingested file=3D"works/W10419A2.xml" fType=3D"O" targetPID=3D"tbrc:=
W10419A2" />
=C2=A0 <failed file=3D"works/W27225.xml" fType=3D"O">
=C2=A0=C2=A0=C2=A0 java.lang.reflect.InvocationTargetException
=C2=A0 </failed>
=C2=A0 <ingested file=3D"works/W16170.xml" fType=3D"O" targetPID=3D"tbrc:W1=
6170" />
=C2=A0 <ingested file=3D"works/W23908A1.xml" fType=3D"O" targetPID=3D"tbrc:=
W23908A1" />
=C2=A0 <ingested file=3D"works/W24605.xml" fType=3D"O" targetPID=3D"tbrc:W2=
4605" />
=C2=A0 <ingested file=3D"works/W2880A1.xml" fType=3D"O" targetPID=3D"tbrc:W=
2880A1" />
=C2=A0 <ingested file=3D"works/W15854.xml" fType=3D"O" targetPID=3D"tbrc:W1=
5854" />
=C2=A0 <ingested file=3D"works/W11908.xml" fType=3D"O" targetPID=3D"tbrc:W1=
1908" />
=C2=A0 <ingested file=3D"works/W3766.xml" fType=3D"O" targetPID=3D"tbrc:W37=
66" />
=C2=A0 <ingested file=3D"works/W6767.xml" fType=3D"O" targetPID=3D"tbrc:W67=
67" />
=C2=A0 <ingested file=3D"works/W15406.xml" fType=3D"O" targetPID=3D"tbrc:W1=
5406" />
=C2=A0 <ingested file=3D"works/W5061.xml" fType=3D"O" targetPID=3D"tbrc:W50=
61" />
=C2=A0 <ingested file=3D"works/W15326.xml" fType=3D"O" targetPID=3D"tbrc:W1=
5326" />
=C2=A0 <ingested file=3D"works/W2377.xml" fType=3D"O" targetPID=3D"tbrc:W23=
77" />
=C2=A0 <ingested file=3D"works/W28427.xml" fType=3D"O" targetPID=3D"tbrc:W2=
8427" />
=C2=A0 <ingested file=3D"works/W27234.xml" fType=3D"O" targetPID=3D"tbrc:W2=
7234" />
=C2=A0 <failed file=3D"works/W13697A1.xml" fType=3D"O">
=C2=A0=C2=A0=C2=A0 java.lang.reflect.InvocationTargetException
=C2=A0 </failed>
=C2=A0 <ingested file=3D"works/W16123A1.xml" fType=3D"O" targetPID=3D"tbrc:=
W16123A1" />
=C2=A0 <failed file=3D"works/W1150.xml" fType=3D"O">
=C2=A0=C2=A0=C2=A0 (500)Internal Server Error
=C2=A0 </failed>
=C2=A0 <failed file=3D"works/W2567.xml" fType=3D"O">
=C2=A0=C2=A0=C2=A0 Replicator returned error: (java.lang.OutOfMemoryError) =
- null
=C2=A0 </failed>
=C2=A0 <failed file=3D"works/W10054.xml" fType=3D"O">
=C2=A0=C2=A0=C2=A0 java.lang.reflect.InvocationTargetException
=C2=A0 </failed>
=C2=A0 <failed file=3D"works/W27190.xml" fType=3D"O">
=C2=A0=C2=A0=C2=A0 Replicator returned error: (java.lang.OutOfMemoryError) =
- null
=C2=A0 </failed>
=C2=A0 <ingested file=3D"works/W3343.xml" fType=3D"O" targetPID=3D"tbrc:W33=
43" />
=C2=A0 <failed file=3D"works/W2673.xml" fType=3D"O">
=C2=A0=C2=A0=C2=A0=20
=C2=A0 </failed>
=C2=A0 <ingested file=3D"works/W3893.xml" fType=3D"O" targetPID=3D"tbrc:W38=
93" />
=C2=A0 <failed file=3D"works/W12015.xml" fType=3D"O">
=C2=A0=C2=A0=C2=A0=20
=C2=A0 </failed>
=C2=A0 <failed file=3D"works/W19509A1.xml" fType=3D"O">
=C2=A0=C2=A0=C2=A0 java.lang.reflect.InvocationTargetException
=C2=A0 </failed>
=C2=A0 <failed file=3D"works/W3721.xml" fType=3D"O">
=C2=A0=C2=A0=C2=A0 Replicator returned error: (java.lang.OutOfMemoryError) =
- null
=C2=A0 </failed>


This led me to up the memory from -Xmx128m to -Xmx1000m.

ciao,
Chris

>Chris,
>
>> a previous effort failed du to out of memory.
>
>To clarify: This was on the client, right?  Your workaround sounds fine fo=
r
>now but I think we can reduce memory utilization for the next release, too=
=2E..
>it shouldn't require that much.
>
>- Chris
>
>
>-----Original Message-----
>From: fedora-users-bounces@... on behalf of Chris Tomlinson
>Sent: Mon 3/28/2005 11:08 PM
>To: Neb Radivojevic; fedora-users@...
>Subject: Re: [Fedora-users] odd ingestion error
>=20
>Dear Neb,
>
>I'm using "fedora-ingest d .....". There are no other users of Fedora duri=
ng
>the ingestion. It appears to be "random" in that there doesn't appear to b=
e a
>pattern such as ever 10th object or anything like that. There are a total =
of
>49,888 objects in the directory and this example was taken after about 18,=
000
>objects had been ingested. I've also set the memory on the jvm via -Xmx100=
0m
>since a previous effort failed du to out of memory.
>
>Thanks,
>Chris
>
>
>>hello Chris
>>
>>did you try to insert this using single object or using batch
>>insert? is it any object or just randomly? what is frequency of
>>this problem? as anyone else using fedora / to access a data?
>>
>>regards,
>>
>>-neb
>>
>>
>>
>>
>>--- Chris Tomlinson <ct@...> wrote:
>>> Hello,
>>>=20
>>> In Fedora 2.0 on Redhat linux: Linux 2.4.21-27.0.2.ELsmp #1
>>> SMP Wed Jan 12 23:35:44 EST 2005 i686 i686 i386 GNU/Linux with
>>> java version "1.4.2_07", we're getting the seemingly random
>>> occurrence of:
>>>=20
>>>=20
>>>   <ingested file=3D"works/W18034A1.xml" fType=3D"O"
>>> targetPID=3D"tbrc:W18034A1" />
>>>   <failed file=3D"works/W19711.xml" fType=3D"O">
>>>     Failure reading
>>>
>>file:///usr/local/fedora-2.0/server/jakarta-tomcat-5.0.28/temp/fedora-ing=
est
>-temp39696.xml
>>>   </failed>
>>>   <ingested file=3D"works/W16658A1.xml" fType=3D"O"
>>> targetPID=3D"tbrc:W16658A1" />
>>>=20
>>>=20
>>> The objects seem rather unremarkable and are well-formed. They
>>> are generated by the same tool at the other objects that seem
>>> to be ingested just fine.
>>>=20
>>> Any ideas?
>>>=20
>>> I've attached the tbrc:W19711 file in case something is amiss
>>> there.
>>>=20
>>> Thanks for your help,
>>> Chris> _______________________________________________
>>> Fedora-users mailing list
>>> Fedora-users@...
>>> http://comm.nsdl.org/mailman/listinfo/fedora-users
>>>=20
>
>_______________________________________________
>Fedora-users mailing list
>Fedora-users@...
>http://comm.nsdl.org/mailman/listinfo/fedora-users
>