Re: [Kosmosfs-users] IllegalArgumentException
Status: Alpha
Brought to you by:
sriramsrao
Menu
▾
▴
Re: [Kosmosfs-users] IllegalArgumentException
|
From: Sriram R. <sri...@gm...> - 2008-11-20 19:32:45
|
Great. Thanks for the find. We'll get this code checked in (to the
KFS tree/hadoop tree).
Sriram
On Thu, Nov 20, 2008 at 9:46 AM, Michael <mic...@gm...> wrote:
> Hey,
> Your hint was helpful and it works now! The code was a little wrong
> because the new BlockLocation constructor needed blkStart not start, but
> that was it.
>
> for(int i=0; i < result.length; ++i) {
> result[i] = new BlockLocation(null, hints[i], blkStart,
> length < blockSize ? length :
> blockSize);
> blkStart += blockSize;
> length -= blockSize;
> }
> return result;
>
> I've attached a diff.
> Thanks for all the help!
>
> Sriram Rao wrote:
>> Wait...
>>
>>> for(int i=0; i < result.length; ++i) {
>>> result[i] = new BlockLocation(null, hints[i], start,
>>> length < BlockSize ? length : blockSize);
>>> System.out.println("Getting block: " + result[i]);
>>> length -= blockSize;
>>> }
>>
>> This looks buggy. start isn't getting updated inside the loop. The
>> loop is supposed to return the locaiton of every block in teh file; it
>> is only returning the value for the first one. What needs to change:
>>
>> long blkStart = start;
>>> for(int i=0; i < result.length; ++i) {
>>> result[i] = new BlockLocation(null, hints[i], start,
>>> length < BlockSize ? length : blockSize);
>>> System.out.println("Getting block: " + result[i]);
>> blkStart += blockSize;
>>> length -= blockSize;
>>> }
>>
>> Can you make this change and verify that it works? There are 10
>> blocks in the file and the code was only returning the location of the
>> first one.
>>
>> Sriram
>>
>> On Wed, Nov 19, 2008 at 8:09 PM, Michael <mic...@gm...> wrote:
>>> Ok so I added a print statement
>>> for(int i=0; i < result.length; ++i) {
>>> result[i] = new BlockLocation(null, hints[i], start,
>>> length < BlockSize ? length : blockSize);
>>> System.out.println("Getting block: " + result[i]);
>>> length -= blockSize;
>>> }
>>> And here was the output
>>>
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> Getting block: 0,67108864,208.100.69.22
>>> java.lang.IllegalArgumentException: Offset 67108864 is outside of file
>>> (0..67108863)
>>>
>>> Where the first number is offset, the second is length, and the last is
>>> host.
>>> It seems that it is giving the same chunk many times. Also, the length
>>> seems to be 1 too large?
>>>
>>> Thanks for helping so much. Do you see what is wrong?
>>>
>>>
>>> Sriram Rao wrote:
>>>> The code:
>>>>
>>>> src/core/org/apache/hadoop/fs/kfs/KosmosFileSystem.java
>>>>
>>>> The function:
>>>> public BlockLocation[] getFileBlockLocations(FileStatus file, long start,...)
>>>>
>>>>
>>>> Sriram
>>>>
>>>> On Wed, Nov 19, 2008 at 2:53 PM, Michael <mic...@gm...> wrote:
>>>>> In which directory is this function. I couldn't find it using grep. the
>>>>> input is a dump of wikipedia dd'ed to 1 gigabyte even. Thanks for the
>>>>> help thus far.
>>>>>
>>>>> Sriram Rao wrote:
>>>>>> Hi,
>>>>>>
>>>>>> From the stack trace and the input, we can't tell where the problem is.
>>>>>> - it sounds like, when you gave the example program ~9 chunks, (each
>>>>>> chunk is 64MB) it worked fine
>>>>>> - with 1GB it is failing.
>>>>>>
>>>>>> To debug:
>>>>>> - in hadoop-kfs code: there is a method getBlockLocations() which is
>>>>>> called by the MapReduce layer to figure out where the blocks of a file
>>>>>> are
>>>>>> - can you add prints in that method to see how many blocks the file
>>>>>> has and where they are
>>>>>> --- the error you are getting is that, KFS is returning you only
>>>>>> one block location for the file, but Hadoop thinks there are multiple
>>>>>> blocks; the error is that you are trying to read past EOF.
>>>>>>
>>>>>> Also, how did you generate the input for wordcount---as in the command
>>>>>> line args?
>>>>>>
>>>>>> Sriram
>>>>>>
>>>>>> On Wed, Nov 19, 2008 at 2:00 PM, Michael <mic...@gm...> wrote:
>>>>>>> Hey,
>>>>>>> Thanks for all the help. Hadoop and Kfs are working together now.
>>>>>>> However, I have a problem. When I run:
>>>>>>> bin/hadoop jar hadoop-0.18.2-examples.jar wordcount input output
>>>>>>> Which is a command that does a mapreduce job on the servers. I get an
>>>>>>> error. I don't know if it is a kfs problem or a hadoop problem. If the
>>>>>>> input is relatively small 500MB it'll work however, at 1GB it gives this
>>>>>>> error:
>>>>>>> 08/11/19 15:56:52 INFO mapred.FileInputFormat: Total input paths to
>>>>>>> process : 1
>>>>>>> 08/11/19 15:56:52 INFO mapred.FileInputFormat: Total input paths to
>>>>>>> process : 1
>>>>>>> java.lang.IllegalArgumentException: Offset 67108864 is outside of file
>>>>>>> (0..67108863)
>>>>>>> at
>>>>>>> org.apache.hadoop.mapred.FileInputFormat.getBlockIndex(FileInputFormat.java:336)
>>>>>>> at
>>>>>>> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:248)
>>>>>>> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742)
>>>>>>> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026)
>>>>>>> at org.apache.hadoop.examples.WordCount.run(WordCount.java:149)
>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>>> at org.apache.hadoop.examples.WordCount.main(WordCount.java:155)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at
>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>> at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>> at
>>>>>>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>>>>>>> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:141)
>>>>>>> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:54)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at
>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>> at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
>>>>>>> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>>>>>>>
>>>>>>> Thanks again!
>>>>>>> Michael
>>>>>>>
>>>>>>> -------------------------------------------------------------------------
>>>>>>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
>>>>>>> Build the coolest Linux based applications with Moblin SDK & win great prizes
>>>>>>> Grand prize is a trip for two to an Open Source event anywhere in the world
>>>>>>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>>>>>>> _______________________________________________
>>>>>>> Kosmosfs-users mailing list
>>>>>>> Kos...@li...
>>>>>>> https://lists.sourceforge.net/lists/listinfo/kosmosfs-users
>>>>>>>
>
> 323a324
>> long blkStart = start;
> 325c326
> < result[i] = new BlockLocation(null, hints[i], start,
> ---
>> result[i] = new BlockLocation(null, hints[i], blkStart,
> 326a328
>> blkStart += blockSize;
>
>
|