ELASTIC MAPREDUCE extraction process

2014-01-14
2014-08-05
  • Ilija Subasic
    Ilija Subasic
    2014-01-14

    Hi,
    Has anyone tried and succeeded in running the extraction process in elastic mapreduce? So far I have no luck in doing so, after some editing of DumpExtractor.java and PageSummary.java I ended up with an error to do with Avro and Reflection. Do not really have any ideas how to go about fixing it.
    Thanks,
    Ilija

    stderr output:

    java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.avro.reflect.ReflectData.<init>(java.lang.ClassLoader)
    at org.apache.avro.mapred.AvroJob.newDataModelInstance(AvroJob.java:219)
    at org.apache.avro.mapred.AvroJob.createDataModel(AvroJob.java:226)
    at org.apache.avro.mapred.AvroSerialization.getSerializer(AvroSerialization.java:106)
    at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:979)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:433)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:377)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)
    Caused by: java.lang.NoSuchMethodException: org.apache.avro.reflect.ReflectData.<init>(java.lang.ClassLoader)
    at java.lang.Class.getConstructor0(Class.java:2706)
    at java.lang.Class.getDeclaredConstructor(Class.java:1985)
    at org.apache.avro.mapred.AvroJob.newDataModelInstance(AvroJob.java:215)
    ... 11 more

     
    • ravimannan2002
      ravimannan2002
      2014-07-31

      Hi,
      Did you ever figure this out? I'm having the same problem. Seems to me like it's a problem with mismatched jars. ReflectData is trying to call a no-arg constructor. This should've been caught at compile time, but you and I running with different jars than we're compiling with.

       
  • ravimannan2002
    ravimannan2002
    2014-08-05

    oh it look like i had a diff version of avro at run time. it was from pig 0.10.0