hadoop-studio-users Mailing List for Hadoop Studio
Brought to you by:
arren
Menu
▾
▴
hadoop-studio-users — Discussion list for Hadoop Studio users
You can subscribe to this list here.
| 2009 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
(1) |
Jul
(5) |
Aug
(11) |
Sep
(6) |
Oct
(29) |
Nov
(34) |
Dec
(25) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2010 |
Jan
(8) |
Feb
(8) |
Mar
(4) |
Apr
(4) |
May
(17) |
Jun
(38) |
Jul
(47) |
Aug
(30) |
Sep
(2) |
Oct
(11) |
Nov
(6) |
Dec
(6) |
| 2011 |
Jan
(13) |
Feb
(8) |
Mar
(7) |
Apr
(16) |
May
(7) |
Jun
(11) |
Jul
(23) |
Aug
(11) |
Sep
(7) |
Oct
|
Nov
(15) |
Dec
(4) |
| 2012 |
Jan
(1) |
Feb
(7) |
Mar
|
Apr
(3) |
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
| 2013 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
(1) |
Oct
|
Nov
|
Dec
|
| 2014 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
(1) |
Sep
|
Oct
|
Nov
|
Dec
|
| 2015 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(1) |
Nov
|
Dec
|
|
From: Habib K. <hab...@gm...> - 2015-10-21 01:28:14
|
bonjour Messieurs, j'ai besoin a kamasphere studio plugin for netbeans svp, Merci |
|
From: Gilsinia L. <lop...@gm...> - 2012-04-18 03:11:05
|
Hi,
I am using the folllowing code to do the reduce side join
/*
* HadoopMapper.java
*
* Created on Apr 8, 2012, 5:39:51 PM
*/
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
// import org.apache.commons.logging.Log;
// import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.contrib.utils.join.*;
/**
*
* @author
*/
public class DataJoin extends Configured implements Tool
{
public static class MapClass extends DataJoinMapperBase
{
protected Text generateInputTag(String inputFile)
{
String datasource = inputFile.split("-")[0];
return new Text(datasource);
}
protected Text generateGroupKey(TaggedMapOutput aRecord)
{
String line = ((Text) aRecord.getData()).toString();
String[] tokens = line.split(",");
String groupKey = tokens[0];
return new Text(groupKey);
}
protected TaggedMapOutput generateTaggedMapOutput(Object
value)
{
TaggedWritable retv = new TaggedWritable((Text)
value);
retv.setTag(this.inputTag);
return retv;
}
}
public static class Reduce extends DataJoinReducerBase
{
protected TaggedMapOutput combine(Object[] tags,
Object[] values)
{
if (tags.length < 2) return null;
String joinedStr = "";
for (int i=0; i<values.length; i++)
{
if (i > 0) joinedStr += ",";
TaggedWritable tw = (TaggedWritable)
values[i];
String line = ((Text)
tw.getData()).toString();
String[] tokens = line.split(",", 2);
joinedStr += tokens[1];
}
TaggedWritable retv = new TaggedWritable(new
Text(joinedStr));
retv.setTag((Text) tags[0]);
return retv;
}
}
public static class TaggedWritable extends TaggedMapOutput
{
private Writable data;
public TaggedWritable(Writable data)
{
this.tag = new Text("");
this.data = data;
}
public Writable getData()
{
return data;
}
public void write(DataOutput out) throws IOException
{
this.tag.write(out);
this.data.write(out);
}
public void readFields(DataInput in) throws IOException
{
this.tag.readFields(in);
this.data.readFields(in);
}
}
public int run(String[] args) throws Exception
{
Configuration conf = getConf();
JobConf job = new JobConf(conf, DataJoin.class);
String[] otherArgs = new
GenericOptionsParser(conf, args).getRemainingArgs();
if (otherArgs.length != 2)
{
System.err.println("Usage: wordcount
<in> <out>");
System.exit(2);
}
Path in = new Path(args[0]);
Path out = new Path(args[1]);
FileInputFormat.setInputPaths(job, in);
FileOutputFormat.setOutputPath(job, out);
job.setJobName("DataJoin");
job.setMapperClass(MapClass.class);
job.setReducerClass(Reduce.class);
job.setInputFormat(TextInputFormat.class);
job.setOutputFormat(TextOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(TaggedWritable.class);
job.set("mapred.textoutputformat.separator", ",");
JobClient.runJob(job);
return 0;
}
public static void main(String[] args) throws Exception
{
int res = ToolRunner.run(new Configuration(),
new DataJoin(),
args);
System.exit(res);
}
}
I am able to compile my code. When I run in hadoop I am getting the
following error with the combiner
12/04/17 19:59:29 INFO mapred.JobClient: map 100% reduce 27%
12/04/17 19:59:38 INFO mapred.JobClient: map 100% reduce 30%
12/04/17 19:59:47 INFO mapred.JobClient: map 100% reduce 33%
12/04/17 20:00:23 INFO mapred.JobClient: Task Id :
attempt_201204061316_0018_r_000000_2, Status : FAILED
java.lang.RuntimeException: java.lang.NoSuchMethodException:
DataJoin$TaggedWritable.<init>()
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:62)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
at
org.apache.hadoop.mapred.Task$ValuesIterator.readNextValue(Task.java:1136)
at
org.apache.hadoop.mapred.Task$ValuesIterator.next(Task.java:1076)
at
org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.moveToNext(ReduceTask.java:246)
at
org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:242)
at
org.apache.hadoop.contrib.utils.join.DataJoinReducerBase.regroup(DataJoinReducerBase.java:106)
I checked some forums and found out that the error may occur due to non
static class. My program has no non static class!
The command I use to run hadoop is
/hadoop/core/bin/hadoop jar /export/scratch/lopez/Join/DataJoin.jar
DataJoin /export/scratch/user/lopez/Join
/export/scratch/user/lopez/Join_Output
and the DataJoin.jar file has DataJoin$TaggedWritable packaged in it
Could someone please help me.
Thank you !
--
Regards,
Lopez Gilsinia Gilroy
I alone know the plans I have for you, plans to bring you prosperity and
not disaster, plans to bring about the future you hope for.Jeremiah 29:11
--
Regards,
Lopez Gilsinia Gilroy
I alone know the plans I have for you, plans to bring you prosperity and
not disaster, plans to bring about the future you hope for.Jeremiah 29:11
|
|
From: Ted R. <tre...@ka...> - 2012-04-04 20:28:57
|
Alicia, It looks like you have a mismatch of the hadoop version on your cluster and that in the connection wizard. As the hadoop version on your cluster is cdh33u0 you will need to select the cdh3 item in the hadoop version drop down menu. I hope that this helps, if you have more questions/problems please don't hesitate to ask. Ted. Ted Reynolds Technical Support Engineer *The Leader in Big Data Analytics for Hadoop* P: (650)292-6113 19200 Stevens Creek Blvd. Suite 130, Cupertino, CA 95014 On Wed, Apr 4, 2012 at 11:14 AM, Alicia Dsouza <dso...@um...> wrote: > Hello > > We have a Hadoop system set up and working in the laboratory, and are able > to access the nodes over VPN/puTTY and run a basic WordCount program > successfully. > Our issue is that we are unable to our Remote HDFS via the Karmasphere > Eclipse plugin. > > The error message is : 'Connection to > blah.institution.edu/xxx.xxx.xx.xxx:9000 failed: > java.net.SocketTimeoutException' > > The server settings in the connection wizard are as follows- > *NameNode Host*: blah.institution.edu > *NameNode Port*: 9000 > *Hadoop Version:* 0.20.2 > *Username:* blahuser > *Group:* blahusergroup > > *** Note: Verified that our live Hadoop version is *hadoop-0.20.2-cdh3u0 * and > the fs.default.name configuration property value is* hdfs:// > blah.institution.edu:9000 **** > * > * > Followed all installation instructions in the Karmapshere guide. Is there > something else that we are missing? Any hints or pointer will be useful. > > Thanks in advance! > > Alicia > > > ------------------------------------------------------------------------------ > Better than sec? Nothing is better than sec when it comes to > monitoring Big Data applications. Try Boundary one-second > resolution app monitoring today. Free. > http://p.sf.net/sfu/Boundary-dev2dev > _______________________________________________ > Hadoop-studio-users mailing list > Had...@li... > https://lists.sourceforge.net/lists/listinfo/hadoop-studio-users > > |
|
From: Alicia D. <dso...@um...> - 2012-04-04 18:14:57
|
Hello We have a Hadoop system set up and working in the laboratory, and are able to access the nodes over VPN/puTTY and run a basic WordCount program successfully. Our issue is that we are unable to our Remote HDFS via the Karmasphere Eclipse plugin. The error message is : 'Connection to blah.institution.edu/xxx.xxx.xx.xxx:9000 failed: java.net.SocketTimeoutException' The server settings in the connection wizard are as follows- *NameNode Host*: blah.institution.edu *NameNode Port*: 9000 *Hadoop Version:* 0.20.2 *Username:* blahuser *Group:* blahusergroup *** Note: Verified that our live Hadoop version is *hadoop-0.20.2-cdh3u0 * and the fs.default.name configuration property value is* hdfs:// blah.institution.edu:9000 **** * * Followed all installation instructions in the Karmapshere guide. Is there something else that we are missing? Any hints or pointer will be useful. Thanks in advance! Alicia |
|
From: Ted R. <tre...@ka...> - 2012-02-29 17:00:49
|
David, Unfortunately the GUI emulator is not a full-blown Hadoop environment, and does not have input splits or counters. The is currently no way to do this in the emulator. Your best work around would be to deploy the job to either the local/embedded Hadoop or to a real Hadoop cluster. Ted. Ted Reynolds Technical Support Engineer *The Leader in Big Data Analytics for Hadoop* P: (650)292-6113 19200 Stevens Creek Blvd. Suite 130, Cupertino, CA 95014 On Fri, Feb 24, 2012 at 10:21 AM, David Walling <dav...@gm...>wrote: > It would appear the workflow gui emulator simply uses the 'do nothing' > static Reporter class in Hadoop. This breaks things if need to do > something like the following in your Mapper: > > <code> > FileSplit fileSplit = (FileSplit) reporter.getInputSplit(); > Path path = fileSplit.getPath(); > String fileName = path.getName(); > valueOut.set(fileName + "@" + key); > </code> > > You will receive the following error in the Mapper tab: > > "java.lang.UnsupportedOperationException: NULL reporter has no input" > > Is there a work around? > > > > ------------------------------------------------------------------------------ > Keep Your Developer Skills Current with LearnDevNow! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-d2d > _______________________________________________ > Hadoop-studio-users mailing list > Had...@li... > https://lists.sourceforge.net/lists/listinfo/hadoop-studio-users > > |
|
From: David W. <dav...@gm...> - 2012-02-24 18:22:09
|
It would appear the workflow gui emulator simply uses the 'do nothing' static Reporter class in Hadoop. This breaks things if need to do something like the following in your Mapper: <code> FileSplit fileSplit = (FileSplit) reporter.getInputSplit(); Path path = fileSplit.getPath(); String fileName = path.getName(); valueOut.set(fileName + "@" + key); </code> You will receive the following error in the Mapper tab: "java.lang.UnsupportedOperationException: NULL reporter has no input" Is there a work around? |
|
From: key m. <key...@gm...> - 2012-02-20 00:23:15
|
Hi Dear Karmasphere support, Got a quick question here - can someone quickly check the mail service (or some other components)? We are not receiving activation email after registered for multiple times. btw, this is Daniel. we have a couple of engineering still fighting with a Hadoop based prototype through the weekend. It will be nice if we can get through the Karmasphere plugin and show the prototype to the whole team in the beginning of incoming week. Apology for bringing this up in such a possibly hash way :). this is also rare for myself. Many Thanks, --Daniel |
|
From: Ted R. <tre...@ka...> - 2012-02-19 19:44:37
|
Scott,
This is just a note to let you know that we have received your question and
are working on the issue. We will get back to you soon. In the mean time
there is some information you can give us that will help us in tracking
down the problem:
* What platform and version are you running on?
* What is the Karmasphere product and version that the question concerns?
* What is the hadoop distribution you are using
* Can you send us the entire contents of the logs, in Analyst there are
three (Log, Karmasphere Debug & Hive on ...), in Studio there is just the
'Console'. To get these:
1. in Analyst go to 'View>>Logs'
a. when the window opens, click the mouse inside the window
b. press ctrl+a followed by ctrl+c (on Mac use command+a &
command+c)
c. open your favorite text editor and paste this in by pressing
ctrl+v (Mac command+v)
d. in Analyst repeat the above for the other two logs.
2. in Studio, if the 'Console' window is not already showing, go to
'Window>>Show View>>Console'.
a. do steps a - c from above as in Analyst.
Thank you for using Karmasphere products.
Ted.
--
--
Ted Reynolds
Technical Support Engineer
*The Leader in Big Data Analytics for Hadoop*
P: (650)292-6113
19200 Stevens Creek Blvd. Suite 130, Cupertino, CA 95014
Ted Reynolds
Technical Support Engineer
*The Leader in Big Data Analytics for Hadoop*
P: (650)292-6113
19200 Stevens Creek Blvd. Suite 130, Cupertino, CA 95014
On Sat, Feb 18, 2012 at 5:50 PM, Scott Selvia <ss...@gm...> wrote:
> After I installed the Karmasphere studio for netbeans 7.1 my system became
> totally un-responsive. Netbeans restarted and then cpu usage on my mac
> went up to 250%. I had to kill Netbeans and then removed all of the
> karmasphere module jars just to get netbeans working again.
>
> Is this normal behavior if you don't have hadoop installed? What is the
> installation procedure?
>
>
>
>
> ------------------------------------------------------------------------------
> Virtualization & Cloud Management Using Capacity Planning
> Cloud computing makes use of virtualization - but cloud computing
> also focuses on allowing computing to be delivered as a service.
> http://www.accelacomm.com/jaw/sfnl/114/51521223/
> _______________________________________________
> Hadoop-studio-users mailing list
> Had...@li...
> https://lists.sourceforge.net/lists/listinfo/hadoop-studio-users
>
|
|
From: Scott S. <ss...@gm...> - 2012-02-19 01:50:38
|
After I installed the Karmasphere studio for netbeans 7.1 my system became totally un-responsive. Netbeans restarted and then cpu usage on my mac went up to 250%. I had to kill Netbeans and then removed all of the karmasphere module jars just to get netbeans working again. Is this normal behavior if you don't have hadoop installed? What is the installation procedure? |
|
From: Gil A. <gi...@ka...> - 2012-02-15 22:45:01
|
Dear Tran, We will look into this promptly and reply back. Thank you, Gil On Tue, Feb 14, 2012 at 1:32 AM, Nam <nam...@gm...> wrote: > Hello, > I registered for Karmasphere Studio Community Edition on the website. > In turn I have received an emal with installation instructions and an > Eclipse update site. > I followed the instructions provided on the installation site (http:// > karmasphere.com/Studio-Eclipse/installation.html). > When I add the Karmasphere update site in Eclipse using the link you > provided me (http://updates.karmasphere.com/dist/*SERIALNUMBER*/ > eclipse/) Eclipse keeps on saying that "There are no items available". > I have tried with Eclipse 3.6.1 and 3.7. > > Could you help me installing Karmasphere Studio? > > And by the way, the email support address provided in the instructions > email (su...@ka...) keeps on being rejected by my mail > provider: > > <su...@ka...>: host smtp.secureserver.net[216.69.186.201] > said: 550 > #5.1.0 Address rejected su...@ka... (in reply to RCPT > TO > command) > > Best regards, > > Tran Nam-Luc > > > ------------------------------------------------------------------------------ > Virtualization & Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > _______________________________________________ > Hadoop-studio-users mailing list > Had...@li... > https://lists.sourceforge.net/lists/listinfo/hadoop-studio-users > |
|
From: Nam <nam...@gm...> - 2012-02-14 09:32:56
|
Hello, I registered for Karmasphere Studio Community Edition on the website. In turn I have received an emal with installation instructions and an Eclipse update site. I followed the instructions provided on the installation site (http:// karmasphere.com/Studio-Eclipse/installation.html). When I add the Karmasphere update site in Eclipse using the link you provided me (http://updates.karmasphere.com/dist/*SERIALNUMBER*/ eclipse/) Eclipse keeps on saying that "There are no items available". I have tried with Eclipse 3.6.1 and 3.7. Could you help me installing Karmasphere Studio? And by the way, the email support address provided in the instructions email (su...@ka...) keeps on being rejected by my mail provider: <su...@ka...>: host smtp.secureserver.net[216.69.186.201] said: 550 #5.1.0 Address rejected su...@ka... (in reply to RCPT TO command) Best regards, Tran Nam-Luc |
|
From: Wang Y. <fay...@gm...> - 2012-01-09 22:04:15
|
Hi, all I new to this Hadoop + netbean developing environment, after install the plugin and configure the job, I get such error when deploying the job, can anyone give some suggestions? Adding residual ClassPath entry jar:file:/Users/yuewang/.netbeans/7.0/modules/ext/commons-codec-1.4.jar!/ to roots. Adding residual ClassPath entry jar:file:/Users/yuewang/.netbeans/7.0/modules/ext/commons-httpclient-3.1.jar!/ to roots. Adding residual ClassPath entry jar:file:/Users/yuewang/.netbeans/7.0/modules/ext/commons-net-1.4.1.jar!/ to roots. Adding residual ClassPath entry jar:file:/Users/yuewang/.netbeans/7.0/modules/ext/oro-2.0.8.jar!/ to roots. Adding residual ClassPath entry jar:file:/Users/yuewang/.netbeans/7.0/modules/ext/log4j-1.2.15.jar!/ to roots. Adding residual ClassPath entry jar:file:/Users/yuewang/.netbeans/7.0/modules/ext/xmlenc-0.52.jar!/ to roots. Adding residual ClassPath entry jar:file:/Users/yuewang/.netbeans/7.0/modules/ext/jets3t-0.7.1.jar!/ to roots. Adding residual ClassPath entry jar:file:/Users/yuewang/.netbeans/7.0/modules/ext/hadoop-0.20.2-core.jar!/ to roots. Invocation failed. java.lang.NullPointerException at java.lang.String.<init>(String.java:210) at org.objectweb.asm.Type.getInternalName(Type.java:505) at com.karmasphere.studio.hadoop.executor.HadoopClientInspector$Visitor.test(HadoopClientInspector.java:127) at com.karmasphere.studio.hadoop.executor.HadoopClientInspector$Visitor.visitMethodInsn(HadoopClientInspector.java:165) at org.objectweb.asm.ClassReader.accept(ClassReader.java:1382) at org.objectweb.asm.ClassReader.accept(ClassReader.java:425) at com.karmasphere.studio.hadoop.executor.HadoopClientInspector.call(HadoopClientInspector.java:111) at com.karmasphere.studio.hadoop.executor.AbstractHadoopExecutorSupport.inspect(AbstractHadoopExecutorSupport.java:147) at com.karmasphere.studio.hadoop.netbeans.executor.NetbeansHadoopExecutorSupport.execute(NetbeansHadoopExecutorSupport.java:152) at com.karmasphere.studio.hadoop.executor.HadoopExecutorAssembler.execute(HadoopExecutorAssembler.java:215) at com.karmasphere.studio.hadoop.target.cluster.local.HadoopLocalCluster.execute(HadoopLocalCluster.java:92) at com.karmasphere.studio.hadoop.executor.HadoopExecutor.execute(HadoopExecutor.java:287) at com.karmasphere.studio.hadoop.executor.HadoopExecutor.run(HadoopExecutor.java:300) at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1424) at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1968) |
|
From: Rakesh K. <rka...@gm...> - 2011-12-29 10:16:59
|
Hi, Any updates on this issue??? Thanks & Regards, Rakesh K. On Wed, Dec 28, 2011 at 1:01 PM, Rakesh Kathpal <rka...@gm...> wrote: > Hi, > > The problem here is that I have created a CDH cluster and Filesystem in > Eclipse with Karmasphere for connecting to my CDH cluster. When I copy a > file from my local filesystem to CDH cluster in Karmasphere it shows the > replication factor to be 3 by default even if my settings in CDH are for > replication factor 1. > > Please help me resolving this issue asap. > > Thanks & Regards, > > Rakesh K. > |
|
From: Ted R. <tre...@ka...> - 2011-12-28 16:25:57
|
Rakesh, Just a note to let you know that we have received your question and will get back to you soon with an answer. Karmasphere Technical Support. On Tue, Dec 27, 2011 at 11:31 PM, Rakesh Kathpal <rka...@gm...> wrote: > Hi, > > The problem here is that I have created a CDH cluster and Filesystem in > Eclipse with Karmasphere for connecting to my CDH cluster. When I copy a > file from my local filesystem to CDH cluster in Karmasphere it shows the > replication factor to be 3 by default even if my settings in CDH are for > replication factor 1. > > Please help me resolving this issue asap. > > Thanks & Regards, > > Rakesh K. > > > ------------------------------------------------------------------------------ > Write once. Port to many. > Get the SDK and tools to simplify cross-platform app development. Create > new or port existing apps to sell to consumers worldwide. Explore the > Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join > http://p.sf.net/sfu/intel-appdev > _______________________________________________ > Hadoop-studio-users mailing list > Had...@li... > https://lists.sourceforge.net/lists/listinfo/hadoop-studio-users > > |
|
From: Rakesh K. <rka...@gm...> - 2011-12-28 07:32:05
|
Hi, The problem here is that I have created a CDH cluster and Filesystem in Eclipse with Karmasphere for connecting to my CDH cluster. When I copy a file from my local filesystem to CDH cluster in Karmasphere it shows the replication factor to be 3 by default even if my settings in CDH are for replication factor 1. Please help me resolving this issue asap. Thanks & Regards, Rakesh K. |
|
From: Travaglini, J. <Jos...@FM...> - 2011-12-12 16:43:18
|
Hello, I'm using the most recent version of the Eclipse plugin of studio. I don't see a way to, through the workflow tabs, define a output value grouping in the driver class (i.e. conf.setoutputvaluegrouping). Has anyone come across this issue? I can set the output key comparator but not the value grouping. Thanks -Joe |
|
From: Ted R. <tre...@ka...> - 2011-11-30 18:26:23
|
On Wed, Nov 30, 2011 at 6:10 AM, Anurag Saxena < anu...@co...> wrote: > I have many gziped log files in a folder which I need to give as input > to workflow in eclipse. How can I give the folder as input to > karmasphere hadoop eclipse workflow. > > Any help will be much appreciated. > > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure > contains a definitive record of customers, application performance, > security threats, fraudulent activity, and more. Splunk takes this > data and makes sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-novd2d > _______________________________________________ > Hadoop-studio-users mailing list > Had...@li... > https://lists.sourceforge.net/lists/listinfo/hadoop-studio-users > -- -- Ted Reynolds Technical Support Engineer *The Leader in Big Data Analytics for Hadoop* P: (650)292-6113 19200 Stevens Creek Blvd. Suite 130, Cupertino, CA 95014 |
|
From: Ted R. <tre...@ka...> - 2011-11-30 18:02:01
|
Anurag, Since the GUI workflow was meant for debugging your jobs on a small amount of data, the Bootstrap tab only allows you to select a single file. However when you submit your job to a cluster, even the embedded Hadoop, you can select a folder. The text in the field for the parameters to pass is editable. Please see the documentation on submitting / deploying jobs for how to do this. Thanks for choosing Karmasphere products. If you have any more issues / questions please don't hesitate to contact us. Ted. On Wed, Nov 30, 2011 at 6:10 AM, Anurag Saxena < anu...@co...> wrote: > I have many gziped log files in a folder which I need to give as input > to workflow in eclipse. How can I give the folder as input to > karmasphere hadoop eclipse workflow. > > Any help will be much appreciated. > > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure > contains a definitive record of customers, application performance, > security threats, fraudulent activity, and more. Splunk takes this > data and makes sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-novd2d > _______________________________________________ > Hadoop-studio-users mailing list > Had...@li... > https://lists.sourceforge.net/lists/listinfo/hadoop-studio-users > -- -- Ted Reynolds Technical Support Engineer *The Leader in Big Data Analytics for Hadoop* P: (650)292-6113 19200 Stevens Creek Blvd. Suite 130, Cupertino, CA 95014 |
|
From: Anurag S. <anu...@co...> - 2011-11-30 15:42:10
|
I have many gziped log files in a folder which I need to give as input to workflow in eclipse. How can I give the folder as input to karmasphere hadoop eclipse workflow. Any help will be much appreciated. |
|
From: Ted R. <tre...@ka...> - 2011-11-29 19:01:29
|
Loris, The quick answer to your question about debugging is, you can't debug in that manner, submitting a job to a hadoop cluster, whether local or remote, is not the same as running a Java program in Eclipse. The debugging you can do is you can visually see if your mappers, reducers and such are working as you expected immediately in the tabs on the workflow. Also if there is an exception somewhere in the process of the job running you will be able to see the exception in the tabs on the workflow and in the log windows. Thank you for trying Karmasphere products. Ted. On Mon, Nov 28, 2011 at 10:27 PM, Loris Degioanni < lor...@ri...> wrote: > Hi list, > I am a beginner with hadoop, and I'm interested in eclipse integration > so I can run my mapreduce programs in the debugger on my local system > before I deploy them into a cluster. I followed the "Local Development, > Debugging and Deployment" tutorial at > http://www.karmasphere.com/ksc/karmasphere-studio.html, but I found that > it doesn't really mention debugging other than in the title. > > So my question is: when I use local system deployment, can I do stuff > like setting breakpoints and step in the code? If yes, can you point me > to the place in the documentation where I learn how to do that? > > Thanks in advance, > Loris > > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure > contains a definitive record of customers, application performance, > security threats, fraudulent activity, and more. Splunk takes this > data and makes sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-novd2d > _______________________________________________ > Hadoop-studio-users mailing list > Had...@li... > https://lists.sourceforge.net/lists/listinfo/hadoop-studio-users > -- -- Ted Reynolds Technical Support Engineer *The Leader in Big Data Analytics for Hadoop* P: (650)292-6113 19200 Stevens Creek Blvd. Suite 130, Cupertino, CA 95014 |
|
From: Loris D. <lor...@ri...> - 2011-11-29 06:27:12
|
Hi list, I am a beginner with hadoop, and I'm interested in eclipse integration so I can run my mapreduce programs in the debugger on my local system before I deploy them into a cluster. I followed the "Local Development, Debugging and Deployment" tutorial at http://www.karmasphere.com/ksc/karmasphere-studio.html, but I found that it doesn't really mention debugging other than in the title. So my question is: when I use local system deployment, can I do stuff like setting breakpoints and step in the code? If yes, can you point me to the place in the documentation where I learn how to do that? Thanks in advance, Loris |
|
From: mailman d. <mai...@gm...> - 2011-11-21 20:20:09
|
Thanks! I figured that after posting the question. Anyway, I appreciate your response! On Thu, Nov 17, 2011 at 10:43 AM, Ted Reynolds <tre...@ka...>wrote: > Hi, > > Did you start a new Java project first? The HadoopJob is a Java class and > as such needs to be in a java project. > > Ted. > > On Wed, Nov 16, 2011 at 8:55 AM, mailman deepak <mai...@gm...>wrote: > >> Hi, >> >> I just setup the community edition. While creating a new job, i use the >> Wizard where it asks for the Source Folder for the new job. Whatever path I >> enter there, it I always get a "Folder Does Not Exist" Error on top. >> >> I'm unable to browse my local file system to specify a directory. Am I >> missing something here? >> >> >> ------------------------------------------------------------------------------ >> All the data continuously generated in your IT infrastructure >> contains a definitive record of customers, application performance, >> security threats, fraudulent activity, and more. Splunk takes this >> data and makes sense of it. IT sense. And common sense. >> http://p.sf.net/sfu/splunk-novd2d >> _______________________________________________ >> Hadoop-studio-users mailing list >> Had...@li... >> https://lists.sourceforge.net/lists/listinfo/hadoop-studio-users >> >> > > > -- > -- > Ted Reynolds > Technical Support Engineer > *The Leader in Big Data Analytics for Hadoop* > > P: (650)292-6113 > 19200 Stevens Creek Blvd. Suite 130, Cupertino, CA 95014 > > > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure > contains a definitive record of customers, application performance, > security threats, fraudulent activity, and more. Splunk takes this > data and makes sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-novd2d > _______________________________________________ > Hadoop-studio-users mailing list > Had...@li... > https://lists.sourceforge.net/lists/listinfo/hadoop-studio-users > > |
|
From: Rakesh K. <rka...@gm...> - 2011-11-21 05:35:08
|
Any updates on this issue? Can anyone please help me out with this? On Fri, Nov 18, 2011 at 11:25 AM, Rakesh Kathpal <rka...@gm...> wrote: > Hi, > > I have tried to follow all instructions from the link > http://www.karmasphere.com/Studio-Eclipse/3-remote-deployment.html > > But still seems to be some error and I am not able to connect. In eclispe > I am getting the following error > > java.io.EOFException.Closed connection before call complete. abondoned > call invocation..................... > > > And on the hadoop name, in the logs I am getting the following error > > 2011-11-18 10:48:00,270 WARN org.apache.hadoop.ipc.Server: Incorrect > header or version mismatch from 180.179.93.110:49444 got version 3 > expected version 4 > 2011-11-18 10:49:00,271 WARN org.apache.hadoop.ipc.Server: Incorrect > header or version mismatch from 180.179.93.110:49445 got version 3 > expected version 4 > 2011-11-18 10:50:00,276 WARN org.apache.hadoop.ipc.Server: Incorrect > header or version mismatch from 180.179.93.110:49446 got version 3 > expected version 4 > 2011-11-18 10:51:00,279 WARN org.apache.hadoop.ipc.Server: Incorrect > header or version mismatch from 180.179.93.110:49447 got version 3 > expected version 4 > > Here the server *180.179.93.110 *is the one I have installed Karmasphere > community edition on. The details of the CDH version are as follows > > [root@hadoop5 ~]# hadoop version > Hadoop 0.20.2-cdh3u1 > Subversion file:///tmp/topdir/BUILD/hadoop-0.20.2-cdh3u1 -r > bdafb1dbffd0d5f2fbc6ee022e1c8df6500fd638 > Compiled by root on Mon Jul 18 09:40:22 PDT 2011 > From source with checksum 3127e3d410455d2bacbff7673bf3284c > > Can anyone tell me if this is a known issue or if there any fix to this > problem?????????????? > > > Thanks & Regards, > > Rakesh K. > |
|
From: Ted R. <tre...@ka...> - 2011-11-18 15:50:56
|
Rakesh, In the 'bootstrap' tab of the work flow, what version of hadoop did you choose? For a CDH3u1 cluster as the target you need to choose 'Cloudera CDH3'. Ted. On Thu, Nov 17, 2011 at 9:55 PM, Rakesh Kathpal <rka...@gm...> wrote: > Hi, > > I have tried to follow all instructions from the link > http://www.karmasphere.com/Studio-Eclipse/3-remote-deployment.html > > But still seems to be some error and I am not able to connect. In eclispe > I am getting the following error > > java.io.EOFException.Closed connection before call complete. abondoned > call invocation..................... > > > And on the hadoop name, in the logs I am getting the following error > > 2011-11-18 10:48:00,270 WARN org.apache.hadoop.ipc.Server: Incorrect > header or version mismatch from 180.179.93.110:49444 got version 3 > expected version 4 > 2011-11-18 10:49:00,271 WARN org.apache.hadoop.ipc.Server: Incorrect > header or version mismatch from 180.179.93.110:49445 got version 3 > expected version 4 > 2011-11-18 10:50:00,276 WARN org.apache.hadoop.ipc.Server: Incorrect > header or version mismatch from 180.179.93.110:49446 got version 3 > expected version 4 > 2011-11-18 10:51:00,279 WARN org.apache.hadoop.ipc.Server: Incorrect > header or version mismatch from 180.179.93.110:49447 got version 3 > expected version 4 > > Here the server *180.179.93.110 *is the one I have installed Karmasphere > community edition on. The details of the CDH version are as follows > > [root@hadoop5 ~]# hadoop version > Hadoop 0.20.2-cdh3u1 > Subversion file:///tmp/topdir/BUILD/hadoop-0.20.2-cdh3u1 -r > bdafb1dbffd0d5f2fbc6ee022e1c8df6500fd638 > Compiled by root on Mon Jul 18 09:40:22 PDT 2011 > From source with checksum 3127e3d410455d2bacbff7673bf3284c > > Can anyone tell me if this is a known issue or if there any fix to this > problem?????????????? > > > Thanks & Regards, > > Rakesh K. > > > ------------------------------------------------------------------------------ > All the data continuously generated in your IT infrastructure > contains a definitive record of customers, application performance, > security threats, fraudulent activity, and more. Splunk takes this > data and makes sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-novd2d > _______________________________________________ > Hadoop-studio-users mailing list > Had...@li... > https://lists.sourceforge.net/lists/listinfo/hadoop-studio-users > > -- -- Ted Reynolds Technical Support Engineer *The Leader in Big Data Analytics for Hadoop* P: (650)292-6113 19200 Stevens Creek Blvd. Suite 130, Cupertino, CA 95014 |
|
From: Rakesh K. <rka...@gm...> - 2011-11-18 05:55:19
|
Hi, I have tried to follow all instructions from the link http://www.karmasphere.com/Studio-Eclipse/3-remote-deployment.html But still seems to be some error and I am not able to connect. In eclispe I am getting the following error java.io.EOFException.Closed connection before call complete. abondoned call invocation..................... And on the hadoop name, in the logs I am getting the following error 2011-11-18 10:48:00,270 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 180.179.93.110:49444 got version 3 expected version 4 2011-11-18 10:49:00,271 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 180.179.93.110:49445 got version 3 expected version 4 2011-11-18 10:50:00,276 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 180.179.93.110:49446 got version 3 expected version 4 2011-11-18 10:51:00,279 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 180.179.93.110:49447 got version 3 expected version 4 Here the server *180.179.93.110 *is the one I have installed Karmasphere community edition on. The details of the CDH version are as follows [root@hadoop5 ~]# hadoop version Hadoop 0.20.2-cdh3u1 Subversion file:///tmp/topdir/BUILD/hadoop-0.20.2-cdh3u1 -r bdafb1dbffd0d5f2fbc6ee022e1c8df6500fd638 Compiled by root on Mon Jul 18 09:40:22 PDT 2011 >From source with checksum 3127e3d410455d2bacbff7673bf3284c Can anyone tell me if this is a known issue or if there any fix to this problem?????????????? Thanks & Regards, Rakesh K. |
10 messages has been excluded from this view by a project administrator.
