win7上eclipse提交mr至hadoop2.6分布式运行错误集锦

作者: weipanp
发布时间:2015-07-21 10:14:42

1、下载hadoop2.6,编译插件

2、Eclipse安装插件

3、安装maven插件,配置jar包,实现自动将工程打包jar

4、创建mapreduce工程,运行wordcount

5、在其中遇到各种错误

(1)java.lang.NullPointerException

修复方法:

下载hadoop-common-2.2.0-bin-master,解压将其中的winutils.exe和hadoop.dll拷贝到HADOOP_HOME/bin下

(2)Exception in thread "main" java.lang.UnsatisfiedLinkError:ConfiguredFailoverProxyProvider

修复方法:

在继承的抽象类ConfiguredFailoverProxyProvider中增加抽象方法

  public abstract boolean useLogicalURI(); 

  public abstract ProxyInfo<TgetProxy();

  public abstract Class<TgetInterface();

  public abstract void performFailover(T currentProxy);

  public abstract void close() throws IOException;

(3)Exception in thread "main" java.lang.UnsatisfiedLinkError: org.Apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V

at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native Method)

修复方法:在hadoop2.6源码里找到NativeCrc32.java,创建与源码一样的包名,拷贝NativeCrc32.java到该包工程目录下。

(4)org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control

修改代码

http://www.imooo.com/windows/windows/1330494.htm

 

YarnRunner.java的390行 (Apache Hadoop2.6的源码)

// Setup the command to run the AM      List<String> vargs = new ArrayList<String>(8);      vargs.add(Environment.JAVA_HOME.$() + "/bin/java");  



改为

vargs.add("$JAVA_HOME/bin/java");  


在YarnRunner.java类中,新增一个路径转换的方法

private void replaceEnvironment(Map<String, String> environment) {        String tmpClassPath = environment.get("CLASSPATH");        tmpClassPath=tmpClassPath.replaceAll(";", ":");        tmpClassPath=tmpClassPath.replaceAll("%PWD%", "\\$PWD");        tmpClassPath=tmpClassPath.replaceAll("%HADOOP_MAPRED_HOME%", "\\$HADOOP_MAPRED_HOME");        tmpClassPath= tmpClassPath.replaceAll("\\\\", "/" );        environment.put("CLASSPATH",tmpClassPath);  }  




在YarnRunner.java的在466行添加:

replaceEnvironment(environment);  

 

(5)Application application_1420601118686_0001 failed 2 times due to AM Container for appattempt_1420601118686_0001_000002 exited with exitCode: 1 

For more detailed output, check application tracking page:http://master:8088/proxy/application_1420601118686_0001/Then, click on links to logs of each attempt. 

Diagnostics: Exception from container-launch. 

Container id: container_1420601118686_0001_02_000001 

Exit code: 1 

Stack trace: ExitCodeException exitCode=1: 

 

添加路径:

Mapreduce-site.xml

   <property>

   <name>mapreduce.application.classpath</name>

   <value>

   /opt/app/hadoop-2.6.0/etc/hadoop,

   /opt/app/hadoop-2.6.0/share/hadoop/common/*,

   /opt/app/hadoop-2.6.0/share/hadoop/common/lib/*,

   /opt/app/hadoop-2.6.0/share/hadoop/hdfs/*,

   /opt/app/hadoop-2.6.0/share/hadoop/hdfs/lib/*,

   /opt/app/hadoop-2.6.0/share/hadoop/mapreduce/*,

   /opt/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,

   /opt/app/hadoop-2.6.0/share/hadoop/yarn/*,

   /opt/app/hadoop-2.6.0/share/hadoop/yarn/lib/*

   </value>

   </property>

Yarn-site.xml

   <property>

   <name>yarn.application.classpath</name>

   <value>

   /opt/app/hadoop-2.6.0/etc/hadoop,

   /opt/app/hadoop-2.6.0/share/hadoop/common/*,

   /opt/app/hadoop-2.6.0/share/hadoop/common/lib/*,

   /opt/app/hadoop-2.6.0/share/hadoop/hdfs/*,

   /opt/app/hadoop-2.6.0/share/hadoop/hdfs/lib/*,

   /opt/app/hadoop-2.6.0/share/hadoop/mapreduce/*,

   /opt/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,

   /opt/app/hadoop-2.6.0/share/hadoop/yarn/*,

   /opt/app/hadoop-2.6.0/share/hadoop/yarn/lib/*

   </value>

   </property>

版权声明:本文为博主原创文章,未经博主允许不得转载。

来源:http://blog.csdn.net/weipanp/article/details/42713121

推荐: