Hadoop - MapReduce 실행 오류

Posted 11 24, 2015 18:41, Filed under: BigData/Hadoop


Hadoop MapReduce 실행시 오류 발생

관련 속성 : mapreduce.application.framework.path

에러 1
java.lang.IllegalArgumentException: Unable to parse '/usr/hdp/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework' as a URI, check the setting for mapreduce.application.framework.path
    at org.apache.hadoop.mapreduce.v2.util.MRApps.getMRFrameworkName(MRApps.java:178)
    at org.apache.hadoop.mapreduce.v2.util.MRApps.setMRFrameworkClasspath(MRApps.java:203)
    at org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:248)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.getInitialClasspath(TaskAttemptImpl.java:620)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:755)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:812)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1527)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1504)
    at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
    at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
    at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
    at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1069)
    at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:145)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1311)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1303)
    at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
    at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.URISyntaxException: Illegal character in path at index 11: /usr/hdp/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework
    at java.net.URI$Parser.fail(URI.java:2829)
    at java.net.URI$Parser.checkChars(URI.java:3002)
    at java.net.URI$Parser.parseHierarchical(URI.java:3086)
    at java.net.URI$Parser.parse(URI.java:3044)
    at java.net.URI.<init>(URI.java:595)
    at org.apache.hadoop.mapreduce.v2.util.MRApps.getMRFrameworkName(MRApps.java:176)
    ... 18 more

에러 2
Exception from container-launch.
Container id: container_e01_1448332875936_0002_01_000002
Exit code: 1
Exception message: /data/hadoop/yarn/local/usercache/hdfs/appcache/application_1448332875936_0002/container_e01_1448332875936_0002_01_000002/launch_container.sh: line 19: $PWD:/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64: bad substitution
/data/hadoop/yarn/local/usercache/hdfs/appcache/application_1448332875936_0002/container_e01_1448332875936_0002_01_000002/launch_container.sh: line 70: $JAVA_HOME/bin/java -server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version} -Xmx1228m -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/app/hadoop/yarn/log/application_1448332875936_0002/container_e01_1448332875936_0002_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 211.111.22.33 55273 attempt_1448332875936_0002_m_000000_0 1099511627778 1>/app/hadoop/yarn/log/application_1448332875936_0002/container_e01_1448332875936_0002_01_000002/stdout 2>/app/hadoop/yarn/log/application_1448332875936_0002/container_e01_1448332875936_0002_01_000002/stderr : bad substitution

Stack trace: ExitCodeException exitCode=1: /data/hadoop/yarn/local/usercache/hdfs/appcache/application_1448332875936_0002/container_e01_1448332875936_0002_01_000002/launch_container.sh: line 19: $PWD:/usr/hdp/${hdp.version}/hadoop/lib/native:/usr/hdp/${hdp.version}/hadoop/lib/native/Linux-amd64-64: bad substitution
/data/hadoop/yarn/local/usercache/hdfs/appcache/application_1448332875936_0002/container_e01_1448332875936_0002_01_000002/launch_container.sh: line 70: $JAVA_HOME/bin/java -server -XX:NewRatio=8 -Djava.net.preferIPv4Stack=true -Dhdp.version=${hdp.version} -Xmx1228m -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/app/hadoop/yarn/log/application_1448332875936_0002/container_e01_1448332875936_0002_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 211.111.22.33 55273 attempt_1448332875936_0002_m_000000_0 1099511627778 1>/app/hadoop/yarn/log/application_1448332875936_0002/container_e01_1448332875936_0002_01_000002/stdout 2>/app/hadoop/yarn/log/application_1448332875936_0002/container_e01_1448332875936_0002_01_000002/stderr : bad substitution

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)
    at org.apache.hadoop.util.Shell.run(Shell.java:487)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1


1. HDFS의 "mapreduce.application.framework.path" 설정


HDFS 디렉토리 생성, mapreduce.tar.gz 파일을 HDFS 에 업로드(put) 하고 권한 설정

su - hdfs -c "hdfs dfs -mkdir -p /hdp/apps/2.3.0.0-<$version>/mapreduce/"
su - hdfs -c "hdfs dfs -put /usr/hdp/2.3.0.0-<$version>/hadoop/mapreduce.tar.gz /hdp/apps/2.3.0.0-<$version>/mapreduce/"
su - hdfs -c "hdfs dfs -chown -R hdfs:hadoop /hdp"
su - hdfs -c "hdfs dfs -chmod -R 555 /hdp/apps/2.3.0.0-<$version>/mapreduce"
su - hdfs -c "hdfs dfs -chmod -R 444 /hdp/apps/2.3.0.0-<$version>/mapreduce/mapreduce.tar.gz"

파일 위치 : $HADOOP_HOME/conf/mapred-site.xml
<property>
 <name>mapreduce.application.framework.path</name>
 <value>/hdp/apps/${hdp.version}/mapreduce/mapreduce.tar.gz#mr-framework</value>
</property>
                 
<property>
 <name>yarn.app.mapreduce.am.admin-comand-opts</name>
 <value>-Dhdp.version=${hdp.version}</value>
</property>

2. Ambari Web "mapreduce.application.framework.path" 설정

사용자 삽입 이미지


3. Client 접속시 설정 추가

 - HDFS 일반 : mapreduce.application.framework.path = /hdp/apps/2.3.2.0-2950/mapreduce/mapreduce.tar.gz#mr-framework

 - TOSBD(Talend Open Studio for Big Data) :
 
사용자 삽입 이미지






※ 위 내용은, 여러 자료를 참고하거나 제가 주관적으로 정리한 것입니다.
   잘못된 정보나 보완이 필요한 부분을, 댓글 또는 메일로 보내주시면 많은 도움이 되겠습니다.
11 24, 2015 18:41 11 24, 2015 18:41


Trackback URL : http://develop.sunshiny.co.kr/trackback/1044

Leave a comment


Recent Posts

  1. HDFS - Python Encoding 오류 처리
  2. HP - Vertica ROS Container 관련 오류...
  3. HDFS - Hive 실행시 System Time 오류
  4. HP - Vertica 사용자 쿼리 이력 테이블...
  5. Client에서 HDFS 환경의 데이터 처리시...

Recent Comments

  1. 안녕하세요^^ 배그핵
  2. 안녕하세요^^ 도움이 되셨다니, 저... sunshiny
  3. 정말 큰 도움이 되었습니다.. 감사합... 사랑은
  4. 네, 안녕하세요. 댓글 남겨 주셔서... sunshiny
  5. 감사합니다 많은 도움 되었습니다!ㅎㅎ 프리시퀸스

Recent Trackbacks

  1. chatbots chatbots %M
  2. london relocation agents london relocation agents %M
  3. how to build chatbot how to build chatbot %M
  4. advanced chatbot advanced chatbot %M
  5. facebook ai chatbot facebook ai chatbot %M

Calendar

«   08 2019   »
        1 2 3
4 5 6 7 8 9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30 31

Bookmarks

  1. 위키피디아
  2. MysqlKorea
  3. 오라클 클럽
  4. API - Java
  5. Apache Hadoop API
  6. Apache Software Foundation
  7. HDFS 생태계 솔루션
  8. DNSBL - Spam Database Lookup
  9. Ready System
  10. Solaris Freeware
  11. Linux-Site
  12. 윈디하나의 솔라나라

Site Stats

TOTAL 2660975 HIT
TODAY 1291 HIT
YESTERDAY 1540 HIT