Oozie - 예제 테스트

Posted 01 15, 2014 21:13, Filed under: BigData/Oozie


# Oozie 예제 테스트


# Oozie 설정 정보
http://oozie.apache.org/docs/3.1.3-incubating/CoordinatorFunctionalSpec.html

# 예제 파일 압축 풀기
[hadoop@master /home/oozie-3.3.2]$ tar -xzvf oozie-examples.tar.gz

# job.properties 파일 설정
위치 : $OOZIE_HOME/examples/apps/map-reduce/job.properties

[hadoop@master ]$ cat job.properties
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

nameNode=hdfs://master:8020
jobTracker=master:8021
queueName=default
examplesRoot=examples

###### HDFS상의 경로 ######
# oozie.wf.application.path=hdfs://master:8020/hdfs_dir/examples/apps/map-reduce
# ${nameNode} => hdfs://master:8020
########################
# 기본 hadoop 사용자 디렉토리 실행시
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce
# 다른 디렉토리 사용시
#oozie.wf.application.path=${nameNode}/hdfs_dir/examples/apps/map-reduce

outputDir=map-reduce


# workflow.xml 설정
위치 : $OOZIE_HOME/examples/apps/map-reduce/workflow.xml
<!--
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at
  
       http://www.apache.org/licenses/LICENSE-2.0
  
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<workflow-app xmlns="uri:oozie:workflow:0.2" name="map-reduce-wf">
    <start to="mr-node"/>
    <action name="mr-node">
        <map-reduce>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
        <!-- 임의정의 : ${nameNode}/hdfs_dir/examples/output-data/${outputDir} -->
                <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>mapred.mapper.class</name>
                    <value>org.apache.oozie.example.SampleMapper</value>
                </property>
                <property>
                    <name>mapred.reducer.class</name>
                    <value>org.apache.oozie.example.SampleReducer</value>
                </property>
                <property>
                    <name>mapred.map.tasks</name>
                    <value>1</value>
                </property>
                <property>
                    <name>mapred.input.dir</name>
                    <value>/user/${wf:user()}/${examplesRoot}/input-data/text</value>
                </property>
                <property>
                    <name>mapred.output.dir</name>
                    <value>/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}</value>
                </property>
            </configuration>
        </map-reduce>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>


# 잡 구동및 확인
[hadoop@master /home/oozie-3.3.2]$ oozie job -oozie http://master:11000/oozie -config examples/apps/map-reduce/job.properties -run
job: 0000003-140114143845917-oozie-hdad-W

[hadoop@master /home/oozie-3.3.2]$ oozie job -oozie http://master:11000/oozie -info 0000003-140114143845917-oozie-hdad-W
Job ID : 0000003-140114143845917-oozie-hdad-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : map-reduce-wf
App Path      : hdfs://master:8020/hdfs_dir/examples/apps/map-reduce
Status        : SUCCEEDED
Run           : 0
User          : hdadm
Group         : -
Created       : 2014-01-15 01:54 GMT
Started       : 2014-01-15 01:54 GMT
Last Modified : 2014-01-15 01:54 GMT
Ended         : 2014-01-15 01:54 GMT
CoordAction ID: -

Actions
------------------------------------------------------------------------------------------------------------------------------------
ID                                                                            Status    Ext ID                 Ext Status Err Code 
------------------------------------------------------------------------------------------------------------------------------------
0000003-140114143845917-oozie-hdad-W@:start:                                  OK        -                      OK         -        
------------------------------------------------------------------------------------------------------------------------------------
0000003-140114143845917-oozie-hdad-W@mr-node                                  OK        job_201401141326_0024  SUCCEEDED  -        
------------------------------------------------------------------------------------------------------------------------------------
0000003-140114143845917-oozie-hdad-W@end                                      OK        -                      OK         -        
------------------------------------------------------------------------------------------------------------------------------------



※ 위 내용은, 여러 자료를 참고하거나 제가 주관적으로 정리한 것입니다.
   잘못된 정보나 보완이 필요한 부분을, 댓글 또는 메일로 보내주시면 많은 도움이 되겠습니다.

"BigData / Oozie" 분류의 다른 글

Oozie - coordinator 잡 실행 (0)2014/02/17
Oozie - Job 실행시 에러 (0)2014/01/15
Oozie - Client Demo (0)2014/01/15
01 15, 2014 21:13 01 15, 2014 21:13


Trackback URL : http://develop.sunshiny.co.kr/trackback/986

Leave a comment

« Previous : 1 : ... 56 : 57 : 58 : 59 : 60 : 61 : 62 : 63 : 64 : ... 648 : Next »

Recent Posts

  1. HDFS - Python Encoding 오류 처리
  2. HP - Vertica ROS Container 관련 오류...
  3. HDFS - Hive 실행시 System Time 오류
  4. HP - Vertica 사용자 쿼리 이력 테이블...
  5. Client에서 HDFS 환경의 데이터 처리시...

Recent Comments

  1. 안녕하세요^^ 배그핵
  2. 안녕하세요^^ 도움이 되셨다니, 저... sunshiny
  3. 정말 큰 도움이 되었습니다.. 감사합... 사랑은
  4. 네, 안녕하세요. 댓글 남겨 주셔서... sunshiny
  5. 감사합니다 많은 도움 되었습니다!ㅎㅎ 프리시퀸스

Recent Trackbacks

  1. wireless communication systems wireless communication systems %M
  2. amazon fire television amazon fire television %M
  3. how to broadcast your own tv station how to broadcast your own tv station %M
  4. elapsed time clock for operating r... elapsed time clock for operating r... %M
  5. Mysql - mysql 설치후 Character set... 멀고 가까움이 다르기 때문 %M

Calendar

«   12 2019   »
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30 31        

Bookmarks

  1. 위키피디아
  2. MysqlKorea
  3. 오라클 클럽
  4. API - Java
  5. Apache Hadoop API
  6. Apache Software Foundation
  7. HDFS 생태계 솔루션
  8. DNSBL - Spam Database Lookup
  9. Ready System
  10. Solaris Freeware
  11. Linux-Site
  12. 윈디하나의 솔라나라

Site Stats

TOTAL 2780514 HIT
TODAY 97 HIT
YESTERDAY 1360 HIT