Oozie - 예제 테스트

Posted 01 15, 2014 21:13, Filed under: BigData/Oozie


# Oozie 예제 테스트


# Oozie 설정 정보
http://oozie.apache.org/docs/3.1.3-incubating/CoordinatorFunctionalSpec.html

# 예제 파일 압축 풀기
[hadoop@master /home/oozie-3.3.2]$ tar -xzvf oozie-examples.tar.gz

# job.properties 파일 설정
위치 : $OOZIE_HOME/examples/apps/map-reduce/job.properties

[hadoop@master ]$ cat job.properties
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

nameNode=hdfs://master:8020
jobTracker=master:8021
queueName=default
examplesRoot=examples

###### HDFS상의 경로 ######
# oozie.wf.application.path=hdfs://master:8020/hdfs_dir/examples/apps/map-reduce
# ${nameNode} => hdfs://master:8020
########################
# 기본 hadoop 사용자 디렉토리 실행시
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce
# 다른 디렉토리 사용시
#oozie.wf.application.path=${nameNode}/hdfs_dir/examples/apps/map-reduce

outputDir=map-reduce


# workflow.xml 설정
위치 : $OOZIE_HOME/examples/apps/map-reduce/workflow.xml
<!--
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at
  
       http://www.apache.org/licenses/LICENSE-2.0
  
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<workflow-app xmlns="uri:oozie:workflow:0.2" name="map-reduce-wf">
    <start to="mr-node"/>
    <action name="mr-node">
        <map-reduce>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <prepare>
        <!-- 임의정의 : ${nameNode}/hdfs_dir/examples/output-data/${outputDir} -->
                <delete path="${nameNode}/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}"/>
            </prepare>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>mapred.mapper.class</name>
                    <value>org.apache.oozie.example.SampleMapper</value>
                </property>
                <property>
                    <name>mapred.reducer.class</name>
                    <value>org.apache.oozie.example.SampleReducer</value>
                </property>
                <property>
                    <name>mapred.map.tasks</name>
                    <value>1</value>
                </property>
                <property>
                    <name>mapred.input.dir</name>
                    <value>/user/${wf:user()}/${examplesRoot}/input-data/text</value>
                </property>
                <property>
                    <name>mapred.output.dir</name>
                    <value>/user/${wf:user()}/${examplesRoot}/output-data/${outputDir}</value>
                </property>
            </configuration>
        </map-reduce>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Map/Reduce failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>


# 잡 구동및 확인
[hadoop@master /home/oozie-3.3.2]$ oozie job -oozie http://master:11000/oozie -config examples/apps/map-reduce/job.properties -run
job: 0000003-140114143845917-oozie-hdad-W

[hadoop@master /home/oozie-3.3.2]$ oozie job -oozie http://master:11000/oozie -info 0000003-140114143845917-oozie-hdad-W
Job ID : 0000003-140114143845917-oozie-hdad-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : map-reduce-wf
App Path      : hdfs://master:8020/hdfs_dir/examples/apps/map-reduce
Status        : SUCCEEDED
Run           : 0
User          : hdadm
Group         : -
Created       : 2014-01-15 01:54 GMT
Started       : 2014-01-15 01:54 GMT
Last Modified : 2014-01-15 01:54 GMT
Ended         : 2014-01-15 01:54 GMT
CoordAction ID: -

Actions
------------------------------------------------------------------------------------------------------------------------------------
ID                                                                            Status    Ext ID                 Ext Status Err Code 
------------------------------------------------------------------------------------------------------------------------------------
0000003-140114143845917-oozie-hdad-W@:start:                                  OK        -                      OK         -        
------------------------------------------------------------------------------------------------------------------------------------
0000003-140114143845917-oozie-hdad-W@mr-node                                  OK        job_201401141326_0024  SUCCEEDED  -        
------------------------------------------------------------------------------------------------------------------------------------
0000003-140114143845917-oozie-hdad-W@end                                      OK        -                      OK         -        
------------------------------------------------------------------------------------------------------------------------------------



※ 위 내용은, 여러 자료를 참고하거나 제가 주관적으로 정리한 것입니다.
   잘못된 정보나 보완이 필요한 부분을, 댓글 또는 메일로 보내주시면 많은 도움이 되겠습니다.

"BigData / Oozie" 분류의 다른 글

Oozie - coordinator 잡 실행 (0)2014/02/17
Oozie - Job 실행시 에러 (0)2014/01/15
Oozie - Client Demo (0)2014/01/15
01 15, 2014 21:13 01 15, 2014 21:13


Trackback URL : http://develop.sunshiny.co.kr/trackback/986

Leave a comment
[로그인][오픈아이디란?]
오픈아이디로만 댓글을 남길 수 있습니다

« Previous : 1 : ... 56 : 57 : 58 : 59 : 60 : 61 : 62 : 63 : 64 : ... 648 : Next »

Recent Posts

  1. HDFS - Python Encoding 오류 처리
  2. HP - Vertica ROS Container 관련 오류...
  3. HDFS - Hive 실행시 System Time 오류
  4. HP - Vertica 사용자 쿼리 이력 테이블...
  5. Client에서 HDFS 환경의 데이터 처리시...

Recent Comments

  1. 안녕하세요^^ 배그핵
  2. 안녕하세요^^ 도움이 되셨다니, 저... sunshiny
  3. 정말 큰 도움이 되었습니다.. 감사합... 사랑은
  4. 네, 안녕하세요. 댓글 남겨 주셔서... sunshiny
  5. 감사합니다 많은 도움 되었습니다!ㅎㅎ 프리시퀸스

Recent Trackbacks

  1. print & mail services print & mail services 13 07
  2. montana fishing vacations montana fishing vacations 13 07
  3. decks colorado decks colorado 13 07
  4. phoenix az closet organizers phoenix az closet organizers 13 07
  5. useful reference useful reference 13 07

Calendar

«   07 2020   »
      1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31  

Bookmarks

  1. 위키피디아
  2. MysqlKorea
  3. 오라클 클럽
  4. API - Java
  5. Apache Hadoop API
  6. Apache Software Foundation
  7. HDFS 생태계 솔루션
  8. DNSBL - Spam Database Lookup
  9. Ready System
  10. Solaris Freeware
  11. Linux-Site
  12. 윈디하나의 솔라나라

Site Stats

TOTAL 3006611 HIT
TODAY 52 HIT
YESTERDAY 1048 HIT