HDFS- Zookeeper 설치

Posted 05 10, 2013 21:16, Filed under: BigData



1) Hadoop 설치 : http://develop.sunshiny.co.kr/868

Zookeeper 설치 버전 : zookeeper-3.4.5

# 환경 설정 확인 : Hadoop, Zookeeper, HBase

[hadoop@master ~]$ vi .bash_profileJAVA_HOME=/home/hadoop/jdk1.7.0_17export JAVA_HOME

export HADOOP_HOME=/home/hadoop/hadoop-1.1.2
export HBASE_HOME=/home/hadoop/hbase
export ZOOKEEPER_HOME=/home/hadoop/zookeeper

PATH=$PATH:$HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HBASE_HOME/bin:$ZOOKEEPER_HOME/bin

export PATH


# 주키퍼 zoo.cfg 파일 설정

[hadoop@master conf]$ ls
configuration.xsl  log4j.properties  zoo_sample.cfg
[hadoop@master conf]$ cp zoo_sample.cfg zoo.cfg
[hadoop@master conf]$ ls
configuration.xsl  log4j.properties  zoo.cfg  zoo_sample.cfg
[hadoop@master conf]$ vi zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
#dataDir=/tmp/zookeeper
dataDir=/home/hadoop/zookeeper/data
dataLogDir=/home/hadoop/zookeeper/datalog

# the port at which the clients will connect
clientPort=2181
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

# 주키퍼 멀티 서버 연결관리시 설정
server.1=master.namenode:2888:3888
server.2=secondary.namenode:2888:3888
server.3=datanode01:2888:3888


# 주키퍼 데이타, 로그 디렉토리 생성

[hadoop@master zookeeper]$ mkdir -p $ZOOKEEPER_HOME/data
[hadoop@master zookeeper]$ mkdir -p $ZOOKEEPER_HOME/datalog


# 설정파일 배포

master.namenode 에 설치되어 있는 zookeeper 디렉토리 전체를 배포

// secondary.namenode
원격전송
scp -r /home/hadoop/zookeeper secondary.namenode:/home/hadoop/zookeeper
 
// datanode
원격전송
scp -r /home/hadoop/zookeeper datanode01:/home/hadoop/zookeeper

scp -r /home/hadoop/zookeeper datanode02:/home/hadoop/zookeeper
scp -r /home/hadoop/zookeeper datanode03:/home/hadoop/zookeeper


# NameNode에서 각 노드별 아이디 값 저장

// master.namenode
[hadoop@master zookeeper]$ echo "1" > /home/hadoop/zookeeper/data/myid

// secondary.namenode
원격전송
ssh hadoop@secondary.namenode "echo "2" > /home/hadoop/zookeeper/data/myid"

// datanode
원격전송
ssh hadoop@datanode01 "echo "3" > /home/hadoop/zookeeper/data/myid"


# NameNode에서 각 노드별 주키퍼 구동

// master.namenode
[hadoop@master zookeeper]$ /home/hadoop/zookeeper/bin/zkServer.sh start

// secondary.namenode 원격구동
ssh hadoop@secondary.namenode "/home/hadoop/zookeeper/bin/zkServer.sh start"

// datanode
원격구동
ssh hadoop@datanode01 "/home/hadoop/zookeeper/bin/zkServer.sh start"


# 주키퍼 구동 확인

[hadoop@zookeeper1 bin]$ jps
7981 TaskTracker
9377 Jps
7894 DataNode
9105 QuorumPeerMain
[hadoop@master bin]$ ./zkCli.sh -server localhost:2181
Connecting to localhost:2181
2013-05-10 15:58:57,713 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2013-05-10 15:58:57,716 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=master.namenode
2013-05-10 15:58:57,716 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.7.0_17
2013-05-10 15:58:57,716 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2013-05-10 15:58:57,717 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/home/hadoop/jdk1.7.0_17/jre
2013-05-10 15:58:57,717 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/home/hadoop/zookeeper/bin/../build/classes:/home/hadoop/zookeeper/bin/../build/lib/*.jar:/home/hadoop/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/hadoop/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/home/hadoop/zookeeper/bin/../lib/netty-3.2.2.Final.jar:/home/hadoop/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hadoop/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hadoop/zookeeper/bin/../zookeeper-3.4.5.jar:/home/hadoop/zookeeper/bin/../src/java/lib/*.jar:/home/hadoop/zookeeper/bin/../conf:
2013-05-10 15:58:57,718 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2013-05-10 15:58:57,718 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2013-05-10 15:58:57,718 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2013-05-10 15:58:57,719 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2013-05-10 15:58:57,719 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2013-05-10 15:58:57,719 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=2.6.32-279.el6.x86_64
2013-05-10 15:58:57,720 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=hadoop
2013-05-10 15:58:57,720 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/home/hadoop
2013-05-10 15:58:57,720 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/hadoop/zookeeper-3.4.5/bin
2013-05-10 15:58:57,721 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@30d23f13
Welcome to ZooKeeper!
2013-05-10 15:58:57,744 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@966] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2013-05-10 15:58:57,747 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@849] - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
JLine support is enabled
[zk: localhost:2181(CONNECTING) 0] 2013-05-10 15:58:57,782 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1207] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x13e8d3a66380000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: localhost:2181(CONNECTED) 0]
[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper]
[zk: localhost:2181(CONNECTED) 1] quit
Quitting...
2013-05-10 15:59:13,635 [myid:] - INFO  [main:ZooKeeper@684] - Session: 0x13e8d3a66380000 closed
2013-05-10 15:59:13,636 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@509] - EventThread shut down


# 주키퍼 마스터에서 노드에 멀티 접속

[hadoop@master bin]$ ./zkCli.sh -server master.namenode,secondary.namenode,datanode01
master.namenode,secondary.namenode,datanode01Connecting to
2013-05-10 17:36:35,412 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2013-05-10 17:36:35,415 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=master.namenode
2013-05-10 17:36:35,415 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.7.0_17
2013-05-10 17:36:35,416 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2013-05-10 17:36:35,416 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/home/hadoop/jdk1.7.0_17/jre
2013-05-10 17:36:35,416 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/home/hadoop/zookeeper/bin/../build/classes:/home/hadoop/zookeeper/bin/../build/lib/*.jar:/home/hadoop/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/hadoop/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/home/hadoop/zookeeper/bin/../lib/netty-3.2.2.Final.jar:/home/hadoop/zookeeper/bin/../lib/log4j-1.2.15.jar:/home/hadoop/zookeeper/bin/../lib/jline-0.9.94.jar:/home/hadoop/zookeeper/bin/../zookeeper-3.4.5.jar:/home/hadoop/zookeeper/bin/../src/java/lib/*.jar:/home/hadoop/zookeeper/bin/../conf:
2013-05-10 17:36:35,417 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2013-05-10 17:36:35,417 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2013-05-10 17:36:35,417 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2013-05-10 17:36:35,418 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2013-05-10 17:36:35,418 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2013-05-10 17:36:35,418 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=2.6.32-279.el6.x86_64
2013-05-10 17:36:35,419 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=hadoop
2013-05-10 17:36:35,419 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/home/hadoop
2013-05-10 17:36:35,419 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/hadoop/zookeeper-3.4.5/bin
2013-05-10 17:36:35,421 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=master.namenode,secondary.namenode,datanode01 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@3b2b116f
Welcome to ZooKeeper!

2013-05-10 17:36:35,439 [myid:] - INFO  [main-SendThread(master.namenode:2181):ClientCnxn$SendThread@966] - Opening socket connection to server master.namenode/192.168.1.17:2181. Will not attempt to authenticate using SASL (unknown error)

2013-05-10 17:36:35,442 [myid:] - INFO  [main-SendThread(master.namenode:2181):ClientCnxn$SendThread@849] - Socket connection established to master.namenode/192.168.1.17:2181, initiating session
JLine support is enabled

2013-05-10 17:36:35,453 [myid:] - INFO  [main-SendThread(master.namenode:2181):ClientCnxn$SendThread@1207] - Session establishment complete on server master.namenode/192.168.1.17:2181, sessionid = 0x13e8d3a66380001, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: master.namenode,secondary.namenode,datanode01(CONNECTED) 0]


# 파일 생성 테스트

# master.namenode 에서 확인
[zk: master.namenode,secondary.namenode,datanode01(CONNECTED) 11] create /e a
Created /e
[zk: master.namenode,secondary.namenode,datanode01(CONNECTED) 12] get /e
a
cZxid = 0x100000006
ctime = Fri May 10 18:18:56 KST 2013
mZxid = 0x100000006
mtime = Fri May 10 18:18:56 KST 2013
pZxid = 0x100000006
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 1
numChildren = 0

# secondary.namenode 에서 확인
[zk: secondary.namenode:2181(CONNECTED) 2] get /e
a
cZxid = 0x100000006
ctime = Fri May 10 18:18:56 KST 2013
mZxid = 0x100000006
mtime = Fri May 10 18:18:56 KST 2013
pZxid = 0x100000006
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 1
numChildren = 0

# datanode01 에서 확인
[zk: datanode01:2181(CONNECTED) 2] get /e
a
cZxid = 0x100000006
ctime = Fri May 10 18:18:56 KST 2013
mZxid = 0x100000006
mtime = Fri May 10 18:18:56 KST 2013
pZxid = 0x100000006
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 1
numChildren = 0



※ 위 내용은, 여러 자료를 참고하거나 제가 주관적으로 정리한 것입니다.
   잘못된 정보나 보완이 필요한 부분을, 댓글 또는 메일로 보내주시면 많은 도움이 되겠습니다.
05 10, 2013 21:16 05 10, 2013 21:16


Trackback URL : http://develop.sunshiny.co.kr/trackback/886

Leave a comment
[로그인][오픈아이디란?]
오픈아이디로만 댓글을 남길 수 있습니다

« Previous : 1 : ... 130 : 131 : 132 : 133 : 134 : 135 : 136 : 137 : 138 : ... 648 : Next »

Recent Posts

  1. HDFS - Python Encoding 오류 처리
  2. HP - Vertica ROS Container 관련 오류...
  3. HDFS - Hive 실행시 System Time 오류
  4. HP - Vertica 사용자 쿼리 이력 테이블...
  5. Client에서 HDFS 환경의 데이터 처리시...

Recent Comments

  1. 안녕하세요^^ 배그핵
  2. 안녕하세요^^ 도움이 되셨다니, 저... sunshiny
  3. 정말 큰 도움이 되었습니다.. 감사합... 사랑은
  4. 네, 안녕하세요. 댓글 남겨 주셔서... sunshiny
  5. 감사합니다 많은 도움 되었습니다!ㅎㅎ 프리시퀸스

Recent Trackbacks

  1. phoenix seo phoenix seo %M
  2. animal mobility services animal mobility services %M
  3. face masks face masks %M
  4. healer healer 5 07
  5. small dog wheelchair small dog wheelchair 5 07

Calendar

«   07 2020   »
      1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31  

Bookmarks

  1. 위키피디아
  2. MysqlKorea
  3. 오라클 클럽
  4. API - Java
  5. Apache Hadoop API
  6. Apache Software Foundation
  7. HDFS 생태계 솔루션
  8. DNSBL - Spam Database Lookup
  9. Ready System
  10. Solaris Freeware
  11. Linux-Site
  12. 윈디하나의 솔라나라

Site Stats

TOTAL 3000415 HIT
TODAY 867 HIT
YESTERDAY 401 HIT