# Redis(레디스) Sentinel을 이용한 HA(High Availability) 구성


사용자 삽입 이미지

Redis - Master, Slave(복제) 구성

참고 : http://redis.io/topics/sentinel

# Sentinel 옵션
monitor : 모니터링할 master 아이피 및 odown 기준을 위한 sentinel quorum(의사결정) 수 설정
              - quorum(의사결정)의 수만큼 장애감지에 따라 Objective down(Failover) 진행
down-after-milliseconds : 장애 감지 이후 몇초 후 sdown으로 전이 될지 설정
failover-timeout : timeout 시간내 Failover 완료시, 나머지 slave에게 새로운 마스터 설정 가능 하도록 command를 보냄
parallel-syncs : 새로 승격된 master와 syncs를 맞출 slave 갯수 설정
auth-pass : master 패스워드


# Redis Failover 판단 과정
* Subjective down (sdown) : 주관적 장애 감지
  - 하나의 Sentinel에서만 장애를 감지 한 상태


* Objective down (odown) : 객관적 장애 감지, Failover(장애극복) 진행
  - 여러 Sentinel(quorum의 수)에서 장애라고 판단 한 경우


# sentinel.conf
센티널은 마스터에 접근해서 슬레이브 노드들의 정보를 찾고, 각 슬레이브에 접근함.
그러므로 센티널 설정은 마스터의 정보만 입력해주면 됨.

> Master
sentinel monitor mymaster 192.168.1.100 6379 1(quorum : 하나의 센티널에서 모니터링)
sentinel down-after-milliseconds mymaster 60000  -- ms단위 : 현재 60초
sentinel failover-timeout mymaster 180000
sentinel parallel-syncs mymaster 1
sentinel auth-pass mymaster 1234  -- Master 노드 접근시 패스워드 필요

# Sentinel 실행
[redis@node01 redis-2.8.8]$ src/redis-sentinel ./sentinel.conf >> log/sentinel.log &

> Sentinel 로그
[9647] 22 Apr 20:17:07.146 # Sentinel runid is 5298af98c048201d21babae6646d8939231a31f8
[9647] 22 Apr 20:17:07.146 # +monitor master mymaster 192.168.1.100 6379 quorum 1
[9647] 22 Apr 20:17:08.150 * +slave slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.100 6379


# Sentinel Failover 테스트


# Sentinel 실행후 Master 서버 종료

> Slave 서버 로그

[7525] 22 Apr 20:19:12.553 - Reading from client: Connection reset by peer
[7525] 22 Apr 20:19:12.553 # Connection with master lost.
[7525] 22 Apr 20:19:12.553 * Caching the disconnected master state.
[7525] 22 Apr 20:19:13.177 * Connecting to MASTER 192.168.1.100:6379
[7525] 22 Apr 20:19:13.178 * MASTER <-> SLAVE sync started
[7525] 22 Apr 20:19:13.179 # Error condition on socket for SYNC: Connection refused
[7525] 22 Apr 20:19:14.185 * Connecting to MASTER 192.168.1.100:6379
[7525] 22 Apr 20:19:14.186 * MASTER <-> SLAVE sync started
[7525] 22 Apr 20:19:14.187 # Error condition on socket for SYNC: Connection refused
[7525] 22 Apr 20:19:15.199 - DB 0: 1129079 keys (0 volatile) in 2097152 slots HT.
[7525] 22 Apr 20:19:15.199 - 3 clients connected (0 slaves), 126039272 bytes in use
[7525] 22 Apr 20:19:15.199 * Connecting to MASTER 192.168.1.100:6379

# Sentinel에서 Failover 발생시 Master, Slave 변경후 sentinel.conf 파일
# sentinel.conf
sentinel monitor mymaster 192.168.1.101 6379 1 -- 기존 슬레이브를 마스터로 인식하고 모니터링
sentinel auth-pass mymaster 1234
sentinel config-epoch mymaster 1
sentinel leader-epoch mymaster 1
sentinel known-slave mymaster 192.168.1.100 6379  -- 기존 마스터를 슬레이브로 인식함.
sentinel known-slave mymaster 192.168.1.102 6379  -- 슬레이브 정의 형식

# Generated by CONFIG REWRITE
port 26379
dir "/home/redis/redis-2.8.8"
sentinel current-epoch 1

# Master 서버의 sentinel 에서 Slave 서버를 Master 서버로 승격 처리

> Sentinel 로그

[2164] 24 Apr 11:06:14.379 # Sentinel runid is f8916b3a81a851478f9e6560d827dfa5af40c785
[2164] 24 Apr 11:06:14.379 # +monitor master mymaster 192.168.1.100 6379 quorum 1
[2164] 24 Apr 11:06:15.382 * +slave slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.100 6379

> Failover 감지
[2164] 24 Apr 11:13:28.788 # +sdown master mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:28.789 # +odown master mymaster 192.168.1.100 6379 #quorum 1/1

[2164] 24 Apr 11:13:28.789 # +new-epoch 1
[2164] 24 Apr 11:13:28.789 # +try-failover master mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:28.795 # +vote-for-leader f8916b3a81a851478f9e6560d827dfa5af40c785 1
[2164] 24 Apr 11:13:28.796 # +elected-leader master mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:28.796 # +failover-state-select-slave master mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:28.850 # +selected-slave slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:28.850 * +failover-state-send-slaveof-noone slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:28.951 * +failover-state-wait-promotion slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:29.853 # +promoted-slave slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:29.853 # +failover-state-reconf-slaves master mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:29.899 # +failover-end master mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:29.899 # +switch-master mymaster 192.168.1.100 6379 192.168.1.101 6379
[2164] 24 Apr 11:13:29.900 * +slave slave 192.168.1.100:6379 192.168.1.100 6379 @ mymaster 192.168.1.101 6379

[2164] 24 Apr 11:13:52.040 # +tilt #tilt mode entered
[2164] 24 Apr 11:14:22.082 # -tilt #tilt mode exited
[2164] 24 Apr 11:14:29.962 # +sdown slave 192.168.1.100:6379 192.168.1.100 6379 @ mymaster 192.168.1.101 6379

> Slave 서버 로그(Master 승격 과정)
[7891] 22 Apr 21:32:54.096 * MASTER <-> SLAVE sync started
[7891] 22 Apr 21:32:54.096 # Error condition on socket for SYNC: Connection refused
[7891] 22 Apr 21:32:54.200 * Discarding previously cached master state.
[7891] 22 Apr 21:32:54.200 * MASTER MODE enabled (user request)

-- redis.conf 파일 변경 : slaveof 마스터 정의 라인 제거
[7891] 22 Apr 21:32:54.203 # CONFIG REWRITE executed with success.
[7891] 22 Apr 21:32:54.835 - Client closed connection
[7891] 22 Apr 21:32:54.836 - Client closed connection
[7891] 22 Apr 21:32:54.913 - Accepted 192.168.1.100:37044
[7891] 22 Apr 21:32:54.914 - Accepted 192.168.1.100:37045
[7891] 22 Apr 21:32:59.151 - DB 0: 1129079 keys (0 volatile) in 2097152 slots HT.
[7891] 22 Apr 21:32:59.151 - 5 clients connected (0 slaves), 126060408 bytes in use

# 중지된 Master 서버 실행
> 기존의 다운 되었던 Master 서버가 Slave 서버로 변경되어짐

> Sentinel 로그
[10012] 22 Apr 21:41:49.765 # -sdown slave 192.168.1.100:6379 192.168.1.100 6379 @ mymaster 192.168.1.101 6379
[10012] 22 Apr 21:41:59.800 * +convert-to-slave slave 192.168.1.100:6379 192.168.1.100 6379 @ mymaster 192.168.1.101 6379
[10012] 22 Apr 21:42:05.232 * +slave slave 192.168.1.100:6379 192.168.1.100 6379 @ resque 192.168.1.101 6379

# 기존 Slave 서버 로그

[7891] 22 Apr 21:41:59.794 - DB 0: 1129079 keys (0 volatile) in 2097152 slots HT.
[7891] 22 Apr 21:41:59.794 - 5 clients connected (0 slaves), 126023584 bytes in use
[7891] 22 Apr 21:42:00.569 - Accepted 192.168.1.100:51686
[7891] 22 Apr 21:42:00.575 * Slave asks for synchronization
[7891] 22 Apr 21:42:00.576 * Full resync requested by slave.
[7891] 22 Apr 21:42:00.576 * Starting BGSAVE for SYNC
[7891] 22 Apr 21:42:00.580 * Background saving started by pid 7959
[7959] 22 Apr 21:42:01.461 * DB saved on disk
[7959] 22 Apr 21:42:01.461 * RDB: 24 MB of memory used by copy-on-write
[7891] 22 Apr 21:42:01.508 * Background saving terminated with success
[7891] 22 Apr 21:42:02.172 * Synchronization with slave succeeded
[7891] 22 Apr 21:42:04.821 - DB 0: 1129079 keys (0 volatile) in 2097152 slots HT.
[7891] 22 Apr 21:42:04.821 - 5 clients connected (1 slaves), 127093248 bytes in use

# 기존 Master 서버 로그

[10043] 22 Apr 21:41:49.469 # Server started, Redis version 2.8.8
[10043] 22 Apr 21:41:49.566 - Accepted 192.168.1.100:39891
[10043] 22 Apr 21:41:49.663 - Accepted 192.168.1.100:39892
[10043] 22 Apr 21:41:50.697 * DB loaded from disk: 1.228 seconds
[10043] 22 Apr 21:41:50.697 * The server is now ready to accept connections on port 6379
[10043] 22 Apr 21:41:50.773 - DB 0: 1129079 keys (0 volatile) in 3145728 slots HT.
[10043] 22 Apr 21:41:50.774 - 2 clients connected (0 slaves), 134423864 bytes in use
[10043] 22 Apr 21:41:55.903 - DB 0: 1129079 keys (0 volatile) in 3145728 slots HT.
[10043] 22 Apr 21:41:55.903 - 2 clients connected (0 slaves), 134350168 bytes in use
[10043] 22 Apr 21:41:59.801 * SLAVE OF 192.168.1.101:6379 enabled (user request)

-- redis.conf 파일 변경 : slaveof 마스터 정보 추가
[10043] 22 Apr 21:41:59.805 # CONFIG REWRITE executed with success.
[10043] 22 Apr 21:42:00.014 * Connecting to MASTER 192.168.1.101:6379
[10043] 22 Apr 21:42:00.017 * MASTER <-> SLAVE sync started
[10043] 22 Apr 21:42:00.018 * Non blocking connect for SYNC fired the event.
[10043] 22 Apr 21:42:00.022 * Master replied to PING, replication can continue...
[10043] 22 Apr 21:42:00.023 * Partial resynchronization not possible (no cached master)
[10043] 22 Apr 21:42:00.027 * Full resync from master: c223ac0110724a82744d44ea078bbb0971a493f1:1
[10043] 22 Apr 21:42:00.958 * MASTER <-> SLAVE sync: receiving 24839757 bytes from master
[10043] 22 Apr 21:42:01.267 - DB 0: 1129079 keys (0 volatile) in 3145728 slots HT.
[10043] 22 Apr 21:42:01.267 - 2 clients connected (0 slaves), 134387080 bytes in use
[10043] 22 Apr 21:42:01.634 * MASTER <-> SLAVE sync: Flushing old data
[10043] 22 Apr 21:42:02.386 * MASTER <-> SLAVE sync: Loading DB in memory
[10043] 22 Apr 21:42:03.525 * MASTER <-> SLAVE sync: Finished with success
[10043] 22 Apr 21:42:05.287 - Accepted 192.168.1.100:39894
[10043] 22 Apr 21:42:05.287 - Accepted 192.168.1.100:39895
[10043] 22 Apr 21:42:08.234 - DB 0: 1129079 keys (0 volatile) in 3145728 slots HT.
[10043] 22 Apr 21:42:08.234 - 5 clients connected (0 slaves), 134412080 bytes in use

위의 Failover 테스트와 같이 Master 노드에 장애가 발생하면 센티널에서 슬레이브를 마스터로 변경하는 작업이 반복됨.


※ 위 내용은, 여러 자료를 참고하거나 제가 주관적으로 정리한 것입니다.
   잘못된 정보나 보완이 필요한 부분을, 댓글 또는 메일로 보내주시면 많은 도움이 되겠습니다.
04 23, 2014 19:00 04 23, 2014 19:00


Trackback URL : http://develop.sunshiny.co.kr/trackback/1006

Leave a comment


Recent Posts

  1. HDFS - Python Encoding 오류 처리
  2. HP - Vertica ROS Container 관련 오류...
  3. HDFS - Hive 실행시 System Time 오류
  4. HP - Vertica 사용자 쿼리 이력 테이블...
  5. Client에서 HDFS 환경의 데이터 처리시...

Recent Comments

  1. 안녕하세요^^ 배그핵
  2. 안녕하세요^^ 도움이 되셨다니, 저... sunshiny
  3. 정말 큰 도움이 되었습니다.. 감사합... 사랑은
  4. 네, 안녕하세요. 댓글 남겨 주셔서... sunshiny
  5. 감사합니다 많은 도움 되었습니다!ㅎㅎ 프리시퀸스

Recent Trackbacks

  1. Mysql - mysql 설치후 Character set... 멀고 가까움이 다르기 때문 %M

Calendar

«   10 2019   »
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    

Bookmarks

  1. 위키피디아
  2. MysqlKorea
  3. 오라클 클럽
  4. API - Java
  5. Apache Hadoop API
  6. Apache Software Foundation
  7. HDFS 생태계 솔루션
  8. DNSBL - Spam Database Lookup
  9. Ready System
  10. Solaris Freeware
  11. Linux-Site
  12. 윈디하나의 솔라나라

Site Stats

TOTAL 2724045 HIT
TODAY 511 HIT
YESTERDAY 589 HIT