# Redis(레디스) Sentinel을 이용한 HA(High Availability) 구성


사용자 삽입 이미지

Redis - Master, Slave(복제) 구성

참고 : http://redis.io/topics/sentinel

# Sentinel 옵션
monitor : 모니터링할 master 아이피 및 odown 기준을 위한 sentinel quorum(의사결정) 수 설정
              - quorum(의사결정)의 수만큼 장애감지에 따라 Objective down(Failover) 진행
down-after-milliseconds : 장애 감지 이후 몇초 후 sdown으로 전이 될지 설정
failover-timeout : timeout 시간내 Failover 완료시, 나머지 slave에게 새로운 마스터 설정 가능 하도록 command를 보냄
parallel-syncs : 새로 승격된 master와 syncs를 맞출 slave 갯수 설정
auth-pass : master 패스워드


# Redis Failover 판단 과정
* Subjective down (sdown) : 주관적 장애 감지
  - 하나의 Sentinel에서만 장애를 감지 한 상태


* Objective down (odown) : 객관적 장애 감지, Failover(장애극복) 진행
  - 여러 Sentinel(quorum의 수)에서 장애라고 판단 한 경우


# sentinel.conf
센티널은 마스터에 접근해서 슬레이브 노드들의 정보를 찾고, 각 슬레이브에 접근함.
그러므로 센티널 설정은 마스터의 정보만 입력해주면 됨.

> Master
sentinel monitor mymaster 192.168.1.100 6379 1(quorum : 하나의 센티널에서 모니터링)
sentinel down-after-milliseconds mymaster 60000  -- ms단위 : 현재 60초
sentinel failover-timeout mymaster 180000
sentinel parallel-syncs mymaster 1
sentinel auth-pass mymaster 1234  -- Master 노드 접근시 패스워드 필요

# Sentinel 실행
[redis@node01 redis-2.8.8]$ src/redis-sentinel ./sentinel.conf >> log/sentinel.log &

> Sentinel 로그
[9647] 22 Apr 20:17:07.146 # Sentinel runid is 5298af98c048201d21babae6646d8939231a31f8
[9647] 22 Apr 20:17:07.146 # +monitor master mymaster 192.168.1.100 6379 quorum 1
[9647] 22 Apr 20:17:08.150 * +slave slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.100 6379


# Sentinel Failover 테스트


# Sentinel 실행후 Master 서버 종료

> Slave 서버 로그

[7525] 22 Apr 20:19:12.553 - Reading from client: Connection reset by peer
[7525] 22 Apr 20:19:12.553 # Connection with master lost.
[7525] 22 Apr 20:19:12.553 * Caching the disconnected master state.
[7525] 22 Apr 20:19:13.177 * Connecting to MASTER 192.168.1.100:6379
[7525] 22 Apr 20:19:13.178 * MASTER <-> SLAVE sync started
[7525] 22 Apr 20:19:13.179 # Error condition on socket for SYNC: Connection refused
[7525] 22 Apr 20:19:14.185 * Connecting to MASTER 192.168.1.100:6379
[7525] 22 Apr 20:19:14.186 * MASTER <-> SLAVE sync started
[7525] 22 Apr 20:19:14.187 # Error condition on socket for SYNC: Connection refused
[7525] 22 Apr 20:19:15.199 - DB 0: 1129079 keys (0 volatile) in 2097152 slots HT.
[7525] 22 Apr 20:19:15.199 - 3 clients connected (0 slaves), 126039272 bytes in use
[7525] 22 Apr 20:19:15.199 * Connecting to MASTER 192.168.1.100:6379

# Sentinel에서 Failover 발생시 Master, Slave 변경후 sentinel.conf 파일
# sentinel.conf
sentinel monitor mymaster 192.168.1.101 6379 1 -- 기존 슬레이브를 마스터로 인식하고 모니터링
sentinel auth-pass mymaster 1234
sentinel config-epoch mymaster 1
sentinel leader-epoch mymaster 1
sentinel known-slave mymaster 192.168.1.100 6379  -- 기존 마스터를 슬레이브로 인식함.
sentinel known-slave mymaster 192.168.1.102 6379  -- 슬레이브 정의 형식

# Generated by CONFIG REWRITE
port 26379
dir "/home/redis/redis-2.8.8"
sentinel current-epoch 1

# Master 서버의 sentinel 에서 Slave 서버를 Master 서버로 승격 처리

> Sentinel 로그

[2164] 24 Apr 11:06:14.379 # Sentinel runid is f8916b3a81a851478f9e6560d827dfa5af40c785
[2164] 24 Apr 11:06:14.379 # +monitor master mymaster 192.168.1.100 6379 quorum 1
[2164] 24 Apr 11:06:15.382 * +slave slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.100 6379

> Failover 감지
[2164] 24 Apr 11:13:28.788 # +sdown master mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:28.789 # +odown master mymaster 192.168.1.100 6379 #quorum 1/1

[2164] 24 Apr 11:13:28.789 # +new-epoch 1
[2164] 24 Apr 11:13:28.789 # +try-failover master mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:28.795 # +vote-for-leader f8916b3a81a851478f9e6560d827dfa5af40c785 1
[2164] 24 Apr 11:13:28.796 # +elected-leader master mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:28.796 # +failover-state-select-slave master mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:28.850 # +selected-slave slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:28.850 * +failover-state-send-slaveof-noone slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:28.951 * +failover-state-wait-promotion slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:29.853 # +promoted-slave slave 192.168.1.101:6379 192.168.1.101 6379 @ mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:29.853 # +failover-state-reconf-slaves master mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:29.899 # +failover-end master mymaster 192.168.1.100 6379
[2164] 24 Apr 11:13:29.899 # +switch-master mymaster 192.168.1.100 6379 192.168.1.101 6379
[2164] 24 Apr 11:13:29.900 * +slave slave 192.168.1.100:6379 192.168.1.100 6379 @ mymaster 192.168.1.101 6379

[2164] 24 Apr 11:13:52.040 # +tilt #tilt mode entered
[2164] 24 Apr 11:14:22.082 # -tilt #tilt mode exited
[2164] 24 Apr 11:14:29.962 # +sdown slave 192.168.1.100:6379 192.168.1.100 6379 @ mymaster 192.168.1.101 6379

> Slave 서버 로그(Master 승격 과정)
[7891] 22 Apr 21:32:54.096 * MASTER <-> SLAVE sync started
[7891] 22 Apr 21:32:54.096 # Error condition on socket for SYNC: Connection refused
[7891] 22 Apr 21:32:54.200 * Discarding previously cached master state.
[7891] 22 Apr 21:32:54.200 * MASTER MODE enabled (user request)

-- redis.conf 파일 변경 : slaveof 마스터 정의 라인 제거
[7891] 22 Apr 21:32:54.203 # CONFIG REWRITE executed with success.
[7891] 22 Apr 21:32:54.835 - Client closed connection
[7891] 22 Apr 21:32:54.836 - Client closed connection
[7891] 22 Apr 21:32:54.913 - Accepted 192.168.1.100:37044
[7891] 22 Apr 21:32:54.914 - Accepted 192.168.1.100:37045
[7891] 22 Apr 21:32:59.151 - DB 0: 1129079 keys (0 volatile) in 2097152 slots HT.
[7891] 22 Apr 21:32:59.151 - 5 clients connected (0 slaves), 126060408 bytes in use

# 중지된 Master 서버 실행
> 기존의 다운 되었던 Master 서버가 Slave 서버로 변경되어짐

> Sentinel 로그
[10012] 22 Apr 21:41:49.765 # -sdown slave 192.168.1.100:6379 192.168.1.100 6379 @ mymaster 192.168.1.101 6379
[10012] 22 Apr 21:41:59.800 * +convert-to-slave slave 192.168.1.100:6379 192.168.1.100 6379 @ mymaster 192.168.1.101 6379
[10012] 22 Apr 21:42:05.232 * +slave slave 192.168.1.100:6379 192.168.1.100 6379 @ resque 192.168.1.101 6379

# 기존 Slave 서버 로그

[7891] 22 Apr 21:41:59.794 - DB 0: 1129079 keys (0 volatile) in 2097152 slots HT.
[7891] 22 Apr 21:41:59.794 - 5 clients connected (0 slaves), 126023584 bytes in use
[7891] 22 Apr 21:42:00.569 - Accepted 192.168.1.100:51686
[7891] 22 Apr 21:42:00.575 * Slave asks for synchronization
[7891] 22 Apr 21:42:00.576 * Full resync requested by slave.
[7891] 22 Apr 21:42:00.576 * Starting BGSAVE for SYNC
[7891] 22 Apr 21:42:00.580 * Background saving started by pid 7959
[7959] 22 Apr 21:42:01.461 * DB saved on disk
[7959] 22 Apr 21:42:01.461 * RDB: 24 MB of memory used by copy-on-write
[7891] 22 Apr 21:42:01.508 * Background saving terminated with success
[7891] 22 Apr 21:42:02.172 * Synchronization with slave succeeded
[7891] 22 Apr 21:42:04.821 - DB 0: 1129079 keys (0 volatile) in 2097152 slots HT.
[7891] 22 Apr 21:42:04.821 - 5 clients connected (1 slaves), 127093248 bytes in use

# 기존 Master 서버 로그

[10043] 22 Apr 21:41:49.469 # Server started, Redis version 2.8.8
[10043] 22 Apr 21:41:49.566 - Accepted 192.168.1.100:39891
[10043] 22 Apr 21:41:49.663 - Accepted 192.168.1.100:39892
[10043] 22 Apr 21:41:50.697 * DB loaded from disk: 1.228 seconds
[10043] 22 Apr 21:41:50.697 * The server is now ready to accept connections on port 6379
[10043] 22 Apr 21:41:50.773 - DB 0: 1129079 keys (0 volatile) in 3145728 slots HT.
[10043] 22 Apr 21:41:50.774 - 2 clients connected (0 slaves), 134423864 bytes in use
[10043] 22 Apr 21:41:55.903 - DB 0: 1129079 keys (0 volatile) in 3145728 slots HT.
[10043] 22 Apr 21:41:55.903 - 2 clients connected (0 slaves), 134350168 bytes in use
[10043] 22 Apr 21:41:59.801 * SLAVE OF 192.168.1.101:6379 enabled (user request)

-- redis.conf 파일 변경 : slaveof 마스터 정보 추가
[10043] 22 Apr 21:41:59.805 # CONFIG REWRITE executed with success.
[10043] 22 Apr 21:42:00.014 * Connecting to MASTER 192.168.1.101:6379
[10043] 22 Apr 21:42:00.017 * MASTER <-> SLAVE sync started
[10043] 22 Apr 21:42:00.018 * Non blocking connect for SYNC fired the event.
[10043] 22 Apr 21:42:00.022 * Master replied to PING, replication can continue...
[10043] 22 Apr 21:42:00.023 * Partial resynchronization not possible (no cached master)
[10043] 22 Apr 21:42:00.027 * Full resync from master: c223ac0110724a82744d44ea078bbb0971a493f1:1
[10043] 22 Apr 21:42:00.958 * MASTER <-> SLAVE sync: receiving 24839757 bytes from master
[10043] 22 Apr 21:42:01.267 - DB 0: 1129079 keys (0 volatile) in 3145728 slots HT.
[10043] 22 Apr 21:42:01.267 - 2 clients connected (0 slaves), 134387080 bytes in use
[10043] 22 Apr 21:42:01.634 * MASTER <-> SLAVE sync: Flushing old data
[10043] 22 Apr 21:42:02.386 * MASTER <-> SLAVE sync: Loading DB in memory
[10043] 22 Apr 21:42:03.525 * MASTER <-> SLAVE sync: Finished with success
[10043] 22 Apr 21:42:05.287 - Accepted 192.168.1.100:39894
[10043] 22 Apr 21:42:05.287 - Accepted 192.168.1.100:39895
[10043] 22 Apr 21:42:08.234 - DB 0: 1129079 keys (0 volatile) in 3145728 slots HT.
[10043] 22 Apr 21:42:08.234 - 5 clients connected (0 slaves), 134412080 bytes in use

위의 Failover 테스트와 같이 Master 노드에 장애가 발생하면 센티널에서 슬레이브를 마스터로 변경하는 작업이 반복됨.


※ 위 내용은, 여러 자료를 참고하거나 제가 주관적으로 정리한 것입니다.
   잘못된 정보나 보완이 필요한 부분을, 댓글 또는 메일로 보내주시면 많은 도움이 되겠습니다.
04 23, 2014 19:00 04 23, 2014 19:00


Trackback URL : http://develop.sunshiny.co.kr/trackback/1006

Leave a comment
[로그인][오픈아이디란?]
오픈아이디로만 댓글을 남길 수 있습니다


Recent Posts

  1. HDFS - Python Encoding 오류 처리
  2. HP - Vertica ROS Container 관련 오류...
  3. HDFS - Hive 실행시 System Time 오류
  4. HP - Vertica 사용자 쿼리 이력 테이블...
  5. Client에서 HDFS 환경의 데이터 처리시...

Recent Comments

  1. 안녕하세요^^ 배그핵
  2. 안녕하세요^^ 도움이 되셨다니, 저... sunshiny
  3. 정말 큰 도움이 되었습니다.. 감사합... 사랑은
  4. 네, 안녕하세요. 댓글 남겨 주셔서... sunshiny
  5. 감사합니다 많은 도움 되었습니다!ㅎㅎ 프리시퀸스

Recent Trackbacks

  1. Roku App Development Roku App Development 3 07
  2. safety goggles safety goggles 2 07
  3. custom wheelchair for dogs custom wheelchair for dogs 30 06
  4. phoenix digital marketing agency phoenix digital marketing agency 29 06
  5. streaming tv channels streaming tv channels 28 06

Calendar

«   07 2020   »
      1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31  

Bookmarks

  1. 위키피디아
  2. MysqlKorea
  3. 오라클 클럽
  4. API - Java
  5. Apache Hadoop API
  6. Apache Software Foundation
  7. HDFS 생태계 솔루션
  8. DNSBL - Spam Database Lookup
  9. Ready System
  10. Solaris Freeware
  11. Linux-Site
  12. 윈디하나의 솔라나라

Site Stats

TOTAL 2998813 HIT
TODAY 106 HIT
YESTERDAY 905 HIT