HDFS - Hive 실행시 System Time 오류

Posted 07 26, 2016 14:53, Filed under: BigData/Hadoop


Linux에서 NTP(Network Time Protocol) 서비스가 실행 되면서 시스템 시간을 주기적으로 동기화 합니다.
NTP(ntpd 데몬)가 실행중이더라도 외부의 TimeServer에 접속이 되질 않는다면, 어느정도 시기가 지나면 서버의 시간이 느려지는 증상이 있습니다.

HDFS 컨테이너에서 요청 허용 시간의 범위를 체크하는데, 시간이 지나면서 각 서버의 시간차가 커지다 보면 작업을 실패합니다.


관련 오류
2016-07-22 17:04:44,437 ERROR [Thread-712]: SessionState (SessionState.java:printError(833)) - Status: Failed
2016-07-22 17:04:44,438 ERROR [Thread-712]: SessionState (SessionState.java:printError(833)) - Vertex failed, vertexName=Map 6, vertexId=vertex_1468916542684_0021_1_00, diagnostics=[Task failed, taskId=task_1468916542684_0021_1_00_000007, diagnostics=[TaskAttempt 0 failed, info=[Container launch failed for container_e54_1468916542684_0021_02_000002 : org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
This token is expired. current time is 1469175137466 found 1469175128093
Note: System times on machines may be out of sync. Check system time and time zones.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:168)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:380)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 1 failed, info=[Container launch failed for container_e54_1468916542684_0021_02_000015 : org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
This token is expired. current time is 1469175138445 found 1469175128918
Note: System times on machines may be out of sync. Check system time and time zones.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:168)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:380)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 2 failed, info=[Container launch failed for container_e54_1468916542684_0021_02_000017 : org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
This token is expired. current time is 1469175139258 found 1469175129970
Note: System times on machines may be out of sync. Check system time and time zones.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:168)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:380)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
], TaskAttempt 3 failed, info=[Container launch failed for container_e54_1468916542684_0021_02_000020 : org.apache.hadoop.yarn.exceptions.YarnException: Unauthorized request to start container.
This token is expired. current time is 1469175140315 found 1469175131018
Note: System times on machines may be out of sync. Check system time and time zones.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:168)
at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:380)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
]], Vertex failed as one or more tasks failed. failedTasks:1, Vertex vertex_1468916542684_0021_1_00 [Map 6] killed/failed due to:null]
2016-07-22 17:04:44,438 ERROR [Thread-712]: SessionState (SessionState.java:printError(833)) - Vertex killed, vertexName=Reducer 3, vertexId=vertex_1468916542684_0021_1_04, diagnostics=[Vertex received Kill while in RUNNING state., Vertex killed as other vertex failed. failedTasks:0, Vertex vertex_1468916542684_0021_1_04 [Reducer 3] killed/failed due to:null]
2016-07-22 17:04:44,439 ERROR [Thread-712]: SessionState (SessionState.java:printError(833)) - DAG failed due to vertex failure. failedVertices:1 killedVertices:5


각 서버의 시간 정보를 확인후에 맞지 않을 경우 변경
 

# 시스템 현재 시간으로 수동 설정

> System Time 확인
[root@hdfs ~]# date
2016. 07. 25. (월) 21:57:52 KST
 
> 시분초 설정 (24시간제로 입력)
[root@hdfs ~]# date -s 23:43:21
 
> 연월일 시분초 바꾸기
[root@hdfs ~]# date -s '2016-7-26 11:21:21'


네트워크 내부에 NTP Server가 셋팅 되어 있다면, 그리고 HDFS 클러스터의 각 서버가 접근 가능하다면,
아래의 TimeServer(time.bora.net) 설정과 같이 내부 NTP 서버로 등록 가능.

# NTP 정보 확인

> ntpd 서비스 구동 여부 확인
[root@hdfs ~]# service ntpd status
ntpd가 정지되었습니다.
 
> ntpstat 명령을 사용하여 NTP 서비스의 상태를 확인
[root@hdfs ~]# ntpstat
synchronised to NTP server (211.233.40.78) at stratum 3
   time correct to within 163 ms
   polling server every 1024 s
 
> 작동 되지 않는 상태
[root@www ~]# ntpstat
Unable to talk to NTP daemon. Is it running?
 
> Time Server의 시간 조회
[root@hdfs ~]# rdate -p time.bora.net
rdate: [time.bora.net]  Mon Jul 25 21:10:57 2016
 

# 시스템 시간 동기화

> Time Server의 시간 동기화 설정(일시적)
[root@hdfs ~]# rdate -s time.bora.net
[root@hdfs ~]# date
2016. 07. 25. (월) 21:44:19 KST
 
> Time Server 시간 주기적 동기화 ( 매일 24시에 동기화 )
[root@hdfs ~]# crontab -e
0 0 * * * rdate -s time.bora.net






※ 위 내용은, 여러 자료를 참고하거나 제가 주관적으로 정리한 것입니다.
   잘못된 정보나 보완이 필요한 부분을, 댓글 또는 메일로 보내주시면 많은 도움이 되겠습니다.
07 26, 2016 14:53 07 26, 2016 14:53


Trackback URL : http://develop.sunshiny.co.kr/trackback/1062

Leave a comment

« Previous : 1 : 2 : 3 : 4 : 5 : 6 : 7 : ... 648 : Next »

Recent Posts

  1. HDFS - Python Encoding 오류 처리
  2. HP - Vertica ROS Container 관련 오류...
  3. HDFS - Hive 실행시 System Time 오류
  4. HP - Vertica 사용자 쿼리 이력 테이블...
  5. Client에서 HDFS 환경의 데이터 처리시...

Recent Comments

  1. 안녕하세요^^ 배그핵
  2. 안녕하세요^^ 도움이 되셨다니, 저... sunshiny
  3. 정말 큰 도움이 되었습니다.. 감사합... 사랑은
  4. 네, 안녕하세요. 댓글 남겨 주셔서... sunshiny
  5. 감사합니다 많은 도움 되었습니다!ㅎㅎ 프리시퀸스

Recent Trackbacks

  1. Mysql - mysql 설치후 Character set... 멀고 가까움이 다르기 때문 %M

Calendar

«   10 2019   »
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    

Bookmarks

  1. 위키피디아
  2. MysqlKorea
  3. 오라클 클럽
  4. API - Java
  5. Apache Hadoop API
  6. Apache Software Foundation
  7. HDFS 생태계 솔루션
  8. DNSBL - Spam Database Lookup
  9. Ready System
  10. Solaris Freeware
  11. Linux-Site
  12. 윈디하나의 솔라나라

Site Stats

TOTAL 2724044 HIT
TODAY 510 HIT
YESTERDAY 589 HIT