Kaf 学习摘要

Summary: Author: 张亚飞 | Read Time: 2 minute read | Published: 2020-04-30
Filed under — Categories: Linux — Tags: Note,

Kaf

go install github.com/birdayz/kaf/cmd/kaf@latest

kaf --version
kaf config add-cluster brtc-test -b test-external-kafka-cluster01.baijiayun.com:19092
kaf config add-cluster brtc-online -b internal-kafka-cluster01.baijiayun.com:9092
kaf config add-cluster gcms-aliyun -b 182.92.104.139:9093,182.92.77.7:9093,123.56.133.237:9093
kaf config add-cluster brtc-logs -b 172.17.10.31:9092,172.17.10.32:9092,172.17.10.33:9092
kaf config get-clusters
kaf config select-cluster
kaf config remove-cluster
kaf node ls

kaf config add-cluster brtc-cluster01 -b 10.16.30.14:9092,10.16.30.15:9093,10.16.30.16:9094
kaf config add-cluster brtc-cluster02 -b internal-kafka-cluster02.baijiayun.com:9092
kaf config add-cluster brtc-cluster03 -b internal-kafka-cluster03.baijiayun.com:9092

kaf topic create rtc-message-test -p 32 -r 2
kaf topic create vcs-sn-event-test -p 3 -r 2
kaf topic create vcs-sn-change-test -p 3 -r 2
kaf topic create collection-test -p 4 -r 2
kaf topic create collection-trtc-test -p 4 -r 2
kaf topic create cdn-pull-test -p 3 -r 2
kaf topic create test-vloud-sdk-log -p 12 -r 2
kaf topic create test-vcollections-player-report -p 3 -r 2
kaf topic create test-vcollections-blive-report -p 3 -r 2

BRTC测试环境kafka服务迁移

旧 kafka 集群

172.17.1.18 172.17.1.19 172.17.1.20

新 kafka 集群

172.17.0.39 172.17.0.40 172.17.0.41

常用命令

重置消费位移

$ kaf group commit consumer_group -t=some_topic -p=0 -o=137638
$ kaf group describe consumer_group

commit not work

重置偏移量

kaf group commit logstash -t=rtc-message-test -p=0 -o=137638

重置特定消费组的消费位移

kaf group commit logstash -t rtc-message-test -o latest --all-partitions
kaf group commit vrc-test -t rtc-message-test -o latest --all-partitions

超赞的Kafka命令行

kaf topic create rtc-message-dev -p 32

100w-500w=283s ==> 80w/min

kaf group describe group-zyf-kafka-msms-stats
kaf consume topic-msms-stream-stat | kaf produce topic-msms-stream-stat

0 7179299464 2 [2] [2] 1 10540846 0 [0] [0] 2 4612190 1 [1] [1] 3 10571529 2 [2] [2] 4 10540744 0 [0] [0] 5 4612108 1 [1] [1] 6 10571393 2 [2] [2]

kaf topic describe topic-msms-stream-stat-online
kaf consume topic-msms-stream-stat-online --raw -p=0 -l=100000 >> 0.log
kaf consume topic-msms-stream-stat-online --raw -p=1 -l=100000 >> 1.log
kaf consume topic-msms-stream-stat-online --raw -p=2 -l=100000 >> 2.log
kaf consume topic-msms-stream-stat-online --raw -p=3 -l=100000 >> 3.log
kaf consume topic-msms-stream-stat-online --raw -p=4 -l=100000 >> 4.log
kaf consume topic-msms-stream-stat-online --raw -p=5 -l=100000 >> 5.log
kaf consume topic-msms-stream-stat-online --raw -p=6 -l=100000 >> 6.log

kaf topic describe topic-msms-stream-stat-online
cat 0.log | kaf produce topic-msms-stream-stat -p=0
cat 1.log | kaf produce topic-msms-stream-stat -p=1
cat 2.log | kaf produce topic-msms-stream-stat -p=2
cat 3.log | kaf produce topic-msms-stream-stat -p=3
cat 4.log | kaf produce topic-msms-stream-stat -p=4
cat 5.log | kaf produce topic-msms-stream-stat -p=5
cat 6.log | kaf produce topic-msms-stream-stat -p=6

cat 00.log | kaf produce topic-msms-stream-stat -p=0

Cor Ethan