网站网站营销特点网站建设实践收获

1、topic主题操作的命令

kafka发送消息会存到主题中
消费者会从主题中获取消息消费
一个topic可能拆分成多个分区（partition）
一个分区可以有多个副本（replicas：负责接收数据的分区副本为leader，其他的为follower）副本数量不能超过broker数量

1.1、创建一个3分区1副本的主题

1.1.1、获取 kafka-topics.sh 的帮助信息

[root@localhost ~]# pwd
/root
[root@localhost ~]# kafka-topics.sh --help
This tool helps to create, delete, describe, or change a topic.
Option                                   Description                            
------                                   -----------                            
--alter                                  Alter the number of partitions,        replica assignment, and/or           configuration for the topic.         
--at-min-isr-partitions                  if set when describing topics, only    show partitions whose isr count is   equal to the configured minimum.     
--bootstrap-server <String: server to    REQUIRED: The Kafka server to connect  connect to>                              to.                                  
--command-config <String: command        Property file containing configs to be config property file>                    passed to Admin Client. This is used only with --bootstrap-server option  for describing and altering broker   configs.                             
--config <String: name=value>            A topic configuration override for the topic being created or altered. The  following is a list of valid         configurations:                      cleanup.policy                        compression.type                      delete.retention.ms                   file.delete.delay.ms                  flush.messages                        flush.ms                              follower.replication.throttled.       replicas                             index.interval.bytes                  leader.replication.throttled.replicas local.retention.bytes                 local.retention.ms                    max.compaction.lag.ms                 max.message.bytes                     message.downconversion.enable         message.format.version                message.timestamp.difference.max.ms   message.timestamp.type                min.cleanable.dirty.ratio             min.compaction.lag.ms                 min.insync.replicas                   preallocate                           remote.storage.enable                 retention.bytes                       retention.ms                          segment.bytes                         segment.index.bytes                   segment.jitter.ms                     segment.ms                            unclean.leader.election.enable        See the Kafka documentation for full   details on the topic configs. It is  supported only in combination with --create if --bootstrap-server option  is used (the kafka-configs CLI       supports altering topic configs with a --bootstrap-server option).        
--create                                 Create a new topic.                    
--delete                                 Delete a topic                         
--delete-config <String: name>           A topic configuration override to be   removed for an existing topic (see   the list of configurations under the --config option). Not supported with the --bootstrap-server option.       
--describe                               List details for the given topics.     
--disable-rack-aware                     Disable rack aware replica assignment  
--exclude-internal                       exclude internal topics when running   list or describe command. The        internal topics will be listed by    default                              
--help                                   Print usage information.               
--if-exists                              if set when altering or deleting or    describing topics, the action will   only execute if the topic exists.    
--if-not-exists                          if set when creating topics, the       action will only execute if the      topic does not already exist.        
--list                                   List all available topics.             
--partitions <Integer: # of partitions>  The number of partitions for the topic being created or altered (WARNING:   If partitions are increased for a    topic that has a key, the partition  logic or ordering of the messages    will be affected). If not supplied   for create, defaults to the cluster  default.                             
--replica-assignment <String:            A list of manual partition-to-broker   broker_id_for_part1_replica1 :           assignments for the topic being      broker_id_for_part1_replica2 ,           created or altered.                  broker_id_for_part2_replica1 :                                                broker_id_for_part2_replica2 , ...>                                           
--replication-factor <Integer:           The replication factor for each        replication factor>                      partition in the topic being         created. If not supplied, defaults   to the cluster default.              
--topic <String: topic>                  The topic to create, alter, describe   or delete. It also accepts a regular expression, except for --create      option. Put topic name in double     quotes and use the '\' prefix to     escape regular expression symbols; e.g. "test\.topic".                    
--topics-with-overrides                  if set when describing topics, only    show topics that have overridden     configs                              
--unavailable-partitions                 if set when describing topics, only    show partitions whose leader is not  available                            
--under-min-isr-partitions               if set when describing topics, only    show partitions whose isr count is   less than the configured minimum.    
--under-replicated-partitions            if set when describing topics, only    show under replicated partitions     
--version                                Display Kafka version.                 
[root@localhost ~]# kafka-topics.sh --help
This tool helps to create, delete, describe, or change a topic.
Option                                   Description                            
------                                   -----------                            
--alter                                  Alter the number of partitions,        replica assignment, and/or           configuration for the topic.         
--at-min-isr-partitions                  if set when describing topics, only    show partitions whose isr count is   equal to the configured minimum.     
--bootstrap-server <String: server to    REQUIRED: The Kafka server to connect  connect to>                              to.                                  
--command-config <String: command        Property file containing configs to be config property file>                    passed to Admin Client. This is used only with --bootstrap-server option  for describing and altering broker   configs.                             
--config <String: name=value>            A topic configuration override for the topic being created or altered. The  following is a list of valid         configurations:                      cleanup.policy                        compression.type                      delete.retention.ms                   file.delete.delay.ms                  flush.messages                        flush.ms                              follower.replication.throttled.       replicas                             index.interval.bytes                  leader.replication.throttled.replicas local.retention.bytes                 local.retention.ms                    max.compaction.lag.ms                 max.message.bytes                     message.downconversion.enable         message.format.version                message.timestamp.difference.max.ms   message.timestamp.type                min.cleanable.dirty.ratio             min.compaction.lag.ms                 min.insync.replicas                   preallocate                           remote.storage.enable                 retention.bytes                       retention.ms                          segment.bytes                         segment.index.bytes                   segment.jitter.ms                     segment.ms                            unclean.leader.election.enable        See the Kafka documentation for full   details on the topic configs. It is  supported only in combination with --create if --bootstrap-server option  is used (the kafka-configs CLI       supports altering topic configs with a --bootstrap-server option).        
--create                                 Create a new topic.                    
--delete                                 Delete a topic                         
--delete-config <String: name>           A topic configuration override to be   removed for an existing topic (see   the list of configurations under the --config option). Not supported with the --bootstrap-server option.       
--describe                               List details for the given topics.     
--disable-rack-aware                     Disable rack aware replica assignment  
--exclude-internal                       exclude internal topics when running   list or describe command. The        internal topics will be listed by    default                              
--help                                   Print usage information.               
--if-exists                              if set when altering or deleting or    describing topics, the action will   only execute if the topic exists.    
--if-not-exists                          if set when creating topics, the       action will only execute if the      topic does not already exist.        
--list                                   List all available topics.             
--partitions <Integer: # of partitions>  The number of partitions for the topic being created or altered (WARNING:   If partitions are increased for a    topic that has a key, the partition  logic or ordering of the messages    will be affected). If not supplied   for create, defaults to the cluster  default.                             
--replica-assignment <String:            A list of manual partition-to-broker   broker_id_for_part1_replica1 :           assignments for the topic being      broker_id_for_part1_replica2 ,           created or altered.                  broker_id_for_part2_replica1 :                                                broker_id_for_part2_replica2 , ...>                                           
--replication-factor <Integer:           The replication factor for each        replication factor>                      partition in the topic being         created. If not supplied, defaults   to the cluster default.              
--topic <String: topic>                  The topic to create, alter, describe   or delete. It also accepts a regular expression, except for --create      option. Put topic name in double     quotes and use the '\' prefix to     escape regular expression symbols; e.g. "test\.topic".                    
--topics-with-overrides                  if set when describing topics, only    show topics that have overridden     configs                              
--unavailable-partitions                 if set when describing topics, only    show partitions whose leader is not  available                            
--under-min-isr-partitions               if set when describing topics, only    show partitions whose isr count is   less than the configured minimum.    
--under-replicated-partitions            if set when describing topics, only    show under replicated partitions     
--version                                Display Kafka version. 

1.1.2、副本因子设置不能超过集群中broker的数量

[root@localhost ~]# kafka-topics.sh --bootstrap-server 192.168.74.148:9092 --create --topic "my_topic1" --partitions 3 --replication-factor 2
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Error while executing topic command : Replication factor: 2 larger than available brokers: 1.
[2024-05-28 10:23:51,616] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 2 larger than available brokers: 1.(kafka.admin.TopicCommand$)

1.1.3、创建一个3分区1副本的主题

[root@localhost ~]# kafka-topics.sh --bootstrap-server 192.168.74.148:9092 --create --topic "my_topic1" --partitions 3 --replication-factor 1
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic my_topic1.

1.1.4、查看所有主题

[root@localhost ~]# kafka-topics.sh --bootstrap-server 192.168.74.148:9092 --list
my_topic1

1.1.5、查看主题详细描述

[root@localhost ~]# kafka-topics.sh --bootstrap-server 192.168.74.148:9092 --describe --topic my_topic1
Topic: my_topic1        TopicId: 9vShOJh5QPapMxCd-76iXg PartitionCount: 3       ReplicationFactor: 1    Configs: segment.bytes=1073741824Topic: my_topic1        Partition: 0    Leader: 0       Replicas: 0     Isr: 0Topic: my_topic1        Partition: 1    Leader: 0       Replicas: 0     Isr: 0Topic: my_topic1        Partition: 2    Leader: 0       Replicas: 0     Isr: 0