[Version] Apache Kafka 3.4

https://www.confluent.io/blog/apache-kafka-3-4-0-new-features-and-updates/

Apache Kafka 3.4: New Features and Updates

 

Kafka Broker, Controller, Producer, Consumer and Admin Client

KIP-866: ZooKeeper to KRaft Migration (Early Access)

KIP-866 adds the capability to migrate a Kafka cluster from ZooKeeper to KRaft mode. The migration copies the cluster metadata from ZooKeeper to the KRaft metadata log. During the migration, brokers are restarted in KRaft mode one at a time allowing the whole migration process to happen without cluster downtime. ZooKeeper to KRaft migrations are Early Access in 3.4 and should not be used in production yet, though we are looking for people to provide feedback on this feature. Notably, support for migrating ACLs and downgrading to ZooKeeper mode during or after the migration is not supported in 3.4.

 

KIP-866: ZooKeeper에서 KRaft로 마이그레이션(초기 액세스)

KIP-866은 Kafka 클러스터를 ZooKeeper에서 KRaft 모드로 마이그레이션하는 기능을 추가합니다. 마이그레이션은 ZooKeeper에서 KRaft 메타데이터 로그로 클러스터 메타데이터를 복사합니다. 마이그레이션하는 동안 브로커는 KRaft 모드에서 한 번에 하나씩 다시 시작되어 클러스터 다운타임 없이 전체 마이그레이션 프로세스가 발생할 수 있습니다. ZooKeeper에서 KRaft로의 마이그레이션은 3.4의 얼리 액세스이며 아직 프로덕션에 사용해서는 안 됩니다. 하지만 이 기능에 대한 피드백을 제공할 사람을 찾고 있습니다. 특히 ACL 마이그레이션 및 마이그레이션 중 또는 마이그레이션 후 ZooKeeper 모드로 다운그레이드에 대한 지원은 3.4에서 지원되지 않습니다.

---

KIP-792: Add generation field into consumer protocol

KIP-792 expands the metadata each group member passes to the group leader in its JoinGroup subscription to include the highest stable generation that consumer was a part of. This information can be used by the partition assignor to determine whether the ownedPartitions claimed by a consumer are still owned and thus valid, and which consumer has the most recent claim to a given partition in case no one has a valid claim to it.

 

KIP-792: 소비자 프로토콜에 생성 필드 추가

KIP-792는 각 그룹 구성원이 JoinGroup 구독에서 그룹 리더에게 전달하는 메타데이터를 확장하여 소비자가 속한 가장 안정적인 세대를 포함합니다. 이 정보는 파티션 할당자가 소비자가 요청한ownedPartitions가 여전히 소유되어 유효한지 여부와 유효한 요청이 없는 경우 지정된 파티션에 대한 가장 최근의 요청을 가진 소비자를 결정하는 데 사용할 수 있습니다.

---

KIP-830: Allow disabling JMX Reporter

KIP-830 includes a new configuration setting that allows you to disable the JMX reporter for environments where it’s not being used. 

 

KIP-830: JMX Reporter 비활성화 허용

KIP-830에는 사용하지 않는 환경에서 JMX 리포터를 비활성화할 수 있는 새로운 구성 설정이 포함되어 있습니다.

---

KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer

KIP-840 provides a new config file option for configuring the MessageReader/Formatter of the console Consumer/Producer by introducing the new parameters --reader-config and --formatter-config.

 

KIP-840: ConsoleProducer/ConsoleConsumer의 MessageReader/MessageFormatter에 대한 구성 파일 옵션

KIP-840은 새 매개변수 --reader-config 및 --formatter-config를 도입하여 콘솔 Consumer/Producer의 MessageReader/Formatter를 구성하기 위한 새로운 구성 파일 옵션을 제공합니다.

---

KIP-854: Separate configuration for producer ID expiry

KIP-854 introduces changes to clean up producer IDs more efficiently, to avoid excess memory usage. It introduces a new timeout parameter, with a default of 1 day, that affects the expiry of producer IDs and updates the old parameter to only affect the expiry of transaction IDs.

 

KIP-854: 생산자 ID 만료에 대한 별도의 구성

KIP-854는 과도한 메모리 사용을 방지하기 위해 생산자 ID를 보다 효율적으로 정리하는 변경 사항을 도입합니다. 생산자 ID의 만료에 영향을 미치고 트랜잭션 ID의 만료에만 영향을 미치도록 이전 매개변수를 업데이트하는 기본값이 1일인 새로운 제한 시간 매개변수를 도입합니다.

---

KIP-876: Time based cluster metadata snapshots

KIP-876 adds a new property that defines the maximum amount of time that the server will wait to generate a snapshot; the default is 1 hour. This feature was only implemented for the KRaft Controllers in 3.4.

 

KIP-876: 시간 기반 클러스터 메타데이터 스냅샷

KIP-876은 서버가 스냅샷을 생성하기 위해 대기하는 최대 시간을 정의하는 새로운 속성을 추가합니다. 기본값은 1시간입니다. 이 기능은 3.4의 KRaft 컨트롤러에만 구현되었습니다.

---

KIP-881: Rack-aware Partition Assignment for Kafka Consumers

KIP-881 is an extension of KIP-392, which makes it so that consumers can be rack-aware when it comes to partition assignments and consumer rebalancing. Note that only the protocol changes have been implemented in 3.4.0 but none of the out-of-the-box assignors utilize this yet, so the rack-aware assignment is only available to those who plug in a custom assignor at this point.

 

KIP-881: Kafka 소비자를 위한 랙 인식 파티션 할당

KIP-881은 KIP-392의 확장으로, 파티션 할당 및 소비자 재조정과 관련하여 소비자가 랙을 인식할 수 있도록 합니다. 프로토콜 변경 사항만 3.4.0에서 구현되었지만 기본 할당자는 아직 이를 활용하지 않으므로 랙 인식 할당은 이 시점에서 사용자 지정 할당자를 연결하는 사람만 사용할 수 있습니다.

---

Kafka Streams

KIP-770: Replace cache.max.bytes.buffering with cache.max.bytes

KIP-770 deprecates the existing cache.max.bytes.buffering config and introduces a new cache.max.bytes config to replace it. The semantics and default value of the cache size config is unchanged. This KIP also adds a new cache.size metric at the DEBUG level for users to monitor the actual size of the Kafka Streams cache.

 

KIP-770: cache.max.bytes.buffering을 cache.max.bytes로 교체

KIP-770은 기존 cache.max.bytes.buffering 구성을 더 이상 사용하지 않으며 이를 대체할 새 cache.max.bytes 구성을 도입합니다. 캐시 크기 구성의 의미와 기본값은 변경되지 않습니다. 또한 이 KIP는 사용자가 Kafka Streams 캐시의 실제 크기를 모니터링할 수 있도록 DEBUG 수준에서 새로운 cache.size 메트릭을 추가합니다.

---

KIP-837: Allow MultiCasting a Result Record

KIP-837 allows users to multicast result records to every partition of downstream sink topics and adds functionality for users to choose to drop result records without sending.

 

KIP-837: 결과 레코드의 멀티캐스팅 허용

KIP-837은 사용자가 다운스트림 싱크 항목의 모든 파티션에 결과 레코드를 멀티캐스트할 수 있도록 하고 사용자가 전송하지 않고 결과 레코드를 삭제하도록 선택할 수 있는 기능을 추가합니다.

---

KIP-865: Support --bootstrap-server in kafka-streams-application-reset

KIP-865 updates the Kafka Streams application reset tool’s server parameter name to conform to the other Kafka tooling by deprecating the --bootstrap-servers parameter and introducing a new --bootstrap-server one in its place.

 

KIP-865: kafka-streams-application-reset에서 --bootstrap-server 지원

KIP-865는 --bootstrap-servers 매개변수를 더 이상 사용하지 않고 그 자리에 새로운 --bootstrap-server 매개변수를 도입하여 다른 Kafka 도구를 준수하도록 Kafka Streams 애플리케이션 재설정 도구의 서버 매개변수 이름을 업데이트합니다.

---

Kafka Connect

KIP-787: MirrorMaker2 custom Kafka Admin interface for managing Kafka resources

KIP-787 allows users to run MirrorMaker2 with custom implementations for the Kafka resource manager and makes it easier to integrate with your ecosystem.  This allows users to load a custom implementation of the Kafka Admin interface through the use of a new ForwardingAdmin class delegate.  A custom resource implementation solves the issues of providing MM2 with the correct authorization and authentication in organizations that use federated or resource management systems. 

 

KIP-787: Kafka 리소스 관리를 위한 MirrorMaker2 사용자 지정 Kafka Admin 인터페이스

KIP-787을 사용하면 사용자가 Kafka 리소스 관리자에 대한 사용자 정의 구현으로 MirrorMaker2를 실행할 수 있으며 에코시스템과 더 쉽게 통합할 수 있습니다. 이를 통해 사용자는 새로운 ForwardingAdmin 클래스 대리자를 사용하여 Kafka Admin 인터페이스의 사용자 정의 구현을 로드할 수 있습니다. 사용자 지정 리소스 구현은 페더레이션 또는 리소스 관리 시스템을 사용하는 조직에서 MM2에 올바른 권한 부여 및 인증을 제공하는 문제를 해결합니다.

---

Summary

In addition to all of the KIPs listed above, Apache Kafka 3.4 includes a number of fixes and improvements. To learn more:

• See the release notes for 3.4.0 for the full list of changes
• Watch the video
• Listen to the podcast
• Download Apache Kafka 3.4 and get started