Release Notes - Kafka - Version 3.7.1

Below is a summary of the JIRA issues addressed in the 3.7.1 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

Improvement

[KAFKA-15905] - Restarts of MirrorCheckpointTask should not permanently interrupt offset translation

[KAFKA-16108] - Backport fix for KAFKA-16093 to 3.7

Bug

[KAFKA-9228] - Reconfigured converters and clients may not be propagated to connector tasks

[KAFKA-15170] - CooperativeStickyAssignor cannot adjust assignment correctly

[KAFKA-16003] - The znode /config/topics is not updated during KRaft migration in "dual-write" mode

[KAFKA-16025] - Streams StateDirectory has orphaned locks after rebalancing, blocking future rebalancing

[KAFKA-16047] - Source connector with EOS enabled have some InitProducerId requests timing out, effectively failing all the tasks & the whole connector

[KAFKA-16073] - Kafka Tiered Storage: Consumer Fetch Error Due to Delayed localLogStartOffset Update During Segment Deletion

[KAFKA-16105] - Reassignment of tiered topics is failing due to RemoteStorageException

[KAFKA-16209] - fetchSnapshot might return null if topic is created before v2.8

[KAFKA-16217] - Transactional producer stuck in IllegalStateException during close

[KAFKA-16222] - KRaft Migration: desanitize entity name when migrate client quotas

[KAFKA-16226] - Java client: Performance regression in Trogdor benchmark with high partition counts

[KAFKA-16277] - CooperativeStickyAssignor does not spread topics evenly among consumer group

[KAFKA-16278] - Missing license for scala related dependencies

[KAFKA-16305] - Optimisation in SslTransportLayer:handshakeUnwrap stalls TLS handshake

[KAFKA-16319] - Wrong broker destinations for DeleteRecords requests when more than one topic is involved and the topics/partitions are led by different brokers

[KAFKA-16322] - Fix CVE-2023-50572 by updating jline from 3.22.0 to 3.25.1

[KAFKA-16347] - Bump ZooKeeper to 3.8.4

[KAFKA-16359] - kafka-clients-3.7.0.jar published to Maven Central is defective

[KAFKA-16369] - Broker may not shut down when SocketServer fails to bind as Address already in use

[KAFKA-16371] - Unstable committed offsets after triggering commits where metadata for some partitions are over the limit

[KAFKA-16386] - NETWORK_EXCEPTIONs from transaction verification are not translated

[KAFKA-16392] - Spurious log warnings: "Ignoring offset partition key with an unexpected format for the second element in the partition key list. Expected type: java.util.Map, actual type: null"

[KAFKA-16408] - kafka-get-offsets / GetOffsetShell doesn't handle --version or --help

[KAFKA-16410] - kafka-leader-election / LeaderElectionCommand doesn't set exit code on error

[KAFKA-16411] - Correctly migrate default client quota entities in KRaft migration

[KAFKA-16428] - Fix bug where config change notification znode may not get created during migration

[KAFKA-16466] - QuorumController is swallowing some exception messages

[KAFKA-16471] - SslTransportLayer may leak SSLEngine resources

[KAFKA-16473] - KafkaDockerWrapper uses wrong cluster ID when formatting log dir

[KAFKA-16511] - Leaking tiered segments

[KAFKA-16539] - Can't update specific broker configs in pre-migration mode

[KAFKA-16563] - migration to KRaft hanging after MigrationClientException

[KAFKA-16570] - FenceProducers API returns "unexpected error" when successful

[KAFKA-16583] - Update from 3.4.0 to 3.7.0 image write failed in Kraft mode

[KAFKA-16606] - JBOD support in KRaft does not seem to be gated by the metadata version

[KAFKA-16622] - Mirromaker2 first Checkpoint not emitted until consumer group fully catches up once

[KAFKA-16692] - InvalidRequestException: ADD_PARTITIONS_TO_TXN with version 4 which is not enabled when upgrading from kafka 3.5 to 3.6

[KAFKA-16757] - Fix broker re-registration issues around MV 3.7-IV2

[KAFKA-16790] - Calls to RemoteLogManager are made before it is configured

[KAFKA-16807] - DescribeLogDirsResponseData#results#topics have unexpected topics having empty partitions

[KAFKA-16814] - KRaft broker cannot startup when `partition.metadata` is missing

[KAFKA-16837] - Kafka Connect fails on update connector for incorrect previous Config Provider tasks

[KAFKA-16838] - Kafka Connect loads old tasks from removed connectors

[KAFKA-16886] - KRaft partition reassignment failed after upgrade to 3.7.0

[KAFKA-16969] - KRaft unable to upgrade to v3.7.1 and later when multiple log dir is set

[KAFKA-16988] - InsufficientResourcesError in ConnectDistributedTest system test

Task

[KAFKA-16605] - Fix the flaky LogCleanerParameterizedIntegrationTest

[KAFKA-16645] - CVEs in 3.7.0 docker image

[KAFKA-16825] - CVE vulnerabilities in Jetty and netty

[KAFKA-16881] - InitialState type leaks into the Connect REST API OpenAPI spec

Test

[KAFKA-15206] - Flaky test RemoteIndexCacheTest.testClose()

[KAFKA-16293] - Test log directory failure in Kraft

[KAFKA-16472] - Integration tests in Java don't really run kraft case

[KAFKA-16559] - allow defining number of disks per broker in TestKitNodes

[KAFKA-16596] - Flaky test – org.apache.kafka.clients.ClientUtilsTest.testParseAndValidateAddressesWithReverseLookup()

[KAFKA-16961] - TestKRaftUpgrade system tests fail in v3.7.1 RC1

Sub-task

[KAFKA-16082] - Broker recreates reassigned partition after logdir failure

[KAFKA-16234] - Log directory failure re-creates partitions in another logdir automatically

[KAFKA-16297] - Race condition while promoting future replica can lead to partition unavailability.

[KAFKA-16365] - AssignmentsManager mismanages completion notifications