千夜同学 | 排查 Logstash 2.4 升级到 5.0 版本后 Kafka 不兼容问题

之前对 ELKB 环境从 2.4 版本升级到最新的 5.0 稳定版本，发现 Kafka 集群运行报错，现在把排查过程记录下。

之前环境：

logstash2.4
logstash-input-kafka-2.0.9
logstash-output-kafka-2.0.5
kafka_2.10-0.8.2.2.tgz

升级后环境：

logstash5.0
logstash-input-kafka-2.0.9
logstash-output-kafka-2.0.5

报错信息：

[2016-11-16T14:35:44,739][ERROR][logstash.inputs.kafka] Unknown setting 'zk_connect' for kafka
[2016-11-16T14:35:44,741][ERROR][logstash.inputs.kafka] Unknown setting 'topic_id' for kafka
[2016-11-16T14:35:44,741][ERROR][logstash.inputs.kafka] Unknown setting 'reset_beginning' for kafka

解决思路

根据错误查看程序哪里报错

grep "Unknown setting" /usr/share/logstash/ -R
/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:self.logger.error("Unknown setting '#{name}' for #{@plugin_name}")

查看程序相关代码，发现需要查看 plugins 的 config 定义文件等

def validate_check_invalid_parameter_names(params)
  invalid_params = params.keys
  # Filter out parameters that match regexp keys.
  # These are defined in plugins like this:
  #   config /foo.*/ => ...
  @config.each_key do |config_key|
    if config_key.is_a?(Regexp)
      invalid_params.reject! { |k| k =~ config_key }
    elsif config_key.is_a?(String)
      invalid_params.reject! { |k| k == config_key }
    end
  end

  if invalid_params.size > 0
    invalid_params.each do |name|
      self.logger.error("Unknown setting '#{name}' for #{@plugin_name}")
    end
    return false
  end # if invalid_params.size > 0
  return true
end # def validate_check_invalid_parameter_names

cd /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5

发现重点查看如下文件：

grep config ./* -R | awk '{print $1}' | uniq 
./CHANGELOG.md:
./DEVELOPER.md:See
./lib/logstash/inputs/kafka.rb:#
./lib/logstash/inputs/kafka.rb:
./README.md:-
Binary

1）首先看 CHANGELOG.md，就有发现 logstash-input-3.0.0.beta1 开始就不在向后兼容，且剔除了 jruby-kafka。

4.0.0 版本说开始支持 Kafka 0.9。 5.0.0 又说开始支持 0.10 切不向后兼容。

看来问题找到了，Kafka 版本是 kafka_2.10-0.8.2.2.tgz，Kafka 版本不兼容导致的。

CHANGELOG.md 部分文档如下：

## 5.0.4
  - Update to Kafka version 0.10.0.1 for bug fixes
## 5.0.0
  - Support for Kafka 0.10 which is not backward compatible with 0.9 broker.
## 4.0.0
  - Republish all the gems under jruby.
  - Update the plugin to the version 2.0 of the plugin api, this change is required for Logstash 5.0 compatibility. See https://github.com/elastic/logstash/issues/5141
  - Support for Kafka 0.9 for LS 5.x
## 3.0.0.beta1
 - Refactor to use new Java based consumer, bypassing jruby-kafka
 - Breaking: Change configuration to match Kafka's configuration. This version is not backward compatible

2）之前看 DEVELOPER.md 文档时，看配置语法都正确，还以为是却少依赖关系 jruby-kafka library 呢，这个在 logstash2.x 是可以使用的。

另外 Kafka 版本写的是 0.8.1.1，看来这个 DEVELOPER.md 没有及时更新，与后面 kafka.rb 文件不一致。

DEVELOPER.md 文档结尾如下：

Dependencies
====================
* Apache Kafka version 0.8.1.1
* jruby-kafka library

3）开始看 README.md 文档，特意看了下 Kafka 的兼容性，看来 logstas-input-kafka5.0.5 和 logstash-output-kafka5.0.4 只能用 kafka0.10 了。

如果想用 kafka0.9 还想用 logstash5.0，logstash-input-kafka 和 logstash-output-kafka 只能降级版本到 4.0.0 了。

### Kafka Compatibility
Here's a table that describes the compatibility matrix for Kafka Broker support. Please remember that it is good advice to upgrade brokers before consumers/producers since brokers target backwards compatibility. The 0.9 broker will work with both the 0.8 consumer and 0.9 consumer APIs but not the other way around.

| Kafka Broker Version | Logstash Version | Input Plugin | Output Plugin | Why? |
|----------------------|------------------|--------------|---------------|------|
| 0.8  | 2.0 - 2.x   | < 3.0.0 | <3.0.0 | Legacy, 0.8 is still popular |
| 0.9  | 2.0 - 2.3.x | 3.0.0   | 3.0.0  | Intermediate release before 0.10 that works with old Ruby Event API `[]` |
| 0.9  | 2.4, 5.0    | 4.0.0   | 4.0.0  | Intermediate release before 0.10 with new get/set API |
| 0.10 | 2.4, 5.0    | 5.0.0   | 5.0.0  | Track latest Kafka release. Not compatible with 0.9 broker |

4）现在看来只能升级 Kafka 版本了。

最后看了下 jar-dependencies 发现了 kafka-clients-0.10.0.1.jar。

ls /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/vendor/jar-dependencies/runtime-jars/
kafka-clients-0.10.0.1.jar  log4j-1.2.17.jar  lz4-1.3.0.jar  slf4j-api-1.7.21.jar  slf4j-log4j12-1.7.21.jar  snappy-java-1.1.2.6.jar

5）/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.0.5/lib/logstash/inputs/kafka.rb。

kafka.rb 部分文档如下：

# This input will read events from a Kafka topic. It uses the the newly designed
# 0.10 version of consumer API provided by Kafka to read messages from the broker.

# Here's a compatibility matrix that shows the Kafka client versions that are compatible with each combination
# of Logstash and the Kafka input plugin: 
 
# [options="header"]
# |==========================================================
# |Kafka Client Version |Logstash Version |Plugin Version |Security Features |Why?
# |0.8 |2.0.0 - 2.x.x|<3.0.0 | |Legacy, 0.8 is still popular 
# |0.9 |2.0.0 - 2.3.x| 3.x.x |Basic Auth, SSL|Works with the old Ruby Event API (`event['product']['price'] = 10`)  
# |0.9 |2.4.0 - 5.0.x| 4.x.x |Basic Auth, SSL|Works with the new getter/setter APIs (`event.set('[product][price]', 10)`)
# |0.10|2.4.0 - 5.0.x| 5.x.x |Basic Auth, SSL|Not compatible with the 0.9 broker 
# |==========================================================
 
# NOTE: We recommended that you use matching Kafka client and broker versions. During upgrades, you should
# upgrade brokers before clients because brokers target backwards compatibility. For example, the 0.9 broker
# is compatible with both the 0.8 consumer and 0.9 consumer APIs, but not the other way around.

6）注意几个关键配置需要修改。

config :bootstrap_servers, :validate => :string, :default => "localhost:9092"
config :group_id, :validate => :string, :default => "logstash"
config :topics, :validate => :array, :default => ["logstash"]
config :consumer_threads, :validate => :number, :default => 1