关于apachestorm：Kafka spout 集成


Kafka spout integration

我正在使用 kafka 0.10.1.1 和storm 1.0.2。在 kafka 集成的storm文档中，我可以看到偏移量仍然使用zookeeper维护，因为我们正在使用zookeeper服务器初始化kafka spout。
我如何使用kafka服务器引导spout。有没有这方面的例子。
来自风暴文档的示例

1
2
3
4
    BrokerHosts hosts = new ZkHosts(zkConnString);
SpoutConfig spoutConfig = new SpoutConfig(hosts, topicName,"/" + topicName, UUID.randomUUID().toString());
spoutConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);

这个使用 zookeeper 的选项工作正常并且正在使用消息。但我无法在 kafkamanager ui 中将消费者组或风暴节点视为消费者。

尝试过的替代方法是这样的。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
KafkaSpoutConfig<String, String> kafkaSpoutConfig = newKafkaSpoutConfig();

KafkaSpout<String, String> spout = new KafkaSpout<>(kafkaSpoutConfig);

 private static KafkaSpoutConfig<String, String> newKafkaSpoutConfig() {

        Map<String, Object> props = new HashMap<>();
        props.put(KafkaSpoutConfig.Consumer.BOOTSTRAP_SERVERS, bootstrapServers);
        props.put(KafkaSpoutConfig.Consumer.GROUP_ID, GROUP_ID);
        props.put(KafkaSpoutConfig.Consumer.KEY_DESERIALIZER,
               "org.apache.kafka.common.serialization.StringDeserializer");
        props.put(KafkaSpoutConfig.Consumer.VALUE_DESERIALIZER,
               "org.apache.kafka.common.serialization.StringDeserializer");
        props.put(KafkaSpoutConfig.Consumer.ENABLE_AUTO_COMMIT,"true");

        String[] topics = new String[1];
        topics[0] = topicName;

        KafkaSpoutStreams kafkaSpoutStreams =
                new KafkaSpoutStreamsNamedTopics.Builder(new Fields("message"), topics).build();

        KafkaSpoutTuplesBuilder<String, String> tuplesBuilder =
                new KafkaSpoutTuplesBuilderNamedTopics.Builder<>(new TuplesBuilder(topicName)).build();

        KafkaSpoutConfig<String, String> spoutConf =
                new KafkaSpoutConfig.Builder<>(props, kafkaSpoutStreams, tuplesBuilder).build();

        return spoutConf;
    }

但这个解决方案在从 kafka 读取几条消息后显示 CommitFailedException。

在您的替代方法中，您可能会得到 CommitFailedException，原因是：

1
props.put(KafkaSpoutConfig.Consumer.ENABLE_AUTO_COMMIT,"true");

直到 Storm 2.0.0-SNAPSHOT(以及自 1.0.6 起)-
不支持 KafkaConsumer 自动提交

来自文档：

Note that KafkaConsumer autocommit is unsupported. The
KafkaSpoutConfig constructor will throw an exception if the
"enable.auto.commit" property is set, and the consumer used by the
spout will always have that property set to false. You can configure
similar behavior to autocommit through the setProcessingGuarantee
method on the KafkaSpoutConfig builder.

参考文献：

  • http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-kafka-client.html
  • http://storm.apache.org/releases/1.0.6/storm-kafka-client.html

Storm-kafka 用普通的kafka 客户端在zookeeper 中以不同的位置和不同的格式写入消费者信息。所以你在kafkamanager ui中看不到它。

你可以找到一些其他的监控工具，比如
https://github.com/keenlabs/capillary.