Apache Storm的安装部署

一、基础介绍

Storm是一个免费开源的分布式实时计算系统。分布式意味着Storm是一个集群，部署在多台机器上。实时便是实时计算，相比于MapReduce的批处理，实时更关注于数据处理的速度和延时。

Apache Storm官网提供了各个版本的下载，体现为apache-storm-*.tar.gz，部署Storm时，直接将其解压，并配置相关配置文件即可。注意到，Storm采用Clojure和Java语言编写，Clojure也是运行在JVM之上的，所以环境上要保证安装Java环境。

Storm运行时体现为Master-Worker集群。Master节点运行nimbus进程，给Work节点分任务。Worker节点运行supervisor进程，负责分配nimbus传递过来的任务，以启动或停止worker进程。nimbus和supervisor都是无状态的，它们之间通过zookeeper来协调任务，也就是将状态信息存放在zookeeper中。

Storm的集群部署为：

二、基础环境

# Linux操作系统版本
root@linux:# lsb_release -a 
No LSB modules are available.
Distributor ID:  Ubuntu
Description:  Ubuntu 18.04.2 LTS
Release:  18.04
Codename:  bionic
# python版本
root@linux:# python --version
Python 2.7.17
root@linux:# python3 --version
Python 3.6.9
# java版本
root@linux:# java -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-8u272-b10-0ubuntu1~18.04-b10)
OpenJDK 64-Bit Server VM (build 25.272-b10, mixed mode)

三、Zookeeper安装

1. 下载Zookeeper包，解压并部署在/opt目录下

tar -xvf apache-zookeeper-3.7.1-bin.tar.gz 
mkdir /opt/zookeeper
chmod 777 /opt/zookeeper/
mv apache-zookeeper-3.7.1-bin.tar.gz /opt/zookeeper/

2. 配置zoo.cfg文件

# The number of milliseconds of each tick
# 心跳时间，单位毫秒
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
# Leader和Follower初始连接时最大的心跳数
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
# 保存Zookeeper数据的目录
dataDir=/opt/zookeeper/zkdata
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60

3.启动 Zookeeper 服务端

root@linux:/opt/zookeeper/apache-zookeeper-3.7.1-bin/bin# ./zkServer.sh start
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/apache-zookeeper-3.7.1-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
root@linux:/opt/zookeeper/apache-zookeeper-3.7.1-bin/bin#

4. 查看进程

root@linux:/opt/zookeeper/apache-zookeeper-3.7.1-bin/bin# jps
18706 Jps
18670 QuorumPeerMain #Zookeeper服务进程
root@linux:/opt/zookeeper/apache-zookeeper-3.7.1-bin/bin# 
查看状态
root@linux:/opt/zookeeper/apache-zookeeper-3.7.1-bin/bin# ./zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper/apache-zookeeper-3.7.1-bin/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: standalone

四、Storm安装

1. 修改conf/storm.yaml文件，修改为本机的IP地址

########### These MUST be filled in for a storm configuration
 storm.zookeeper.servers: #Zookeeper主机列表
     - "30.0.0.218"
 nimbus.seeds: ["30.0.0.218"] #master候选者

2. 打开/etc/profile文件，增加如下：

export PATH=$PATH:/opt/apache-storm-2.3.0/bin

3. 执行命令：source /etc/profile。
4. 按照顺序启动：

storm nimbus &
storm supervisor &
storm ui &

5. 查看启动进程：

root@linux:# jps
22817 UIServer
22549 Nimbus
22709 Supervisor
20775 QuorumPeerMain
23039 Jps

可能会遇到的问题：

端口冲突问题：
root@linux:/opt# Running: java -server -Ddaemon.name=ui -Dstorm.options= -Dstorm.home=/opt/apache-storm-2.3.0 -Dstorm.log.dir=/opt/apache-storm-2.3.0/logs -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64 -Dstorm.conf.file= -cp /opt/apache-storm-2.3.0/*:/opt/apache-storm-2.3.0/lib/*:/opt/apache-storm-2.3.0/extlib/*:/opt/apache-storm-2.3.0/extlib-daemon/*:/opt/apache-storm-2.3.0/lib-webapp/*:/opt/apache-storm-2.3.0/conf -Xmx768m -Djava.deserialization.disabled=true -Dlogfile.name=ui.log -Dlog4j.configurationFile=/opt/apache-storm-2.3.0/log4j2/cluster.xml org.apache.storm.daemon.ui.UIServer
Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8080
  at org.apache.storm.daemon.ui.UIServer.main(UIServer.java:183)
Caused by: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8080
  at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:346)
  at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:308)
  at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
  at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
  at org.eclipse.jetty.server.Server.doStart(Server.java:394)
  at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
  at org.apache.storm.daemon.ui.UIServer.main(UIServer.java:179)
Caused by: java.net.BindException: Address already in use
  at sun.nio.ch.Net.bind0(Native Method)
  at sun.nio.ch.Net.bind(Net.java:461)
  at sun.nio.ch.Net.bind(Net.java:453)
  at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
  at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
  at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342)
  ... 7 more


执行命令：lsof -i:8080，可以看到8080被zookeeper占用。
修改zookeeper的zoo.cfg文件，添加如下：
admin.serverPort=8008
端口冲突解决。