无微不至之RocketMq源码深度讲解(3)-NameServer分析

开头语:

上一节大致讲了Broker组件将自身的服务元数据(官方叫路由信息)注册到NameServer中进行内存存储。以及讲了一下RocketMq客户端请求在服务端如何进行处理的。那么这一节,将详细的讲解Broker的服务元素据是如何传送给NameServer的,以及NameServer是以什么样的数据结构进行存储的。

流程简介

Broker组件是如何上报自身信息给NameServer的?
首先这里要说的是Broker组件启动流程是与上一节讲的NameServer组件的启动流程是一摸一样的,Broker首先通过BrokerStartup的main方法,经过一系列的配置文件参数封装,然后创建BrokerController实例,最终调用BrokerController#initialize方法进行一些初始化操作,这些初始化操作具体是什么,将在后续文章中详细讲解。再调用BrokerController#start方法将一些内部组件进行启动。其中将服务元素据注册到NameServer上也是再里面进行的。我们定位到start方法注册的代码块中:
image.png
从图片中可以看到在BrokerController#start方法中有2处调用了registerBrokerAll方法进行发路由注册,其中第一处是强制进行注册,第二处是定时注册,每隔10-60s之间,具体取决于我们的registerNameServerPeriod参数。我们直接进入到registerBrokerAll方法中进行讲解:

registerBrokerAll方法(服务元素据注册)

public synchronized void registerBrokerAll(final boolean checkOrderConfig, boolean oneway, boolean forceRegister) {
    TopicConfigSerializeWrapper topicConfigWrapper = this.getTopicConfigManager().buildTopicConfigSerializeWrapper();
 //1.判断是否有读写权限
 if (!PermName.isWriteable(this.getBrokerConfig().getBrokerPermission())
        || !PermName.isReadable(this.getBrokerConfig().getBrokerPermission())) {
        ConcurrentHashMap<String, TopicConfig> topicConfigTable = new ConcurrentHashMap<String, TopicConfig>();
 //将topicConfig进行复制一份
 for (TopicConfig topicConfig : topicConfigWrapper.getTopicConfigTable().values()) {
            TopicConfig tmp =
                new TopicConfig(topicConfig.getTopicName(), topicConfig.getReadQueueNums(), topicConfig.getWriteQueueNums(),
 this.brokerConfig.getBrokerPermission());
 topicConfigTable.put(topicConfig.getTopicName(), tmp);
 }
        topicConfigWrapper.setTopicConfigTable(topicConfigTable);
 }
    //2强制注册,或者向nameServer发送请求,看是否nameserver与本地dataVersion一致
 if (forceRegister || needRegister(this.brokerConfig.getBrokerClusterName(),
 this.getBrokerAddr(),
 this.brokerConfig.getBrokerName(),
 this.brokerConfig.getBrokerId(),
 this.brokerConfig.getRegisterBrokerTimeoutMills())) {
        doRegisterBrokerAll(checkOrderConfig, oneway, topicConfigWrapper);
 }
}

1.从代码块可以看到首先是从TopicConfigManager(路由信息管理类,负责TopicConfig的版本管理,持久化和内存之间的转换)类中获取到TopicConfigSerializeWrapper,这个包装类主要有TopicConfig列表与DataVersion(数据版本)2部分组成。将Broker端的TopicConfig数据结构复制一份,我们这里需要着重了解的是TopicConfig这个数据结构中具体都包含了哪些东西以及这些配置是什么时候添加进去的.

TopicConfig(路由信息)

image.png
从图中可以看出topicConfig主要包含一下几个参数:
topicName:主题名
readQueueNums:读队列数(相当于kafka的分区概念,用于consumer端)
writeQueueNums:写队列数(用于producer端)
perm:topic拥有的权限(默认拥有读写)
2.从判断条件中可以看出只有开启强制注册与需要注册的时候,才会像NameServer注册Broker自身的元素据.
强制注册是由我们外部配置决定的,那么合适需要注册呢?

答:对topicConfig列表配置信息的增删改都会触发DataVersion版本号的递增,那么在needRegister方法中的逻辑是:
**image.png
名词解释:(个人的理解)
clusterName:集群名(默认DefaultCluster,可通过brokerClusterName配置参数进行修改),NameServer可能同时涉及到多个集群,一个集群包含多个brokerName

brokerAddr:当前broker对外暴露的地址 ip+port(默认ip是取本地网卡的内网ip,可以通过brokerIp1参数进行修改,port默认为8888,可以通过listernPort参数进行修改)

brokerName:broker名称,一个brokerName可能对应多个BrokerId,也就是包含一组broker服务。在一组broker服务中,一般只包含BrokerId为0的(也就是Master)和多个非0的BrokerId(Slave)

timeoutMills:请求超时时间

(1)向NameServer发送一个查询请求,业务code为RequestCode.QUERY_DATA_VERSION,在上一节中,已经讲到不同的业务code可以在服务断(此处指的是NameServer)对应不同的NettyRequestProcessor实现类进行处理。但是在上结我们已经看到所有发送给NameServer的请求,都是统一由DefaultRequestProcessor进行处理的。因为在NameServerController启动流程中只注册了这一个RequestProcessor。
(2)将服务元素据及TopicConfig发送给NameServer集群,由NameServer集群各自返回当前BrokerAddr的TopicConfig的DataVersion,最终由Broker判断是否与自身的DataVersion一致,如果NameServer集群服务有一台不一致,说明TopicConfig和NameServer
集群有差异,则需要将Broker自身的TopicConfig上报给NameServer进行更新覆盖。
(3)讲完了大致流程之后,我们看到needRegister方法中一个brokerOuterApi组件,这个组件承担了与NameServer建立连接并发送请求的职责。这个组件是在BrokerController的构造方法中进行实例化的。

BrokerOuterAPI组件:

this.brokerOuterAPI = new BrokerOuterAPI(nettyClientConfig);
这个NettyClientConfig就是我们从broker.conf配置文件中配置的,通过反射将参数值注入到nettyClientConfig实例中。NettyClientConfig核心配置参数主要包括:

image.png

image.png
从构造方法,可以看出其实内部维护的时NettyRemotingClient这个类,而这个类继承了NettyRemotingAbstract类,上节讲到的NettyRemotingServer也继承了这个父类。而这个父类实现了processMessageReceived这个方法,也就是我们上节讲到的。在这个方法中封装了处理request和respons的通用方法(具体看上一节)。
NettyRemotingClient这个类也是Producer和Consumer端的核心类。都是由这个类完成与服务端的RPC调用。Broker相对于Producer和Consumer来说时服务断.NameServer相对于Producer和Consuemr和Broker来说是服务端。
registerRpcHook这个方法忽略不讲,就是在RPC调用前后的前置处理和后置处理。
直接进入到NettyRemotingClient构造方法中:
image.png
构造方法中主要工作利用NettyClientConfig配置参数是初始化一些线程池以及netty客户端BootStartp启动类需要用到的线程组(用来处与服务断建立连接)以及2个信号量的实例,这些线程池具体的用处将在后面讲到。
BrokerOuterAPI实例创建完成后,在调用BrokerController的start方法时,会调用到BrokerOuterAPI#start方法,BrokerOuterAPI的start方法如下:


public void start() {
    //客户端 默认工作线程数是4
 this.defaultEventExecutorGroup = new DefaultEventExecutorGroup(
        nettyClientConfig.getClientWorkerThreads(),
 new ThreadFactory() {
            private AtomicInteger threadIndex = new AtomicInteger(0);
 @Override
 public Thread newThread(Runnable r) {
                return new Thread(r, "NettyClientWorkerThread_" + this.threadIndex.incrementAndGet());
 }
        });
 Bootstrap handler = this.bootstrap.group(this.eventLoopGroupWorker).channel(NioSocketChannel.class)
        .option(ChannelOption.TCP_NODELAY, true)
        .option(ChannelOption.SO_KEEPALIVE, false)
        .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, nettyClientConfig.getConnectTimeoutMillis())
        .option(ChannelOption.SO_SNDBUF, nettyClientConfig.getClientSocketSndBufSize())
        .option(ChannelOption.SO_RCVBUF, nettyClientConfig.getClientSocketRcvBufSize())
        .handler(new ChannelInitializer<SocketChannel>() {
            @Override
 public void initChannel(SocketChannel ch) throws Exception {
                ChannelPipeline pipeline = ch.pipeline();
 if (nettyClientConfig.isUseTLS()) {
                    if (null != sslContext) {
                        pipeline.addFirst(defaultEventExecutorGroup, "sslHandler", sslContext.newHandler(ch.alloc()));
 log.info("Prepend SSL handler");
 } else {
                        log.warn("Connections are insecure as SSLContext is null!");
 }
                }
                pipeline.addLast(
                    defaultEventExecutorGroup,
 new NettyEncoder(),
 new NettyDecoder(),
 new IdleStateHandler(0, 0, nettyClientConfig.getClientChannelMaxIdleTimeSeconds()),
 new NettyConnectManageHandler(),
 new NettyClientHandler());
 }
        });
 //每秒扫描超时的request并做回调处理
 this.timer.scheduleAtFixedRate(new TimerTask() {
        @Override
 public void run() {
            try {
                NettyRemotingClient.this.scanResponseTable();
 } catch (Throwable e) {
                log.error("scanResponseTable exception", e);
 }
        }
    }, 1000 * 3, 1000);
 if (this.channelEventListener != null) {
        this.nettyEventExecutor.start();
 }
}
**BrokerController的start方法主要做了以下几件事:**
1.根据我们配置clientWorkerThreads工作线程数初始化一个线程池用来处理io事件,初始化一个Netty的客户端启动类BootStrap实例,然后添加一系列handler到BootStrap中,Handler的处理顺序同上节一直。这里注意到pipeline.addLast(defaultEventExecutorGroup,...)指定我们之前实例化的线程池作为这些handler的工作线程池。这里没与服务断建立连接呢,而只是初始化了一个BootStrap实例,等真正调用的时候才会与服务端进行连接,然后将连接后的Channel缓存起来(后续讲到),提高代码的可用性。
2.第二件事主要是定期扫描每次请求对应的ResponseFuture,判断对应请求是否已经超时,将超时请求对应ResponseFuture移除,并回调ResponseFuture中的InvokeCallback回调接口进行处理。代码如下:
public void scanResponseTable() {
    final List<ResponseFuture> rfList = new LinkedList<ResponseFuture>();
//请求对应的ResponseFuture列表
 Iterator<Entry<Integer, ResponseFuture>> it = this.responseTable.entrySet().iterator();
 while (it.hasNext()) {
        Entry<Integer, ResponseFuture> next = it.next();
 ResponseFuture rep = next.getValue();
 if ((rep.getBeginTimestamp() + rep.getTimeoutMillis() + 1000) <= System.currentTimeMillis()) {
            rep.release();
//移除
 it.remove();
 rfList.add(rep);
 log.warn("remove timeout request, " + rep);
 }
    }
    for (ResponseFuture rf : rfList) {
        try {
//执行回调函数
            executeInvokeCallback(rf);
 } catch (Throwable e) {
            log.warn("scanResponseTable, operationComplete Exception", e);
 }
    }
}

private void executeInvokeCallback(final ResponseFuture responseFuture) {
    boolean runInThisThread = false;
//获取线程池
 ExecutorService executor = this.getCallbackExecutor();
 if (executor != null) {
        try {
            executor.submit(new Runnable() {
                @Override
 public void run() {
                    try {
                        responseFuture.executeInvokeCallback();
 } catch (Throwable e) {
                        log.warn("execute callback in executor exception, and callback throw", e);
 } finally {
                        responseFuture.release();
 }
                }
            });
 } catch (Exception e) {
            runInThisThread = true;
 log.warn("execute callback in executor exception, maybe executor busy", e);
 }
    } else {
        runInThisThread = true;
 }
//运行在当前线程中
    if (runInThisThread) {
        try {
            responseFuture.executeInvokeCallback();
 } catch (Throwable e) {
            log.warn("executeInvokeCallback Exception", e);
 } finally {
            responseFuture.release();
 }
    }
}

在executeInvokeCallback中获取到线程池默认是我们先前在NettyRemotingClient创建的publicExecutor线程池,也就是利用clientCallbackThreads这个参数创建出来的。
RPC请求过程
准备工作都做完了,接下来就分析客户端(这里指的是Broker)是如何发起RPC调用请求到服务端(这里是NameServer)的。

直接进入到BrokerOuterAPI#needRegister方法中(判断当前Broker的TopicConfig是否需要注册到NameServer):

public List<Boolean> needRegister(
    final String clusterName,
 final String brokerAddr,
 final String brokerName,
 final long brokerId,
 final TopicConfigSerializeWrapper topicConfigWrapper,
 final int timeoutMills) {
    final List<Boolean> changedList = new CopyOnWriteArrayList<>();
 List<String> nameServerAddressList = this.remotingClient.getNameServerAddressList();
 if (nameServerAddressList != null && nameServerAddressList.size() > 0) {
        //异步执行,线程结果聚合
 final CountDownLatch countDownLatch = new CountDownLatch(nameServerAddressList.size());
 for (final String namesrvAddr : nameServerAddressList) {
            brokerOuterExecutor.execute(new Runnable() {
                @Override
 public void run() {
                    try {
                        //准备请求头数据
 QueryDataVersionRequestHeader requestHeader = new QueryDataVersionRequestHeader();
 requestHeader.setBrokerAddr(brokerAddr);
 requestHeader.setBrokerId(brokerId);
 requestHeader.setBrokerName(brokerName);
 requestHeader.setClusterName(clusterName);
 RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.QUERY_DATA_VERSION, requestHeader);
 //准备请求体
 request.setBody(topicConfigWrapper.getDataVersion().encode());
 //真正发起RPC调用
 RemotingCommand response = remotingClient.invokeSync(namesrvAddr, request, timeoutMills);
 DataVersion nameServerDataVersion = null;
 Boolean changed = false;
 switch (response.getCode()) {
                            case ResponseCode.SUCCESS: {
                                QueryDataVersionResponseHeader queryDataVersionResponseHeader =
                                    (QueryDataVersionResponseHeader) response.decodeCommandCustomHeader(QueryDataVersionResponseHeader.class);
 changed = queryDataVersionResponseHeader.getChanged();
 byte[] body = response.getBody();
 if (body != null) {
                                    nameServerDataVersion = DataVersion.decode(body, DataVersion.class);
 if (!topicConfigWrapper.getDataVersion().equals(nameServerDataVersion)) {
                                        changed = true;
 }
                                }
                                if (changed == null || changed) {
                                    changedList.add(Boolean.TRUE);
 }
                            }
                            default:
                                break;
 }
                        log.warn("Query data version from name server {} OK,changed {}, broker {},name server {}", namesrvAddr, changed, topicConfigWrapper.getDataVersion(), nameServerDataVersion == null ? "" : nameServerDataVersion);
 } catch (Exception e) {
                        changedList.add(Boolean.TRUE);
 log.error("Query data version from name server {}  Exception, {}", namesrvAddr, e);
 } finally {
                        countDownLatch.countDown();
 }
                }
            });
 }
        try {
            countDownLatch.await(timeoutMills, TimeUnit.MILLISECONDS);
 } catch (InterruptedException e) {
            log.error("query dataversion from nameserver countDownLatch await Exception", e);
 }
    }
    return changedList;
}

从代码块中可以看到,对NameServer集群地址遍历,依次发送请求,主要操作包含:
(1)实例化CountDown,并设置数量为NameServer集群服务数。
(2)准备请求的请求头,请求头。
(3)然后发起RPC调用(invokeSync同步调用)。
(4)将NameServer返回的DataVersion与当前Broker的进行对比,相同则无需注册,不同则则需要注册。|
(5)countDown.await方法一直阻塞到所有NameServer都响应结果。

PS:CountDown这种用法适用于异步执行批量任务。

接着看NettyRemotingClient的invokerSync方法:

public RemotingCommand invokeSync(String addr, final RemotingCommand request, long timeoutMillis)
    throws InterruptedException, RemotingConnectException, RemotingSendRequestException, RemotingTimeoutException {
    long beginStartTime = System.currentTimeMillis();
 //根据请求地址  建立连接 有可能是连接broker或者namerServer
 final Channel channel = this.getAndCreateChannel(addr);
 if (channel != null && channel.isActive()) {
        try {
            //前置回调函数调用
 doBeforeRpcHooks(addr, request);
 long costTime = System.currentTimeMillis() - beginStartTime;
 if (timeoutMillis < costTime) {
                throw new RemotingTimeoutException("invokeSync call timeout");
 }
            RemotingCommand response = this.invokeSyncImpl(channel, request, timeoutMillis - costTime);
 //后置回调函数调用
 doAfterRpcHooks(RemotingHelper.parseChannelRemoteAddr(channel), request, response);
 return response;
 } catch (RemotingSendRequestException e) {
            log.warn("invokeSync: send request exception, so close the channel[{}]", addr);
 this.closeChannel(addr, channel);
 throw e;
 } catch (RemotingTimeoutException e) {
            if (nettyClientConfig.isClientCloseSocketIfTimeout()) {
                this.closeChannel(addr, channel);
 log.warn("invokeSync: close socket because of timeout, {}ms, {}", timeoutMillis, addr);
 }
            log.warn("invokeSync: wait response timeout exception, the channel[{}]", addr);
 throw e;
 }
    } else {
        this.closeChannel(addr, channel);
 throw new RemotingConnectException(addr);
 }
}

首先要注意的时getAndCreateChannel这个方方法,这个方法是根据传入的addr先从缓存中获取先前创建的channel,如果该addr从未建立连接,则通过我们在NettyRemotingClient构造方法构造出来的BootStrap启动类,与传入的addr服务端进行连接,如果传入的addr为空,则是从NameServer集群中选取一个地址建立连接。代码如下:

private Channel getAndCreateChannel(final String addr) throws RemotingConnectException, InterruptedException {
    if (null == addr) {
        return getAndCreateNameserverChannel();
 }
    ChannelWrapper cw = this.channelTables.get(addr);
 if (cw != null && cw.isOK()) {
        return cw.getChannel();
 }
    return this.createChannel(addr);
}

createChannel代码如下:

private Channel createChannel(final String addr) throws InterruptedException {
    ChannelWrapper cw = this.channelTables.get(addr);
 if (cw != null && cw.isOK()) {
        return cw.getChannel();
 }
    if (this.lockChannelTables.tryLock(LOCK_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS)) {
        try {
            boolean createNewConnection;
 cw = this.channelTables.get(addr);
 if (cw != null) {
                if (cw.isOK()) {
                    return cw.getChannel();
 } else if (!cw.getChannelFuture().isDone()) {
                    createNewConnection = false;
 } else {
                    this.channelTables.remove(addr);
 createNewConnection = true;
 }
            } else {
                createNewConnection = true;
 }
            if (createNewConnection) {
                //建立连接
 ChannelFuture channelFuture = this.bootstrap.connect(RemotingHelper.string2SocketAddress(addr));
 log.info("createChannel: begin to connect remote host[{}] asynchronously", addr);
 cw = new ChannelWrapper(channelFuture);
 this.channelTables.put(addr, cw);
 }
        } catch (Exception e) {
            log.error("createChannel: create channel exception", e);
 } finally {
            this.lockChannelTables.unlock();
 }
    } else {
        log.warn("createChannel: try to lock channel table, but timeout, {}ms", LOCK_TIMEOUT_MILLIS);
 }
    if (cw != null) {
        ChannelFuture channelFuture = cw.getChannelFuture();
 if (channelFuture.awaitUninterruptibly(this.nettyClientConfig.getConnectTimeoutMillis())) {
            if (cw.isOK()) {
                log.info("createChannel: connect remote host[{}] success, {}", addr, channelFuture.toString());
 return cw.getChannel();
 } else {
                log.warn("createChannel: connect remote host[" + addr + "] failed, " + channelFuture.toString(), channelFuture.cause());
 }
        } else {
            log.warn("createChannel: connect remote host[{}] timeout {}ms, {}", addr, this.nettyClientConfig.getConnectTimeoutMillis(),
 channelFuture.toString());
 }
    }
    return null;
}

在invokeSync方法中可以看出真正核心是掉调用NettyRemotingClient的invokeSyncImpl方法,只是在调用invokeSyncImpl前后调用我们在BrokerOuterAPI的构造方法中注册的RpcHook实现类。我们直接进入到invokeSyncImpl方法中:

public RemotingCommand invokeSyncImpl(final Channel channel, final RemotingCommand request,
 final long timeoutMillis)
    throws InterruptedException, RemotingSendRequestException, RemotingTimeoutException {
    //静态全局自增id
 final int opaque = request.getOpaque();
 try {
        final ResponseFuture responseFuture = new ResponseFuture(channel, opaque, timeoutMillis, null, null);
 this.responseTable.put(opaque, responseFuture);
 final SocketAddress addr = channel.remoteAddress();
 channel.writeAndFlush(request).addListener(new ChannelFutureListener() {
            @Override
 public void operationComplete(ChannelFuture f) throws Exception {
                if (f.isSuccess()) {
                    responseFuture.setSendRequestOK(true);
 return; } else {
                    responseFuture.setSendRequestOK(false);
 }
                responseTable.remove(opaque);
 responseFuture.setCause(f.cause());
 //此时唤醒countDownLatch.await
 responseFuture.putResponse(null);
 log.warn("send a request command to channel <" + addr + "> failed.");
 }
        });
 RemotingCommand responseCommand = responseFuture.waitResponse(timeoutMillis);
 if (null == responseCommand) {
            if (responseFuture.isSendRequestOK()) {
                throw new RemotingTimeoutException(RemotingHelper.parseSocketAddressAddr(addr), timeoutMillis,
 responseFuture.getCause());
 } else {
                throw new RemotingSendRequestException(RemotingHelper.parseSocketAddressAddr(addr), responseFuture.getCause());
 }
        }
        return responseCommand;
 } finally {
        this.responseTable.remove(opaque);
 }
}

在这个方法,我们首先看到上一节讲到的opaqu这个参数,这个参数是一个全局唯一递增的请求标识.当请求得到响应时,则通过这个opaqu参数找到对应的ResponseFuture,将结果通过ReponseFuture#putResponse赋值,并唤醒阻塞线程。在参数准备后,通过我们之前与服务端建立的channel通道,调用writeAndFlush传输到服务端。然后等待服务端的响应。我们还可以看到在此创建除了ResponseFuture这个类的实例,并以opaque为key存入responseTable这个全局静态变量中,用来处理请求当这个请求得到响应时,拿到对应的客户端并唤醒发起请求的客户端阻塞的线程。由此可见当在集群环境下,opaque是会重复的,但是每台机器上的opaque全局唯一。

我们回到之前讲解到的BrokerController#doRegisterBrokerAll方法,将brokerController的路由信息发送至NameServer:

private void doRegisterBrokerAll(boolean checkOrderConfig, boolean oneway,
 TopicConfigSerializeWrapper topicConfigWrapper) {
//通过brokerOuterAPI发送注册请求到NameServer
    List<RegisterBrokerResult> registerBrokerResultList = this.brokerOuterAPI.registerBrokerAll(
        this.brokerConfig.getBrokerClusterName(),
 this.getBrokerAddr(),
 this.brokerConfig.getBrokerName(),
 this.brokerConfig.getBrokerId(),
 this.getHAServerAddr(),
 topicConfigWrapper,
 this.filterServerManager.buildNewFilterServerList(),
 oneway,
 this.brokerConfig.getRegisterBrokerTimeoutMills(),
 this.brokerConfig.isCompressedRegister());
 if (registerBrokerResultList.size() > 0) {
        RegisterBrokerResult registerBrokerResult = registerBrokerResultList.get(0);
 if (registerBrokerResult != null) {
            if (this.updateMasterHAServerAddrPeriodically && registerBrokerResult.getHaServerAddr() != null) {
    this.messageStore.updateHaMasterAddress(registerBrokerResult.getHaServerAddr());
 }
            this.slaveSynchronize.setMasterAddr(registerBrokerResult.getMasterAddr());
 if (checkOrderConfig) {
                this.getTopicConfigManager().updateOrderTopicConfig(registerBrokerResult.getKvTable());
 }
        }
    }
}

从方法中可以看出,实际调用的还是BrokerOuterAPI的registerBrokerAll方法,那么具体请求流程和上面大致一样,只是请求头和请求参数因不同的业务请求而不同,所以我们只要关心客户端发送的是哪些参数,服务端Server是怎么处理当前请求的)以及返回的是什么参数。
注册路由请求返送参数:
image.png
请求头:
brokerAddr,BrokerId,BrokerName,clusterName同上面参数含义一样,新增了一个haServerAddr这个参数,这个参数是由我们配置的BrokerIp2+haListenPort拼接的,具体使用来SLAVE通过该端口号与服务Master进行数据同步,同步的数据只是一些配置信息,比如TopicConfig(路由信息),ConsumerOffset(消费偏移量)等,具体的同步过程后面章节将会讲到。
bodyCrc32是对请求体进行请求体内容进行冗余校验,保证数据在传输过程中的正确性。
请求体:
除了前面讲到的TopicConfigSerializeWrapper路由信息+DataVersion数据版本外,还新增了一个FilterServerList列表(具体作用后面章节详细讲解)
客户端请求发送至NameServer服务端后,NameServer在DefaultRequestProcessor中进行处理,通过上一章节的讲解,我们可以了解到,当一个请求到达服务端后,会调用对应的业务请求的processRequest方法,在NameServer所以业务请求统一由DefaultRequestProcessor处理,那么我们进入到DefaultRequestProcessor#processRequest方法中:

public RemotingCommand processRequest(ChannelHandlerContext ctx,
 RemotingCommand request) throws RemotingCommandException {
    if (ctx != null) {
        log.debug("receive request, {} {} {}",
 request.getCode(),
 RemotingHelper.parseChannelRemoteAddr(ctx.channel()),
 request);
 }
    switch (request.getCode()) {
        case RequestCode.PUT_KV_CONFIG:
            return this.putKVConfig(ctx, request);
 case RequestCode.GET_KV_CONFIG:
            return this.getKVConfig(ctx, request);
 case RequestCode.DELETE_KV_CONFIG:
            return this.deleteKVConfig(ctx, request);
 case RequestCode.QUERY_DATA_VERSION:
            return queryBrokerTopicConfig(ctx, request);
 //注册broker路由信息
 case RequestCode.REGISTER_BROKER:
            Version brokerVersion = MQVersion.value2Version(request.getVersion());
 if (brokerVersion.ordinal() >= MQVersion.Version.V3_0_11.ordinal()) {
                return this.registerBrokerWithFilterServer(ctx, request);
 } else {
                return this.registerBroker(ctx, request);
 }
        case RequestCode.UNREGISTER_BROKER:
            return this.unregisterBroker(ctx, request);
 case RequestCode.GET_ROUTEINFO_BY_TOPIC:
            return this.getRouteInfoByTopic(ctx, request);
 case RequestCode.GET_BROKER_CLUSTER_INFO:
            return this.getBrokerClusterInfo(ctx, request);
 case RequestCode.WIPE_WRITE_PERM_OF_BROKER:
            return this.wipeWritePermOfBroker(ctx, request);
 case RequestCode.GET_ALL_TOPIC_LIST_FROM_NAMESERVER:
            return getAllTopicListFromNameserver(ctx, request);
 case RequestCode.DELETE_TOPIC_IN_NAMESRV:
            return deleteTopicInNamesrv(ctx, request);
 case RequestCode.GET_KVLIST_BY_NAMESPACE:
            return this.getKVListByNamespace(ctx, request);
 case RequestCode.GET_TOPICS_BY_CLUSTER:
            return this.getTopicsByCluster(ctx, request);
 case RequestCode.GET_SYSTEM_TOPIC_LIST_FROM_NS:
            return this.getSystemTopicListFromNs(ctx, request);
 case RequestCode.GET_UNIT_TOPIC_LIST:
            return this.getUnitTopicList(ctx, request);
 case RequestCode.GET_HAS_UNIT_SUB_TOPIC_LIST:
            return this.getHasUnitSubTopicList(ctx, request);
 case RequestCode.GET_HAS_UNIT_SUB_UNUNIT_TOPIC_LIST:
            return this.getHasUnitSubUnUnitTopicList(ctx, request);
 case RequestCode.UPDATE_NAMESRV_CONFIG:
            return this.updateConfig(ctx, request);
 case RequestCode.GET_NAMESRV_CONFIG:
            return this.getConfig(ctx, request);
 default:
            break;
 }
    return null;
}

在这个方法中我们可以看到,对应不同的业务请求code都有相对应的处理逻辑,我们直接定位到REGISTER_BROKER这个请求CODE的处理逻辑:

    //注册broker路由信息
case RequestCode.REGISTER_BROKER:
    Version brokerVersion = MQVersion.value2Version(request.getVersion());
 if (brokerVersion.ordinal() >= MQVersion.Version.V3_0_11.ordinal()) {
        return this.registerBrokerWithFilterServer(ctx, request);
 } else {
        return this.registerBroker(ctx, request);
 }

从这段代码中可以看出,FilterServerList这个请求体参数是在3.0.11版本后才加入的。我们直接进入到registerBrokerWithFilterServer这个方法中:

public RemotingCommand registerBrokerWithFilterServer(ChannelHandlerContext ctx, RemotingCommand request)
    throws RemotingCommandException {
    final RemotingCommand response = RemotingCommand.createResponseCommand(RegisterBrokerResponseHeader.class);
 final RegisterBrokerResponseHeader responseHeader = (RegisterBrokerResponseHeader) response.readCustomHeader();
 final RegisterBrokerRequestHeader requestHeader =
        (RegisterBrokerRequestHeader) request.decodeCommandCustomHeader(RegisterBrokerRequestHeader.class);
 //1.crc差错校验
 if (!checksum(ctx, request, requestHeader)) {
        response.setCode(ResponseCode.SYSTEM_ERROR);
 response.setRemark("crc32 not match");
 return response;
 }
    RegisterBrokerBody registerBrokerBody = new RegisterBrokerBody();
 if (request.getBody() != null) {
        try {
            registerBrokerBody = RegisterBrokerBody.decode(request.getBody(), requestHeader.isCompressed());
 } catch (Exception e) {
            throw new RemotingCommandException("Failed to decode RegisterBrokerBody", e);
 }
    } else {
        registerBrokerBody.getTopicConfigSerializeWrapper().getDataVersion().setCounter(new AtomicLong(0));
 registerBrokerBody.getTopicConfigSerializeWrapper().getDataVersion().setTimestamp(0);
 }
    //2.将请求参数保存到对应的内存数据结构
 RegisterBrokerResult result = this.namesrvController.getRouteInfoManager().registerBroker(
        requestHeader.getClusterName(),
 requestHeader.getBrokerAddr(),
 requestHeader.getBrokerName(),
 requestHeader.getBrokerId(),
 requestHeader.getHaServerAddr(),
 registerBrokerBody.getTopicConfigSerializeWrapper(),
 registerBrokerBody.getFilterServerList(),
 ctx.channel());
 responseHeader.setHaServerAddr(result.getHaServerAddr());
 responseHeader.setMasterAddr(result.getMasterAddr());
 byte[] jsonValue = this.namesrvController.getKvConfigManager().getKVListByNamespace(NamesrvUtil.NAMESPACE_ORDER_TOPIC_CONFIG);
 response.setBody(jsonValue);
 response.setCode(ResponseCode.SUCCESS);
 response.setRemark(null);
 return response;
}

这个方法主要逻辑就是做了crc校验,然后将我们的请求参数和请求体通过RouterInfoManager(路由信息管理类)registerBroker方法进行处理,转换,最后存储在内存中。在registerBroker方法返回结果中,需要着重注意的是haServerAddr与masterAddr这2个参数。
我们进入到RouterInfoManager#registerBroker方法中:|

public RegisterBrokerResult registerBroker(
    final String clusterName,
 final String brokerAddr,
 final String brokerName,
 final long brokerId,
 final String haServerAddr,
 final TopicConfigSerializeWrapper topicConfigWrapper,
 final List<String> filterServerList,
 final Channel channel) {
    RegisterBrokerResult result = new RegisterBrokerResult();
 try {
        try {
            //1.获取写锁,保证只有一个线程进入当前代码块
 this.lock.writeLock().lockInterruptibly();
 //2.集群对应broker列表
 Set<String> brokerNames = this.clusterAddrTable.get(clusterName);
 if (null == brokerNames) {
                brokerNames = new HashSet<String>();
 this.clusterAddrTable.put(clusterName, brokerNames);
 }
            brokerNames.add(brokerName);
 boolean registerFirst = false;
 BrokerData brokerData = this.brokerAddrTable.get(brokerName);
 if (null == brokerData) {
                registerFirst = true;
 //3.保存broker组的一些信息
 brokerData = new BrokerData(clusterName, brokerName, new HashMap<Long, String>());
 this.brokerAddrTable.put(brokerName, brokerData);
 }
            Map<Long, String> brokerAddrsMap = brokerData.getBrokerAddrs();
 Iterator<Entry<Long, String>> it = brokerAddrsMap.entrySet().iterator();
 while (it.hasNext()) {
                Entry<Long, String> item = it.next();
 if (null != brokerAddr && brokerAddr.equals(item.getValue()) && brokerId != item.getKey()) {
                    it.remove();
 }
            }
            String oldAddr = brokerData.getBrokerAddrs().put(brokerId, brokerAddr);
 registerFirst = registerFirst || (null == oldAddr);
 //当前注册路由角色为Master
 if (null != topicConfigWrapper
                && MixAll.MASTER_ID == brokerId) {
                //路由信息不同或第一次注册,则创建与broker对应的队列信息
 if (this.isBrokerTopicConfigChanged(brokerAddr, topicConfigWrapper.getDataVersion())
                    || registerFirst) {
                    ConcurrentMap<String, TopicConfig> tcTable =
                        topicConfigWrapper.getTopicConfigTable();
 if (tcTable != null) {
                        for (Map.Entry<String, TopicConfig> entry : tcTable.entrySet()) {
                            this.createAndUpdateQueueData(brokerName, entry.getValue());
 }
                    }
                }
            }
            //4.保存当前brokerAddr最后一次活跃事件与版本号以及haServerAddr
 BrokerLiveInfo prevBrokerLiveInfo = this.brokerLiveTable.put(brokerAddr,
 new BrokerLiveInfo(
                    System.currentTimeMillis(),
 topicConfigWrapper.getDataVersion(),
 channel,
 haServerAddr));
 if (null == prevBrokerLiveInfo) {
                log.info("new broker registered, {} HAServer: {}", brokerAddr, haServerAddr);
 }
            //处理filterServer
 if (filterServerList != null) {
                if (filterServerList.isEmpty()) {
                    this.filterServerTable.remove(brokerAddr);
 } else {
                    this.filterServerTable.put(brokerAddr, filterServerList);
 }
            }
            //如果当前角色为SLAVE,则查找master注册的brokerAddr和haServerAddr,用来更新SLAVE与Master在建立连接同步配置信息以及消息数据
 if (MixAll.MASTER_ID != brokerId) {
                String masterAddr = brokerData.getBrokerAddrs().get(MixAll.MASTER_ID);
 if (masterAddr != null) {
                    BrokerLiveInfo brokerLiveInfo = this.brokerLiveTable.get(masterAddr);
 if (brokerLiveInfo != null) {
                        result.setHaServerAddr(brokerLiveInfo.getHaServerAddr());
 result.setMasterAddr(masterAddr);
 }
                }
            }
        } finally {
            this.lock.writeLock().unlock();
 }
    } catch (Exception e) {
        log.error("registerBroker Exception", e);
 }
    return result;
}

在这个方法中,我们就可以看到路由信息在NameServer是怎样处理的,也可以从代码中看出clusterName与BrokerName,BrokerName与brokerId是什么关系:
一个集群对应多个brokerName,一个brokerName对应多个broker服务,但是一般只有一个brokerId为0的。
image.png

除了保留了一些broker服务之间的关系和地址之外,NameServer还保存了Topic的配置信息,Topic的配置信息就是由Broker发送过来的TopicConfig这个数据结构转换成QueueData的。具体的转换逻辑是在createAndUpdateQueueData方法中完成的:

private void createAndUpdateQueueData(final String brokerName, final TopicConfig topicConfig) {
    QueueData queueData = new QueueData();
 queueData.setBrokerName(brokerName);
 queueData.setWriteQueueNums(topicConfig.getWriteQueueNums());
 queueData.setReadQueueNums(topicConfig.getReadQueueNums());
 queueData.setPerm(topicConfig.getPerm());
 queueData.setTopicSynFlag(topicConfig.getTopicSysFlag());
 List<QueueData> queueDataList = this.topicQueueTable.get(topicConfig.getTopicName());
 if (null == queueDataList) {
        queueDataList = new LinkedList<QueueData>();
 queueDataList.add(queueData);
 this.topicQueueTable.put(topicConfig.getTopicName(), queueDataList);
 log.info("new topic registered, {} {}", topicConfig.getTopicName(), queueData);
 } else {
        boolean addNewOne = true;
 Iterator<QueueData> it = queueDataList.iterator();
 while (it.hasNext()) {
            QueueData qd = it.next();
 if (qd.getBrokerName().equals(brokerName)) {
                if (qd.equals(queueData)) {
                    addNewOne = false;
 } else {
                    log.info("topic changed, {} OLD: {} NEW: {}", topicConfig.getTopicName(), qd,
 queueData);
 it.remove();
 }
            }
        }
        if (addNewOne) {
            queueDataList.add(queueData);
 }
    }
}

从这个方法中可以看到QueueData主要涵盖的信息有,该topic属于哪一个brokerName的,读队列数,写队列数,读写权限等。然后这些信息保存在nameServer的topicQueueTable这个MAP结构的内存中。key为topic的名字,value为QueueData列表,说明一个topic在不同的brokerName都有一份QueueData配置,相同brokerName的master和slave角色具有相同的配置,因为slave的topicConfig是从master定时同步更新下来的。

评论

暂无评论