Storm客户端提交任务失败原因分析

Easter79
• 阅读 489

storm客户端提交topology失败:

java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException
        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141)
        at backtype.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:176)
        at backtype.storm.StormSubmitter.submitTopologyWithProgressBar(StormSubmitter.java:158)
        at cn.com.tiza.dataquality.service.service.JobService.start(JobService.java:93)
        at cn.com.tiza.dataquality.service.service.JobService.submit(JobService.java:149)
        at cn.com.tiza.dataquality.service.resource.JobResource.execute(JobResource.java:33)
        at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:160)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
        at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)

nimbus.log:

2017-11-17T16:12:27.038+0800 b.s.d.nimbus [WARN] Topology submission exception. (topology name='dq23') #<IllegalArgumentException java.lang.IllegalArgumentException: storm-local/nimbus/inb
ox/stormjar-bc004b80-5f6e-4b94-9505-911511e5cc1f.jar to copy to storm-local/nimbus/stormdist/dq23-94-1510906347 does not exist!>
2017-11-17T16:12:27.038+0800 o.a.t.s.TNonblockingServer [ERROR] Unexpected exception while invoking!
java.lang.IllegalArgumentException: storm-local/nimbus/inbox/stormjar-bc004b80-5f6e-4b94-9505-911511e5cc1f.jar to copy to storm-local/nimbus/stormdist/dq23-94-1510906347 does not exist!
        at backtype.storm.daemon.nimbus$fn__4364.invoke(nimbus.clj:1173) ~[storm-core-0.9.7.jar:0.9.7]
        at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307) ~[storm-core-0.9.7.jar:0.9.7]
        at backtype.storm.daemon.nimbus$fn__4261$exec_fn__1104__auto__$reify__4274.submitTopologyWithOpts(nimbus.clj:953) ~[storm-core-0.9.7.jar:0.9.7]
        at backtype.storm.daemon.nimbus$fn__4261$exec_fn__1104__auto__$reify__4274.submitTopology(nimbus.clj:966) ~[storm-core-0.9.7.jar:0.9.7]
        at backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240) ~[storm-core-0.9.7.jar:0.9.7]
        at backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228) ~[storm-core-0.9.7.jar:0.9.7]
        at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32) ~[storm-core-0.9.7.jar:0.9.7]
        at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34) ~[storm-core-0.9.7.jar:0.9.7]
        at org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632) ~[storm-core-0.9.7.jar:0.9.7]
        at org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201) [storm-core-0.9.7.jar:0.9.7]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]

经过分析nimbus有个clean-inbox的机制来定时清理inbox中的jar文件,并有两个配置项来设置定时策略:

/**
 * How often nimbus should wake the cleanup thread to clean the inbox.
 * @see NIMBUS_INBOX_JAR_EXPIRATION_SECS
 */
public static final String NIMBUS_CLEANUP_INBOX_FREQ_SECS = "nimbus.cleanup.inbox.freq.secs";

/**
 * The length of time a jar file lives in the inbox before being deleted by the cleanup thread.
 *
 * Probably keep this value greater than or equal to NIMBUS_CLEANUP_INBOX_JAR_EXPIRATION_SECS.
 * Note that the time it takes to delete an inbox jar file is going to be somewhat more than
 * NIMBUS_CLEANUP_INBOX_JAR_EXPIRATION_SECS (depending on how often NIMBUS_CLEANUP_FREQ_SECS
 * is set to).
 * @see NIMBUS_CLEANUP_FREQ_SECS
 */
public static final String NIMBUS_INBOX_JAR_EXPIRATION_SECS = "nimbus.inbox.jar.expiration.secs";

NIMBUS_CLEANUP_INBOX_FREQ_SECS: 表示nimbus多久唤醒一次清理线程去进行清理;

NIMBUS_INBOX_JAR_EXPIRATION_SECS:表示jar文件在inbox中存活的时长,在清理线程清理之前如果到期了就会被清理

另一方面,通过storm-core提供的StormSubmitter.submitTopology的方法进行提交任务时,上传jar包的逻辑如下:

    private static String submittedJar = null;

    private static void submitJar(Map conf, ProgressListener listener) {
        if(submittedJar==null) {
            LOG.info("Jar not uploaded to master yet. Submitting jar...");
            String localJar = System.getProperty("storm.jar");
            submittedJar = submitJar(conf, localJar, listener);
        } else {
            LOG.info("Jar already uploaded to master. Not submitting jar.");
        }
    }

只要客户端进程不停,jar包就只上传一次。

所以等一个小时后,jar会被清除,重新提交任务就找不到inbox中的jar文件。

点赞
收藏
评论区
推荐文章
blmius blmius
2年前
MySQL:[Err] 1292 - Incorrect datetime value: ‘0000-00-00 00:00:00‘ for column ‘CREATE_TIME‘ at row 1
文章目录问题用navicat导入数据时,报错:原因这是因为当前的MySQL不支持datetime为0的情况。解决修改sql\mode:sql\mode:SQLMode定义了MySQL应支持的SQL语法、数据校验等,这样可以更容易地在不同的环境中使用MySQL。全局s
Wesley13 Wesley13
2年前
java将前端的json数组字符串转换为列表
记录下在前端通过ajax提交了一个json数组的字符串,在后端如何转换为列表。前端数据转化与请求varcontracts{id:'1',name:'yanggb合同1'},{id:'2',name:'yanggb合同2'},{id:'3',name:'yang
Wesley13 Wesley13
2年前
HADOOP性能优化和运维
集群中任意一个节点都可以被用来提交认任务,虽然通常我们使用master节点提交任务。HADOOP客户端不参与计算和存储,专门用来上传下载文件和提交任务。性能优化4大块:!(https://static.oschina.net/uploads/space/2017/0406/111251_QSBe_192561.png)具体优化如下:1.选
Stella981 Stella981
2年前
Android So动态加载 优雅实现与原理分析
背景:漫品Android客户端集成适配转换功能(基于目标识别(So库35M)和人脸识别库(5M)),导致apk体积50M左右,为优化客户端体验,决定实现So文件动态加载.!(https://oscimg.oschina.net/oscnet/00d1ff90e4b34869664fef59e3ec3fdd20b.png)点击上方“蓝字”关注我
Wesley13 Wesley13
2年前
mysql设置时区
mysql设置时区mysql\_query("SETtime\_zone'8:00'")ordie('时区设置失败,请联系管理员!');中国在东8区所以加8方法二:selectcount(user\_id)asdevice,CONVERT\_TZ(FROM\_UNIXTIME(reg\_time),'08:00','0
Stella981 Stella981
2年前
HTML5新增input标签属性
一.inputtype属性1<formaction""2邮箱<inputtype"email"name""id""<inputtype"submit"value"提交"<br/<br/3手机号码<inputtype"tel"name
Easter79 Easter79
2年前
Storm VS Flink ——性能对比
!(https://oscimg.oschina.net/oscnet/2cec00eb2dccf5fdec8def77207da253a86.jpg)1.背景ApacheFlink和ApacheStorm是当前业界广泛使用的两个分布式实时计算框架。其中ApacheStorm(以下简称“Storm”)在美团点评实时
Easter79 Easter79
2年前
Storm 性能优化
原文地址:https://www.jianshu.com/p/f645eb7944b0目录1.场景假设2.调优步骤和方法3.Storm的部分特性4.Storm并行度5.Storm消息机制6.StormUI解析7.性能优化场景假设在介绍Storm的
Easter79 Easter79
2年前
Storm消息处理可靠性保证
Storm可以保证每一个从spout发出的消息能被完全处理。本章描述storm是如何完成这个保证以及用户如何从storm的可靠性能力获益的。消息“完全处理”的含义一个tuple从spout发出后可能会触发成千上万的tuple基于它而创建。以workcount的topology为例考虑下:TopologyBuild
Wesley13 Wesley13
2年前
Java多线程导致的的一个事物性问题
业务场景我们现在有一个类似于文件上传的功能,各个子站点接受业务,业务上传文件,各个子站点的文件需要提交到总站点保存,文件是按批次提交到总站点的,也就是说,一个批次下面约有几百个文件。      考虑到白天提交这么多文件会影响到子站点其他系统带宽,我们将分站点的文件提交到总站点这个操作过程独立出来,放到晚上来做,具体时间是晚上7:00到早上7:00。
Easter79
Easter79
Lv1
今生可爱与温柔,每一样都不能少。
文章
2.8k
粉丝
5
获赞
1.2k