解说Dolpinscheduler 3.2.2-prepare分支(相比3.2.1分支)
文章目录
2024-02-07 开始看
1、Add Server HealthIndicator (#15274)
其实就是alert/api/master/worker 增加健康标识metrics指标统计
2、Improvement Add UT for end time check when force success task(#15562)
优化UT而已
3、merge schema 330 into 321 and change docs (#15582)
升级sql整理修改
4、release: change 3.2.1 docs (#15585)
修改文档
5、Fix switch task will throw exception when there is not parameters (#15594)
说白了,其实就是想说SwithTask不用全部变量和本地变量也能跑,注意 : 吐槽一句,SwitchTask有本地变量么(不能设置啊),你判断TaskInstance varPool(varParams),永远为true

流程定义如下 :
C节点是这样的配置 :
流程实例的结果 :
注意 : 发现流程实例是成功状态,SwitchTask是失败状态,后面任务确实成功状态,其实就是上面的BUG修复那里抛了一个异常
不相信可以看SwitchTask任务日志如下 :
INFO] 2024-06-03 11:14:16.468 +0800 - Begin to execute 1 condition: "A" == "A"
[ERROR] 2024-06-03 11:14:16.706 +0800 - Task execute failed, due to meet an exception
java.lang.IllegalArgumentException: globalParams and varParams are both empty, please check it.
at org.apache.dolphinscheduler.server.master.utils.SwitchTaskUtils.generateContentWithTaskParams(SwitchTaskUtils.java:69)
at org.apache.dolphinscheduler.server.master.runner.task.switchtask.SwitchLogicTask.calculateConditionResult(SwitchLogicTask.java:109)
at org.apache.dolphinscheduler.server.master.runner.task.switchtask.SwitchLogicTask.handle(SwitchLogicTask.java:72)
at org.apache.dolphinscheduler.server.master.runner.execute.SyncMasterTaskExecutor.executeTask(SyncMasterTaskExecutor.java:44)
at org.apache.dolphinscheduler.server.master.runner.execute.MasterTaskExecutor.run(MasterTaskExecutor.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[INFO] 2024-06-03 11:14:16.707 +0800 - ***********************************************************************************************
[INFO] 2024-06-03 11:14:16.707 +0800 - ********************************* Finalize task instance ************************************
[INFO] 2024-06-03 11:14:16.707 +0800 - ***********************************************************************************************
[ERROR] 2024-06-03 11:14:16.707 +0800 - Get a exception when execute the task, will try to cancel the task
java.lang.IllegalArgumentException: globalParams and varParams are both empty, please check it.
at org.apache.dolphinscheduler.server.master.utils.SwitchTaskUtils.generateContentWithTaskParams(SwitchTaskUtils.java:69)
at org.apache.dolphinscheduler.server.master.runner.task.switchtask.SwitchLogicTask.calculateConditionResult(SwitchLogicTask.java:109)
at org.apache.dolphinscheduler.server.master.runner.task.switchtask.SwitchLogicTask.handle(SwitchLogicTask.java:72)
at org.apache.dolphinscheduler.server.master.runner.execute.SyncMasterTaskExecutor.executeTask(SyncMasterTaskExecutor.java:44)
at org.apache.dolphinscheduler.server.master.runner.execute.MasterTaskExecutor.run(MasterTaskExecutor.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)6、Feature-14863 Support to manage what worker groups can be used for a project. (#15600)
其实想解决的问题很简单,就是之前安全中心 -> Worker分组管理的所有worker列表,Task在选择的Worker分组的时候都可以看得见,供选择,如下 : 
此 PR 解决的其实就是假如安全中心创建了很多Worker分组,项目是可以绑定Worker分组的,如下,为什么这么干呢?其实很简单,因为比如说管理员是可以对项目授权的,比如说管理员给你授权了项目A,项目A只能在test1 worker分组上跑,形成worker分组的权限控制

7、Fix recover pause workflowInstance may generate failover taskInstance (#15611)

其实就是想这样实现 :
if (state == TaskExecutionStatus.SUBMITTED_SUCCESS
|| state == TaskExecutionStatus.DELAY_EXECUTION
|| state == TaskExecutionStatus.DISPATCH) {
// The taskInstance is not in running, directly takeover it
} else if (tryToTakeOverTaskInstance(existTaskInstance)) {
log.info("Success take over task {}", existTaskInstance.getName());
continue;
} else {
// set the task instance state to fault tolerance
existTaskInstance.setFlag(Flag.NO);
existTaskInstance.setState(TaskExecutionStatus.NEED_FAULT_TOLERANCE);
releaseTaskGroupIfNeeded(existTaskInstance);
validTaskMap.remove(existTaskInstance.getTaskCode());
taskInstanceDao.updateById(existTaskInstance);
existTaskInstance = cloneTolerantTaskInstance(existTaskInstance);
log.info("task {} cannot be take over will generate a tolerant task instance",
existTaskInstance.getName());
}看代码意思其实就是多了一个如果是DISPATCH状态的话,就会重新进行任务下发执行。正常情况下,这个时候下游任务应该是暂停状态才对
9、Bug list paging missing totalpage (#15619)

感觉没有我解决的好,哈哈 :
public Integer getTotalPage() {
if (pageSize == null || pageSize == 0) {
pageSize = 10;
}
this.totalPage = (int) Math.ceil((double) this.total / (double) this.pageSize);
return this.totalPage;
}10、Fix out parameter may loss (#15617)和 Fix workflow instance restart failed due to duplicate key in varpool (#16001)
其实这两个PR算是一个问题的解决,就是为了解决输出参数,'${setValue(output=1)}' 这样的写法,注意1位置不能使用变量进行替换,这也有点low。看源码其实就是硬解析,然后放入到上游taskInstance varPool中(存在数据库中)
org.apache.dolphinscheduler.plugin.task.api.AbstractCommandExecutor#parseProcessOutput
private void parseProcessOutput(Process process) {
// todo: remove this this thread pool.
ExecutorService getOutputLogService = ThreadUtils
.newSingleDaemonScheduledExecutorService("ResolveOutputLog-thread-" + taskRequest.getTaskName());
getOutputLogService.execute(() -> {
TaskOutputParameterParser taskOutputParameterParser = new TaskOutputParameterParser();
try (BufferedReader inReader = new BufferedReader(new InputStreamReader(process.getInputStream()))) {
LogUtils.setTaskInstanceLogFullPathMDC(taskRequest.getLogPath());
String line;
while ((line = inReader.readLine()) != null) {
logBuffer.add(line);
// TODO 主要是这里的代码
taskOutputParameterParser.appendParseLog(line);
}
processLogOutputIsSuccess = true;
} catch (Exception e) {
log.error("Parse var pool error", e);
processLogOutputIsSuccess = true;
} finally {
LogUtils.removeTaskInstanceLogFullPathMDC();
}
// TODO 这里是不是赋值了
taskOutputParams = taskOutputParameterParser.getTaskOutputParams();
});
...............
}org.apache.dolphinscheduler.plugin.task.api.parser.TaskOutputParameterParser#appendParseLog
Pair<String, String> keyValue = parseOutputParam(String.join("\n", currentTaskOutputParam));org.apache.dolphinscheduler.plugin.task.api.parser.TaskOutputParameterParser#parseOutputParam
protected Pair<String, String> parseOutputParam(String outputParam) {
if (StringUtils.isEmpty(outputParam)) {
log.info("The task output param is empty");
return ImmutablePair.nullPair();
}
if ((!outputParam.startsWith("${setValue(") && !outputParam.startsWith("#{setValue("))
|| !outputParam.endsWith(")}")) {
log.info("The task output param {} should start with '${setValue(' or '#{setValue(' and end with ')}'",
outputParam);
return ImmutablePair.nullPair();
}
String keyValueExpression = outputParam.substring(11, outputParam.length() - 2);
if (!keyValueExpression.contains("=")) {
log.warn("The task output param {} should composite with key=value", outputParam);
return ImmutablePair.nullPair();
}
String[] keyValue = keyValueExpression.split("=", 2);
return ImmutablePair.of(keyValue[0], keyValue[1]);
}11、[TEST] add task definition log service test (#15631)
添加任务定义日志测试类,_log其实就是会记录历史版本
12、Fix Hive jdbc url (#15627)

其实就是把hive jdbc url之后如果有参数是用;进行分割的,而不是?分割的
13、Improvement add PageInfo UT (#15622)
添加PageInfo的测试类而已
14、[Doc] fix CNCF logo (#15634)
修改logo而已
15、Update SeatunnelTask.java (#15621)
待更新。。。,如感兴趣,点赞加关注,谢谢!!!
评论
登录后参与技术讨论,分享你的经验与见解。
登录 / 注册暂无评论