解说Dolpinscheduler 3.2.2-prepare分支(相比3.2.1分支)

2024-02-07 开始看

1、Add Server HealthIndicator (#15274)

其实就是alert/api/master/worker 增加健康标识metrics指标统计

2、Improvement Add UT for end time check when force success task(#15562)

优化UT而已

3、merge schema 330 into 321 and change docs (#15582)

升级sql整理修改

4、release: change 3.2.1 docs (#15585)

修改文档

5、Fix switch task will throw exception when there is not parameters (#15594)

说白了,其实就是想说SwithTask不用全部变量和本地变量也能跑,注意 : 吐槽一句,SwitchTask有本地变量么(不能设置啊),你判断TaskInstance varPool(varParams),永远为true

image.png

流程定义如下 :
image.png
C节点是这样的配置 :
image.png
流程实例的结果 :
image.png

注意 : 发现流程实例是成功状态,SwitchTask是失败状态,后面任务确实成功状态,其实就是上面的BUG修复那里抛了一个异常

不相信可以看SwitchTask任务日志如下 :

INFO] 2024-06-03 11:14:16.468 +0800 - Begin to execute 1 condition: "A" == "A" 
[ERROR] 2024-06-03 11:14:16.706 +0800 - Task execute failed, due to meet an exception
java.lang.IllegalArgumentException: globalParams and varParams are both empty, please check it.
    at org.apache.dolphinscheduler.server.master.utils.SwitchTaskUtils.generateContentWithTaskParams(SwitchTaskUtils.java:69)
    at org.apache.dolphinscheduler.server.master.runner.task.switchtask.SwitchLogicTask.calculateConditionResult(SwitchLogicTask.java:109)
    at org.apache.dolphinscheduler.server.master.runner.task.switchtask.SwitchLogicTask.handle(SwitchLogicTask.java:72)
    at org.apache.dolphinscheduler.server.master.runner.execute.SyncMasterTaskExecutor.executeTask(SyncMasterTaskExecutor.java:44)
    at org.apache.dolphinscheduler.server.master.runner.execute.MasterTaskExecutor.run(MasterTaskExecutor.java:110)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[INFO] 2024-06-03 11:14:16.707 +0800 - ***********************************************************************************************
[INFO] 2024-06-03 11:14:16.707 +0800 - *********************************  Finalize task instance  ************************************
[INFO] 2024-06-03 11:14:16.707 +0800 - ***********************************************************************************************
[ERROR] 2024-06-03 11:14:16.707 +0800 - Get a exception when execute the task, will try to cancel the task
java.lang.IllegalArgumentException: globalParams and varParams are both empty, please check it.
    at org.apache.dolphinscheduler.server.master.utils.SwitchTaskUtils.generateContentWithTaskParams(SwitchTaskUtils.java:69)
    at org.apache.dolphinscheduler.server.master.runner.task.switchtask.SwitchLogicTask.calculateConditionResult(SwitchLogicTask.java:109)
    at org.apache.dolphinscheduler.server.master.runner.task.switchtask.SwitchLogicTask.handle(SwitchLogicTask.java:72)
    at org.apache.dolphinscheduler.server.master.runner.execute.SyncMasterTaskExecutor.executeTask(SyncMasterTaskExecutor.java:44)
    at org.apache.dolphinscheduler.server.master.runner.execute.MasterTaskExecutor.run(MasterTaskExecutor.java:110)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

6、Feature-14863 Support to manage what worker groups can be used for a project. (#15600)

其实想解决的问题很简单,就是之前安全中心 -> Worker分组管理的所有worker列表,Task在选择的Worker分组的时候都可以看得见,供选择,如下 :
image.png

此 PR 解决的其实就是假如安全中心创建了很多Worker分组,项目是可以绑定Worker分组的,如下,为什么这么干呢?其实很简单,因为比如说管理员是可以对项目授权的,比如说管理员给你授权了项目A,项目A只能在test1 worker分组上跑,形成worker分组的权限控制
image.png
image.png

7、Fix recover pause workflowInstance may generate failover taskInstance (#15611)

image.png
其实就是想这样实现 :

if (state == TaskExecutionStatus.SUBMITTED_SUCCESS 
    || state == TaskExecutionStatus.DELAY_EXECUTION
    || state == TaskExecutionStatus.DISPATCH) {
    // The taskInstance is not in running, directly takeover it
} else if (tryToTakeOverTaskInstance(existTaskInstance)) {
    log.info("Success take over task {}", existTaskInstance.getName());
    continue;
} else {
    // set the task instance state to fault tolerance
    existTaskInstance.setFlag(Flag.NO);
    existTaskInstance.setState(TaskExecutionStatus.NEED_FAULT_TOLERANCE);
    releaseTaskGroupIfNeeded(existTaskInstance);

    validTaskMap.remove(existTaskInstance.getTaskCode());
    taskInstanceDao.updateById(existTaskInstance);
    existTaskInstance = cloneTolerantTaskInstance(existTaskInstance);
    log.info("task {} cannot be take over will generate a tolerant task instance",
            existTaskInstance.getName());
}

看代码意思其实就是多了一个如果是DISPATCH状态的话,就会重新进行任务下发执行。正常情况下,这个时候下游任务应该是暂停状态才对

9、Bug list paging missing totalpage (#15619)

image.png
感觉没有我解决的好,哈哈 :

public Integer getTotalPage() {
    if (pageSize == null || pageSize == 0) {
        pageSize = 10;
    }

    this.totalPage = (int) Math.ceil((double) this.total / (double) this.pageSize);
    return this.totalPage;
}

10、Fix out parameter may loss (#15617)和 Fix workflow instance restart failed due to duplicate key in varpool (#16001)

其实这两个PR算是一个问题的解决,就是为了解决输出参数,'${setValue(output=1)}' 这样的写法,注意1位置不能使用变量进行替换,这也有点low。看源码其实就是硬解析,然后放入到上游taskInstance varPool中(存在数据库中)
org.apache.dolphinscheduler.plugin.task.api.AbstractCommandExecutor#parseProcessOutput

private void parseProcessOutput(Process process) {
        // todo: remove this this thread pool.
        ExecutorService getOutputLogService = ThreadUtils
                .newSingleDaemonScheduledExecutorService("ResolveOutputLog-thread-" + taskRequest.getTaskName());
        getOutputLogService.execute(() -> {
            TaskOutputParameterParser taskOutputParameterParser = new TaskOutputParameterParser();
            try (BufferedReader inReader = new BufferedReader(new InputStreamReader(process.getInputStream()))) {
                LogUtils.setTaskInstanceLogFullPathMDC(taskRequest.getLogPath());
                String line;
                while ((line = inReader.readLine()) != null) {
                    logBuffer.add(line);
                    // TODO 主要是这里的代码
                    taskOutputParameterParser.appendParseLog(line);
                }
                processLogOutputIsSuccess = true;
            } catch (Exception e) {
                log.error("Parse var pool error", e);
                processLogOutputIsSuccess = true;
            } finally {
                LogUtils.removeTaskInstanceLogFullPathMDC();
            }
            // TODO 这里是不是赋值了
            taskOutputParams = taskOutputParameterParser.getTaskOutputParams();
        });
        ...............
}

org.apache.dolphinscheduler.plugin.task.api.parser.TaskOutputParameterParser#appendParseLog

Pair<String, String> keyValue = parseOutputParam(String.join("\n", currentTaskOutputParam));

org.apache.dolphinscheduler.plugin.task.api.parser.TaskOutputParameterParser#parseOutputParam

protected Pair<String, String> parseOutputParam(String outputParam) {
        if (StringUtils.isEmpty(outputParam)) {
            log.info("The task output param is empty");
            return ImmutablePair.nullPair();
        }
        if ((!outputParam.startsWith("${setValue(") && !outputParam.startsWith("#{setValue("))
                || !outputParam.endsWith(")}")) {
            log.info("The task output param {} should start with '${setValue(' or '#{setValue(' and end with ')}'",
                    outputParam);
            return ImmutablePair.nullPair();
        }
        String keyValueExpression = outputParam.substring(11, outputParam.length() - 2);
        if (!keyValueExpression.contains("=")) {
            log.warn("The task output param {} should composite with key=value", outputParam);
            return ImmutablePair.nullPair();
        }

        String[] keyValue = keyValueExpression.split("=", 2);
        return ImmutablePair.of(keyValue[0], keyValue[1]);
    }

11、[TEST] add task definition log service test (#15631)

添加任务定义日志测试类,_log其实就是会记录历史版本

12、Fix Hive jdbc url (#15627)

image.png
其实就是把hive jdbc url之后如果有参数是用;进行分割的,而不是?分割的

13、Improvement add PageInfo UT (#15622)

添加PageInfo的测试类而已

14、[Doc] fix CNCF logo (#15634)

修改logo而已

15、Update SeatunnelTask.java (#15621)

待更新。。。,如感兴趣,点赞加关注,谢谢!!!

评论

暂无评论