[Bug] [DolphinScheduler] Duplicate DAG task relations grow on repeated publication (Dinky 1.2.5) #4588

Search before asking

• I had searched in the issues and found no similar issues.

What happened

Environment

Dinky: 1.2.5
Apache DolphinScheduler: 3.2.2
Apache Flink: 1.20.5

Description

When I repeatedly published tasks from Dinky to an existing DolphinScheduler workflow, the number of DAG task relations increased dramatically. Eventually, the workflow DAG page became very slow or unresponsive.

How I found the issue

I noticed the problem while adding data warehouse tasks to an existing workflow. I inspected the DolphinScheduler metadata tables and compared the total task relation count with the number of unique dependency edges.

Version 25: 10,239 total relations, 19 unique edges, 10,220 duplicates.
Version 26: 20,479 total relations, 20 unique edges, 20,459 duplicates.
Version 27: 40,959 total relations, 21 unique edges, 40,938 duplicates.

The total relation count roughly doubled after each publication, even though only one new unique edge was added each time.

Investigation

I traced the publication flow to SchedulerServiceImpl.java, particularly pushAddTask() and updateProcessDefinition().

In updateProcessDefinition(), Dinky obtains the relation list from dagData.getProcessTaskRelationList() and passes the list to DolphinScheduler without deduplicating it first.

This is a missing defensive check that allows existing duplicate relations to be propagated into the next workflow version. The original source of the duplicated relations still requires further investigation.

Source file (branch 1.2):
https://github.com/DataLinkDC/dinky/blob/1.2/dinky-admin/src/main/java/org/dinky/service/impl/SchedulerServiceImpl.java

Local fix

I modified SchedulerServiceImpl.updateProcessDefinition() to:

1. Deduplicate relations by the pair (preTaskCode, postTaskCode) before updating the workflow.
2. Detect conflicting task versions or relation conditions rather than silently discarding different relations.
3. Log relation counts before and after deduplication.

I recompiled the modified Java class, backed up the original Dinky JAR, and deployed the patched JAR in our environment.

Validation

Dinky logs confirmed that the deduplication logic executed:

DAG relation dedup, before=40959, after=21
DAG relation dedup, before=77, after=13

After subsequent task publication, the current DolphinScheduler workflows showed:

ODS: 21 total relations, 21 unique edges, 0 duplicates.
DIM: 14 total relations, 14 unique edges, 0 duplicates.
DWD: 39 total relations, 39 unique edges, 0 duplicates.

The DWD workflow had previously contained 53 relations with 37 unique edges (16 duplicates). The current workflow now contains 39 unique edges with no duplicates.

Expected behavior

Existing dependency edges should be retained once each. Publishing a new task should add only the necessary new relations, without multiplying existing ones.

Request

Could maintainers review whether defensive relation deduplication and conflict checking should be added to SchedulerServiceImpl.updateProcessDefinition(), or whether the underlying issue should be addressed elsewhere in the Dinky-DolphinScheduler integration?

I can prepare a source-level pull request if this approach is appropriate.

Thank you for maintaining Dinky.

What you expected to happen

When publishing a new task from Dinky to an existing DolphinScheduler workflow, all existing task dependencies should be preserved without duplication.

Only the necessary new task relations should be added. The total number of relations should match the number of unique dependency edges, and the workflow DAG should remain responsive even after multiple task publications.

How to reproduce

When publishing a new task from Dinky to an existing DolphinScheduler workflow, the workflow should preserve all existing task dependencies without creating duplicates.

Only the necessary new task relations should be added. The total number of relations should remain consistent with the number of unique dependency edges, and the workflow DAG should remain responsive.

Anything else

1. Set up Dinky 1.2.5 with Apache DolphinScheduler 3.2.2.
2. Create a DolphinScheduler workflow containing multiple tasks and dependencies.
3. Publish an additional task from Dinky to the existing workflow.
4. Repeat the task publication process several times, adding one task at a time.
5. Check the DolphinScheduler metadata table t_ds_process_task_relation_log and compare the total number of task relations with the number of unique dependency edges.
6. In the affected environment, the number of relations grows rapidly, even though only a small number of new dependencies are added.
7. Open the workflow DAG page in DolphinScheduler. As the number of duplicated relations increases, the page becomes very slow or unresponsive.

Observed results in our environment:

• Version 25: 10,239 relations, 19 unique edges.
• Version 26: 20,479 relations, 20 unique edges.
• Version 27: 40,959 relations, 21 unique edges.

The issue appears to involve duplicate task relations being propagated during workflow updates in SchedulerServiceImpl.updateProcessDefinition().

Note: These steps describe the conditions observed in our environment. Reproduction in a clean installation has not yet been independently confirmed.

Version

1.2.0

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct