dex: enforce execution timeout for activities by mehab · Pull Request #6741 · DependencyTrack/dependency-track

mehab · 2026-07-17T14:05:14Z

Description

Adds an optional per-attempt execution timeout to dex activities, and enables it for policy evaluation.

Activities may now be registered with an executionTimeout in addition to their lock timeout. ActivityTaskWorker enforces it: when a single attempt exceeds the timeout, the execution is cancelled (thread interrupt) and the task fails, to be retried according to its retry policy, or terminally once attempts are exhausted.

The timeout is opt-in. Activities registered without one behave exactly as before. It is enabled for eval-project-policies with a default of 10 minutes, configurable via dt.dex.engine.activity.eval-project-policies.execution-timeout-ms, so that an expensive or pathological policy can no longer occupy a worker indefinitely.

Addressed Issue

Relates to: #6740

Additional Details

Test case provided for replicating the scenario where no heartbeat and lock time out results in another pod picking up the same task.
https://github.com/mehab/dependency-track/blob/feature/policy-eval-heartbeat-max-duration/apiserver/src/test/java/org/dependencytrack/policy/PolicyEvaluationLockThrashTest.java#L81

Checklist

I have read and understand the contributing guidelines
This PR fixes a defect, and I have provided tests to verify that the fix is effective
This PR implements an enhancement, and I have provided tests to verify that it works as intended
This PR introduces changes to the database model, and I have updated the migration changelog accordingly
This PR introduces new or alters existing behavior, and I have updated the documentation accordingly
This PR is a substantial change (per the ADR criteria), and I have added an ADR under docs/adr/

Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com>

owasp-dt-bot · 2026-07-17T14:05:34Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

codacy-production · 2026-07-17T14:06:41Z

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Coverage 85.71% diff coverage · +0.14% coverage variation

Metric Results

Coverage variation ✅ +0.14% coverage variation (-1.00%)

Diff coverage ✅ 85.71% diff coverage (70.00%)

View coverage diff in Codacy

Coverage variation details

Coverable lines Covered lines Coverage

Common ancestor commit (444f433) Report Missing Report Missing Report Missing

Head commit (87ac7c0) 42868 (+31) 37291 (+89) 86.99% (+0.14%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

Coverable lines Covered lines Diff coverage

Pull request (#6741) 35 30 85.71%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

¹ Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct.

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer}
_{TIP This summary will be updated as you push new changes.}

nscuro

Thanks for the PR!

The heartbeating itself looks good, but I think we can make the deadline portion a bit cleaner, such that it can work for all activities.

In ActivityTaskWorker, task execution is offloaded to a (virtual) thread:

dependency-track/dex/engine/src/main/java/org/dependencytrack/dex/engine/ActivityTaskWorker.java

Lines 114 to 122 in d3429f4

    
           final Future<Object> future; 
        
           try { 
        
               future = executionExecutor.submit( 
        
                       () -> activityMetadata.executor().execute(ctx, arg)); 
        
           } catch (RejectedExecutionException e) { 
        
               logger.debug("Execution executor is shut down; Abandoning task"); 
        
               abandon(task); 
        
               return; 
        
           }

The Future returned by the above submit is then "awaited" via Future#get, and the task's outcome is recorded via engine.onTaskEvent:

dependency-track/dex/engine/src/main/java/org/dependencytrack/dex/engine/ActivityTaskWorker.java

Lines 124 to 167 in d3429f4

    
                       try { 
        
                           final Object activityResult = future.get(); 
        
                           final Payload result; 
        
                           try { 
        
                               result = activityMetadata.resultConverter().convertToPayload(activityResult); 
        
                           } catch (RuntimeException e) { 
        
                               logger.warn("Failed to serialize activity result; Failing task terminally", e); 
        
                               engine.onTaskEvent( 
        
                                       new ActivityTaskFailedEvent( 
        
                                               task, 
        
                                               new TerminalApplicationFailureException(e), 
        
                                               /* retryAt */ null)); 
        
                               return; 
        
                           } 
        
                           engine.onTaskEvent(new ActivityTaskCompletedEvent(task, result)); 
        
                       } catch (ExecutionException e) { 
        
                           final Throwable cause = e.getCause(); 
        
                           if (cause instanceof InterruptedException || executionExecutor.isShutdown()) { 
        
                               logger.debug("Activity was interrupted or worker is shutting down; abandoning task"); 
        
                               abandon(task); 
        
                           } else if (cause instanceof TaskLockLostException) { 
        
                               logger.warn(""" 
        
                                       Activity task lock was lost, likely because execution took longer than \ 
        
                                       the lock timeout without a heartbeat; discarding this execution as \ 
        
                                       another worker has taken over the task"""); 
        
                           } else { 
        
                               final Instant retryAt = computeRetryAt(task, cause); 
        
                               if (retryAt == null) { 
        
                                   logger.warn("Activity failed terminally after {} attempt(s)", task.attempt(), cause); 
        
                               } else { 
        
                                   logger.warn( 
        
                                           "Activity failed; Next retry due at {} (attempt {}/{})", 
        
                                           retryAt, task.attempt() + 1, task.retryPolicy().maxAttempts(), cause); 
        
                               } 
        
                               engine.onTaskEvent(new ActivityTaskFailedEvent(task, cause, retryAt)); 
        
                           } 
        
                       } catch (InterruptedException e) { 
        
                           logger.debug("Interrupted while waiting for activity execution to complete; Abandoning task"); 
        
                           future.cancel(true); 
        
                           abandon(task); 
        
                           Thread.currentThread().interrupt(); 
        
                       } 
        
                   } 
        
               }

Future#get accepts a timeout via Future#get(long, TimeUnit). This is where the deadline should go. The invocation will throw a TimeoutException if the deadline is exceeded. When this happens, we cancel the Future, which will interrupt its thread.

Something like this:

final Duration executionTimeout = activityMetadata.executionTimeout();
try {
    final Object activityResult = executionTimeout != null
            ? future.get(executionTimeout.toMillis(), TimeUnit.MILLISECONDS)
            : future.get();
    // ...
} catch (TimeoutException e) {
    future.cancel(/* interruptIfRunning */ true);
    // Report failure, computeRetryAt, etc.
} catch (ExecutionException e) {
    // ...
} catch (InterruptedException e) {
    // ...
}

The benefit is that interruption will also work when the activity is blocked on I/O (e.g. DB queries) and activity doesn't have to actively check "did I exceed my deadline?"

Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com>

…tion properties if not set by users Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com>

Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com>

nscuro · 2026-07-24T11:40:21Z

@mehab In light of #6795 I've been thinking about solving the heartbeating differently. As it is right now, heartbeating needs to be invoked manually so ctx.maybeHeartbeat() "bleeds" everywhere, and everyone needs to remember calling it.

I've been thinking to do heartbeats automatically in the background instead, so the manual invocations would disappear entirely. The work in this PR around execution timeouts already solves one part of your problem (activities running too long), so I think it's worth shipping that in isolation, and handle the heartbeating part separately.

So my suggestion would be:

Address the PR comments around execution timeouts (e.g. the property naming)
Revert the heartbeat changes
Get this merged

The heartbeating would then be an immediate follow-up.

WDYT?

nscuro · 2026-07-24T12:09:57Z

@mehab I added a draft ADR for the heartbeating solution here: #6807

mehab · 2026-07-25T07:05:51Z

Sounds good @nscuro i will check it on tuesday and complete the changes as you suggest with automated heartbeats. The solution is definitely cleaner than heartbeat tracking by all activities manually.

Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com>

nscuro

Thanks!

FYI I updated title and description of the PR to reflect the final state, not what was originally proposed.

mehab · 2026-07-29T04:40:46Z

Thanks niklas 😊

* changes for issue number: 6740 created upstream Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com> * addressed reiew comments on PR Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com> * conig get value is not optional. Its default value comes from application properties if not set by users Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com> * fixed breaking test Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com> * addressed pr review comments Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com> --------- Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com> (cherry picked from commit 14ba4e3) Signed-off-by: nscuro <nscuro@protonmail.com>

changes for issue number: 6740 created upstream

ba83e7c

Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com>

mehab changed the title ~~changes for issue number: 6740 created upstream~~ Add heartbeat for policy evaluation with a configurable max timeout Jul 17, 2026

nscuro requested changes Jul 17, 2026

View reviewed changes

nscuro added this to the 5.x milestone Jul 17, 2026

nscuro added the enhancement New feature or request label Jul 17, 2026

mehab added 2 commits July 17, 2026 17:30

addressed reiew comments on PR

252ed52

Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com>

conig get value is not optional. Its default value comes from applica…

eb17895

…tion properties if not set by users Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com>

mehab requested a review from nscuro July 17, 2026 16:41

fixed breaking test

65378ac

Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com>

nscuro requested changes Jul 21, 2026

View reviewed changes

addressed pr review comments

87ac7c0

Signed-off-by: Meha Bhargava <meha.bhargava2@gmail.com>

mehab requested a review from nscuro July 28, 2026 11:04

nscuro changed the title ~~Add heartbeat for policy evaluation with a configurable max timeout~~ dex: enforce execution timeout for activities Jul 28, 2026

nscuro modified the milestones: 5.x, 5.1 Jul 28, 2026

nscuro approved these changes Jul 28, 2026

View reviewed changes

nscuro merged commit 14ba4e3 into DependencyTrack:main Jul 28, 2026
20 checks passed

This was referenced Jul 28, 2026

dex: define default activity execution timeout #6836

Merged

Improve responsiveness to thread interruptions #6837

Merged

nscuro added the backport/5.0.4 label Jul 30, 2026

nscuro mentioned this pull request Jul 30, 2026

Backport: dex: enforce execution timeout for activities #6861

Merged

2 tasks done

Navigation Menu

Repository navigation

dex: enforce execution timeout for activities - #6741

dex: enforce execution timeout for activities#6741
nscuro merged 5 commits into
DependencyTrack:mainfrom
mehab:feature/policy-eval-heartbeat-max-duration

mehab commented Jul 17, 2026 •

edited by nscuro

Loading

Uh oh!

owasp-dt-bot commented Jul 17, 2026 •

edited

Loading

Uh oh!

codacy-production Bot commented Jul 17, 2026 •

edited

Loading

Uh oh!

nscuro left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nscuro commented Jul 24, 2026

Uh oh!

nscuro commented Jul 24, 2026

Uh oh!

mehab commented Jul 25, 2026

Uh oh!

nscuro left a comment

Uh oh!

Uh oh!

mehab commented Jul 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	final Future<Object> future;
	try {
	future = executionExecutor.submit(
	() -> activityMetadata.executor().execute(ctx, arg));
	} catch (RejectedExecutionException e) {
	logger.debug("Execution executor is shut down; Abandoning task");
	abandon(task);
	return;
	}

	try {
	final Object activityResult = future.get();
	final Payload result;
	try {
	result = activityMetadata.resultConverter().convertToPayload(activityResult);
	} catch (RuntimeException e) {
	logger.warn("Failed to serialize activity result; Failing task terminally", e);
	engine.onTaskEvent(
	new ActivityTaskFailedEvent(
	task,
	new TerminalApplicationFailureException(e),
	/* retryAt */ null));
	return;
	}
	engine.onTaskEvent(new ActivityTaskCompletedEvent(task, result));
	} catch (ExecutionException e) {
	final Throwable cause = e.getCause();
	if (cause instanceof InterruptedException \|\| executionExecutor.isShutdown()) {
	logger.debug("Activity was interrupted or worker is shutting down; abandoning task");
	abandon(task);
	} else if (cause instanceof TaskLockLostException) {
	logger.warn("""
	Activity task lock was lost, likely because execution took longer than \
	the lock timeout without a heartbeat; discarding this execution as \
	another worker has taken over the task""");
	} else {
	final Instant retryAt = computeRetryAt(task, cause);
	if (retryAt == null) {
	logger.warn("Activity failed terminally after {} attempt(s)", task.attempt(), cause);
	} else {
	logger.warn(
	"Activity failed; Next retry due at {} (attempt {}/{})",
	retryAt, task.attempt() + 1, task.retryPolicy().maxAttempts(), cause);
	}
	engine.onTaskEvent(new ActivityTaskFailedEvent(task, cause, retryAt));
	}
	} catch (InterruptedException e) {
	logger.debug("Interrupted while waiting for activity execution to complete; Abandoning task");
	future.cancel(true);
	abandon(task);
	Thread.currentThread().interrupt();
	}
	}
	}

Conversation

mehab commented Jul 17, 2026 • edited by nscuro Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Addressed Issue

Additional Details

Checklist

Uh oh!

owasp-dt-bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

codacy-production Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Up to standards ✅

Uh oh!

nscuro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nscuro commented Jul 24, 2026

Uh oh!

nscuro commented Jul 24, 2026

Uh oh!

mehab commented Jul 25, 2026

Uh oh!

nscuro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mehab commented Jul 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mehab commented Jul 17, 2026 •

edited by nscuro

Loading

owasp-dt-bot commented Jul 17, 2026 •

edited

Loading

codacy-production Bot commented Jul 17, 2026 •

edited

Loading