Skip to content

Backport: Move vuln data source mirroring off JDO - #7384

Merged
nscuro merged 1 commit into
DependencyTrack:5.1.xfrom
nscuro:backport-pr-7382
Sep 20, 2026
Merged

nscuro merged 1 commit into
DependencyTrack:5.1.xfrom
nscuro:backport-pr-7382

Conversation

@nscuro

@nscuro nscuro commented Sep 20, 2026

Copy link
Copy Markdown
Member

Description

Refactors the mirror-vuln-data-source activity to use JDBI instead of JDO. This enables proper use of batch operations which significantly slashes database round trips and consequently wall clock time.

Addressed Issue

Fixes #5934
Fixes #6971
Fixes #7346
Backports #7382

Additional Details

N/A

Checklist

  • I have read and understand the contributing guidelines
  • This PR fixes a defect, and I have provided tests to verify that the fix is effective
  • This PR implements an enhancement, and I have provided tests to verify that it works as intended
  • This PR introduces changes to the database model, and I have updated the migration changelog accordingly
  • This PR introduces new or alters existing behavior, and I have updated the documentation accordingly
  • This PR is a substantial change (per the ADR criteria), and I have added an ADR under docs/adr/

Refactors the `mirror-vuln-data-source` activity to use JDBI instead of JDO. This enables proper use of batch operations which significantly slashes database round trips and consequently wall clock time.

Having tested this with OSV's Ubuntu ecosystem, the new implementation issues over 1000x fewer database queries.

The old implementation also showed drastic degradation after ~30k synchronized vulnerabilities. I was able to observe throughput dropping from ~210 vulns per second to ~11/s. This was caused by queries on the `VULNERABLESOFTWARE` table that happened to match large numbers of records, e.g. those targeting the `linux` package. These queries were executed once per affected version. And naturally during a mirror run, the data already present in the DB grows, making the slowdown worse as the run progressed. This is likely the root cause of DependencyTrack#5934.

With the refactored code, processing the OSV Ubuntu ecosystem locally is stable at ~280 vulns per second. For Chainguard it's ~3200/s. The difference between the two is caused by Ubuntu publishing few(-er) advisories with *many* affected versions, and Chainguard publishing *many* advisories.

Note that the old JDO-based logic was not removed and is still used by a few REST API v1 endpoints. I opted to not do a full replacement now to keep the blast radius constrained.

Adapted for 5.1.x, where AFFECTEDVERSIONATTRIBUTION.LAST_SEEN still exists
and is NOT NULL: the column is written alongside FIRST_SEEN.

Signed-off-by: nscuro <nscuro@protonmail.com>
(cherry picked from commit 58b679f)
@nscuro nscuro added this to the 5.1.1 milestone Sep 20, 2026
@nscuro nscuro added the defect Something isn't working label Sep 20, 2026
@owasp-dt-bot

Copy link
Copy Markdown

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@codacy-production

codacy-production Bot commented Sep 20, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 97 complexity

Metric Results
Complexity 97

View in Codacy

🟢 Coverage 97.43% diff coverage · +0.07% coverage variation

Metric Results
Coverage variation ✅ +0.07% coverage variation (-1.00%)
Diff coverage ✅ 97.43% diff coverage (70.00%)

View coverage diff in Codacy

Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (8e16806) 44453 38851 87.40%
Head commit (80ac1f9) 44810 (+357) 39194 (+343) 87.47% (+0.07%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#7384) 428 417 97.43%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@nscuro
nscuro merged commit 807708a into DependencyTrack:5.1.x Sep 20, 2026
20 checks passed
@nscuro
nscuro deleted the backport-pr-7382 branch September 20, 2026 18:19
@4s1m0v 4s1m0v mentioned this pull request Sep 28, 2026
2 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

defect Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants