Repository navigation
Backport: Move vuln data source mirroring off JDO - #7384
Merged
Merged
Conversation
Refactors the `mirror-vuln-data-source` activity to use JDBI instead of JDO. This enables proper use of batch operations which significantly slashes database round trips and consequently wall clock time. Having tested this with OSV's Ubuntu ecosystem, the new implementation issues over 1000x fewer database queries. The old implementation also showed drastic degradation after ~30k synchronized vulnerabilities. I was able to observe throughput dropping from ~210 vulns per second to ~11/s. This was caused by queries on the `VULNERABLESOFTWARE` table that happened to match large numbers of records, e.g. those targeting the `linux` package. These queries were executed once per affected version. And naturally during a mirror run, the data already present in the DB grows, making the slowdown worse as the run progressed. This is likely the root cause of DependencyTrack#5934. With the refactored code, processing the OSV Ubuntu ecosystem locally is stable at ~280 vulns per second. For Chainguard it's ~3200/s. The difference between the two is caused by Ubuntu publishing few(-er) advisories with *many* affected versions, and Chainguard publishing *many* advisories. Note that the old JDO-based logic was not removed and is still used by a few REST API v1 endpoints. I opted to not do a full replacement now to keep the blast radius constrained. Adapted for 5.1.x, where AFFECTEDVERSIONATTRIBUTION.LAST_SEEN still exists and is NOT NULL: the column is written alongside FIRST_SEEN. Signed-off-by: nscuro <nscuro@protonmail.com> (cherry picked from commit 58b679f)
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 97 |
🟢 Coverage 97.43% diff coverage · +0.07% coverage variation
Metric Results Coverage variation ✅ +0.07% coverage variation (-1.00%) Diff coverage ✅ 97.43% diff coverage (70.00%) Coverage variation details
Coverable lines Covered lines Coverage Common ancestor commit (8e16806) 44453 38851 87.40% Head commit (80ac1f9) 44810 (+357) 39194 (+343) 87.47% (+0.07%) Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch:
<coverage of head commit> - <coverage of common ancestor commit>Diff coverage details
Coverable lines Covered lines Diff coverage Pull request (#7384) 428 417 97.43% Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified:
<covered lines added or modified>/<coverable lines added or modified> * 100%
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Refactors the
mirror-vuln-data-sourceactivity to use JDBI instead of JDO. This enables proper use of batch operations which significantly slashes database round trips and consequently wall clock time.Addressed Issue
Fixes #5934
Fixes #6971
Fixes #7346
Backports #7382
Additional Details
N/A
Checklist
This PR implements an enhancement, and I have provided tests to verify that it works as intendedThis PR introduces changes to the database model, and I have updated the migration changelog accordinglyThis PR introduces new or alters existing behavior, and I have updated the documentation accordinglyThis PR is a substantial change (per the ADR criteria), and I have added an ADR underdocs/adr/