Repository navigation
SRI hashes changed on multiple libraries yesterday and some SRI hashes are wrong #14210
Description
Yesterday (2023-07-31), I noticed that the SRI hash for https://cdnjs.cloudflare.com/ajax/libs/vue/3.3.2/vue.global.min.js (https://cdnjs.com/libraries/vue/3.3.2) changed. It used to be sha512-6mO8pNkTyFMzOwbajocp9NbJzQUVV+zBFPZW8pKpIKnjhHYY19ez+tISSNkkvaBsj/nHq/E2FANyg6YlB7A+dQ==. Now it's sha512-twbmYKE2H3LPbArlkbjDppF/XJ+GVTM/RvEceYhFMEl6xi1PF1Q5004mNU8BFp3WKatWanAFKU2SQbdk4PdjXg==.
Also, I noticed that the SRI hash for https://cdnjs.cloudflare.com/ajax/libs/vue/3.3.4/vue.global.prod.min.js is wrong. https://cdnjs.com/libraries/vue/3.3.4 says it should be sha512-pHXx64U1UkFexsKz00jfapuz0FgKq+a+5dBTuitirYVcJzaxPqTbNbCWsUiFyTYVYmHMiql9zLCeod+IpAJUew==. However, https://www.srihash.org/ says it should be sha512-39BSQXI5q1XlvVhLfFRidKG8KM6Tr6VS/XSnNo6N/A0ZXexHCeoUI/s+ulujQy3UREjoLNrMnFat8VI0mMugWA==.
I did some debugging and found these logs:
- Processing logs: cdnjs/logs@cb82d2b
- KV Publish logs: cdnjs/logs@1feb270
From that, it looks like version 3.3.2 was published on 5/12, but was updated yesterday at 12:07:49. You can also see that the SRI hashes changed. I'm not sure if it's normal that old packages will be regenerated periodically.
Based on comparing an old copy of vue.global.min.js with the new one, it seems like something changed in the minification process, so different variable names were used.
I searched commits in https://github.com/cdnjs/logs for "update kv-publish". From that, I found several other libraries updated yesterday where the SRI tags changed:
From them, it definitely seems like the issue is that:
- Existing versions were regenerated.
- If those versions had minified versions, those ended up with their SRI hash changing.
I still don't know if that explains why vue/3.3.4/vue.global.prod.min.js is wrong.
Activity
MattIPv4 commentedon Aug 1, 2023
Hey!
It looks like a backfill was run yesterday against a large set of libraries to populate them in new infrastructure -- this backfill was not meant to update the files on the CDN, nor the SRI hashes, as we consider all files on the CDN to be immutable, it was only meant to populate the libraries in the new infrastructure.
However, due to a bug in the processing logic, that being that the minification dependencies used weren't pinned to a specific version, many of the minified files were regenerated during the backfill processing and the new versions were pushed to the CDN with slightly different contents and new SRI hashes.
I flagged this internally this morning when I discovered that the backfill had caused SRI hashes to change, and the folks involved in the backfill have put a fix in place to pin the minification dependencies and re-run the backfill again, which should hopefully result in all the changes files and SRI hashes being updated again back to what they were before (this second backfill run is actively processing as we speak, it just takes a while to get through everything).
I have an open question for the folks running this logic around how we plan to verify that we are back to the state we were in before this backfill was run, so that folks' sites aren't broken with SRI hash mismatches going forward. I will update here as I hear more.
Apologies to anyone whose site this has broken!
ssigwart commentedon Aug 1, 2023
Thanks for the quick reply, @MattIPv4!
For verifying the old SRI tags, perhaps you can find any
kv-publish.logthat was updated yesterday and grab the original SRI hashes from there.MattIPv4 commentedon Aug 1, 2023
https://status.cdnjs.com/incidents/j85nrhmbcbc5 has been published to track this as an incident impacting the CDN.
xtuc commentedon Aug 1, 2023
Those files are back to normal. I will invalidate the cache globally, please let me know if this fixes the issue for you.
ssigwart commentedon Aug 1, 2023
Thanks, @xtuc.
For the 3.3.2 update, that did solve it. However, I had to do a hard browser refresh to get the corrected version because it looks like the caching headers allow the browser to reuse the same file for up to 355 days. I think that's going to have lingering effects for users that happened to visit websites in the last day or so and have the incorrect version cached. I don't think there's much that can be done about that though.
For the 3.3.4 update, https://cdnjs.com/libraries/vue/3.3.4 is showing the correct SRI hash now, but https://cdnjs.com/libraries/vue still has
sha512-pHXx64U1UkFexsKz00jfapuz0FgKq+a+5dBTuitirYVcJzaxPqTbNbCWsUiFyTYVYmHMiql9zLCeod+IpAJUew==.AKHorizon commentedon Aug 2, 2023
still having issue with vue 3.3.4
https://cdnjs.cloudflare.com/ajax/libs/vue/3.3.4/vue.global.min.js is showing:
sha512-fcdKKm7U9XeFv9U8RsC9vwQfGoLk2vnUXhFAzJk1bCtWUvNxJP2HaSSW1+RopF16gpk4MLkYFbYukbuQX7e4Ug==MattIPv4 commentedon Aug 2, 2023
cdnjs/logs@8a96350 indicates the asset in KV has been rolled back to the prior version and matches the origin SRI hash (cdnjs/logs@46f0f30). I suspect you're still seeing a cached version of the bad state from yesterday?
@xtuc has there been any thought given to purging the CDN cache as a whole, given how many libraries and files were impacted by this incident? Of course, end-users are going to be suffering from this for roughly a year with the cache headers we have in place, but at least we can clear out any bad responses from the CDN itself?
wh1t3t1p commentedon Aug 3, 2023
MattIPv4 commentedon Aug 3, 2023
@xtuc Could you take a look at this? https://github.com/cdnjs/logs/commits/prod/packages/c/codemirror/5.48.4 doesn't have any of the kv-publish logs I'd expect to verify whether the SRI hashes were changed/reverted as part of this incident?
Martin-F commentedon Aug 3, 2023
I just ran into this problem, and found this:
...so, here I am.
My affected code uses https://cdnjs.com/libraries/codemirror/5.38.0:
I8NyGs4wjbMuBSUE40o55W6k6P7tu/7G28/JGUUYCIs=9T+2aDjQSkwR+EUy1CF0CdcHIcSvYBLTIcE0bIfGRvg=/fNlNrKCT1Q8cEd3M2yaup/lkUpN1LQEG109ZPNskYk=GTw5k2DzzSlHiUfE7e9WjJDJYfmRjieIe1laeXbQGFU=QmQ8vh5c1+UzpnyWXYB7nnY1aqpJmXPVkFWtJHJkLnE=G6wKF6cDqgsd/06cqf2ilYycFusA8WE+pnc5K9Q/8LE=ylo1K/gUyfpfh6s3tKjY8Q6voK8/TCnx8eZaZeoMldo=84b94SFsvUq1IsfeaC/aVoLY2KZ5WSx6RabGswnga8s=YVb7n24DLwrjN0DghcwJYL17MI/X7e3c4EJZr6PwNdY=oc1I6s4YVpw+rzBOMU7PGtwtXfnQrweg0vm7009U+3g=CcVYnZJocbbLCyjOo/tA2QEH56hhTBEPZX+che0RYSo=CzgL3sVoqXsZU1T+JHgK8qZnnB3OKdvGcbb+n+KPMkU=xtuc commentedon Aug 4, 2023
Our cdnjs.cloudflare.com main storage got out of sync with the cdnjs/cdnjs git repo, after syncing:
@Martin-F:
@AKHorizon:
@wh1t3t1p :
These libraries SRIs went back to normal. Also, I globally invalidated the cache for them.
wh1t3t1p commentedon Aug 4, 2023
thanks for the resolution, much appreciated!
Martin-F commentedon Aug 4, 2023
@xtuc can confirm that's fixed for me now, thank you!
21 remaining items
shatterproof commentedon Jun 22, 2024
@jackielii
Try
sha512-9KsixJxn00SQfLkrArzXRFi3QyqP4RyIzYgj2Ho4YJSLEOHS00uhDQdc7dv6N8PS5OsYZWKiY0/7DsZRlcv/DQ==for the integrity attribute.@xPaw
Try
sha512-v2CJ7UaYy4JwqLDIrZUI/4hqeoQieOmAZNXBeQyjo21dadnwR+8ZaIJVT8EE2iyI61OV8e6M8PP2/4hpQINQ/g==for the integrity attribute.These were calculated using srihash.org.
For dylanvdmerwe and Kuuchuu's issues the SRIs do match those from the hash calculator, so this is even more concerning since it indicates the files were arriving changed. I checked the SRI currently being returned from cdnjs for
moment.min.js2.30.1 and it is incorrect again.@xtuc
A bunch of SRIs are out of sync again listed in the comments above (and I'm sure many others as well), could you please revisit this? Not sure if they were missed in the first pass or if the bug reoccurred.
This makes cdnjs unusable unfortunately since we cannot rely on integrity checks if the contents of static files are changing, in addition to many SRIs on site being out of sync. It would be great to see this fixed.
xPaw commentedon Nov 26, 2024
To this day the site is giving wrong integrity hashes for some scripts.
blazing-panda commentedon Jan 16, 2025
Still having this issue with:
AllanJard commentedon Aug 26, 2025
Just a quick note to say that the moment 2.30.1 hash is still being given incorrectly: https://cdnjs.com/libraries/moment.js .
Is when is given when clicking on the "script" button, but as noted the hash is wrong. Is there a way to have the site recompute the hashes?
lucciu5 commentedon Dec 18, 2025
I am experiencing issues with moment.min.js@2.30.1 having a wrong hash
andrewfrazer03 commentedon Jul 24, 2026
Jquery 4.0.0 added to the CDN after the migration has the wrong SRI hash.
Hash on cdnjs.com:
sha512-8LENNbXmzI/Gbj+OwXmqR6V4QaUAw0/porPzy1+dQoJqC0JPHedWoe0DDOTL2uHA5XXJyIsPtiMHH86pVlay6A==
actual:
sha512-YuCuk5nNmVIUfKROKeV3fpZZ5Vt9vsnq8nExr5JwEJc2r1YDVmDfujcq373eHIzjqdxwCzoKpxngIaAdRUyg3A==
MattIPv4 commentedon Jul 27, 2026
@andrewfrazer03 thanks for flagging. Does indeed look like that version was published by the new bot: https://github.com/cdnjs/logs/tree/prod/packages/j/jquery/4.0.0
And I can confirm that we have the wrong SRI hash for it:
@simonabadoiu any ideas how the bot is ending up with the wrong hash?
I believe the original bot had a bug where it'd use the hash of the source minified file while storing our own generated minified file, due to a race.
simonabadoiu commentedon Jul 28, 2026
Hello, Thanks for flagging this, and sorry for the trouble. Thanks Matt for checking as well 🙏
Checked it out: the hash on api.cdnjs.com (sha512-8LEN…6A==) is the right one. It matches what npm ships in jquery-4.0.0.tgz exactly, so if you've got that in your integrity attribute you're fine and don't need to change anything. The problem was the file itself, not the hash. It looks like the same compression issue we hit in #14305. We re-ingested jquery 4.0.0 on July 21 and R2 has had the correct file since then, but our cache holds files for about a year and it's per-datacenter, so depending on where you are you might still be getting the old bad copy. That's most likely what you saw. We're going to purge the cache for everything we re-ingested so this clears up everywhere.
One more thing worth mentioning: the re-ingest didn't show up in cdnjs/logs, which made it look like nothing had been republished. Turns out our audit logging writes the same content back to GitHub when the file list hasn't changed, and GitHub just no-ops that, so no commit. We've got a fix for it and will deploy shortly.
Will update here once the purge is done.
simonabadoiu commentedon Jul 28, 2026
The cache purge is done.
@andrewfrazer03 can you confirm on your end? One thing to watch: our headers let browsers hold these files for ~355 days, so you may need a hard refresh to pick up the corrected copy.
MattIPv4 commentedon Jul 28, 2026
Things look better via curl now:
AllanJard commentedon Jul 28, 2026
The SRI hash for https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.30.1/moment.min.js on this page still appears to be wrong I'm afraid. I've done a hard refresh and a second browser just to make sure.
MattIPv4 commentedon Jul 28, 2026
@AllanJard that version would have been processed by the old bot many years ago (https://github.com/cdnjs/logs/tree/prod/packages/m/moment.js/2.30.1), and so likely hit the original bug I alluded to that this issue was originally for.
@simonabadoiu do we have any plans to refresh the SRI hashes we're storing against the actual assets, to clear up leftovers from that bug? I feel like we'd maybe talked about doing that as part of the migration work?
andrewfrazer03 commentedon Jul 28, 2026
@simonabadoiu I can confirm Jquery 4.0.0 and other recently updated library script tags copied from cdnjs.com are now working as expected on my end.
simonabadoiu commentedon Jul 29, 2026
@MattIPv4, @andrewfrazer03 yes, we have a task tracked for refreshing the stored SRIs for old versions affected by that bug. It isn't in flight yet, I'll post here when we start. Thanks for your patience 🙏
rkerr commentedon Aug 3, 2026
Having run into this myself for jquery 4.0.0 I just wanted to add a little bit more info that may be useful to others. The difference between the two hashes is that one represents the file with a newline at the end (78748 bytes) and one is the same file without a newline (78747 bytes). Although cdnjs is now serving up the version with the newline (the hash starting 8LEN) end user browsers have a reasonable chance of having the version without the newline in their cache. Whilst flushing the cache is an option if you have control over the browser, I found it better to just put both hashes in the script tag for now as that lets the scripts load regardless of what the browser has cached.
Jquery UI 1.14.2 also has a similar issue, with two different versions with and without a newline at the end.