Repository navigation
Provide a way to check the mean of distribution of test cases' labels #409
Description
cover, coverTable and checkCoverage provide ways to target some specific distribution of test cases, and fail the test if this target is not reached within some confidence interval.
I would like to write a Property that expresses some requirement about the mean of the distribution of test cases. This property provides some context on what I am trying to achieve. In this particular case I would like to test the mean of retryCount property of the test cases to be 2.
Activity
MaximilianAlgehed commentedon Jul 1, 2024
This sounds like a reasonable thing to want indeed. However, before we can discuss details of combinator design etc. I would be really grateful for a tiny example right here that makes it crystal clear what you want to achieve.
MaximilianAlgehed commentedon Jul 1, 2024
Perfect, that's exactly what I was looking for.
The issue I foresee here is that if we want to fail a test the way we do with insufficient coverage we need to know what kind of distribution we are dealing with. In the case of boolean yes-no coverage that's easy - it's always a binomial distribution. That fact makes it possible to answer the question "is the coverage higher than X% with Y% confidence?" (Y defaults to something very close to 100% to make sure that things aren't flaky). With means this becomes much more tricky. Firstly, I don't think it would be sufficient to specify "I want the mean to be X" but rather "I want the mean to be X +- Z" and then to answer the question "is this true with a confidence of at least Y%" one would probably need to know more about the distribution (or assumed distribution?) of test cases.