-
Notifications
You must be signed in to change notification settings - Fork 528
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Measure SLOs across whole suite, not single tests #779
Comments
Alternatively (I'm actually leaning towards saying that it would be better, but I'm happy to hear arguments against) we could:
|
I think that we can even hardcode the set of things that we start there - it would also visibly simplify the configs... |
@mm4tt - FYI |
I like the new per-suite measurements you propose. This is indeed a better, long-term solution. Mine was suppose to be rather a hacky prototype to validate the concept. However I'd rather avoid hard-coding any set of measurements, this would decrease the flexibility clusterloader2 gives us. But I'm open for debate. |
Could it become part of the TestSuite - https://github.com/kubernetes/perf-tests/blob/master/clusterloader2/docs/test_suite_design.md ? |
I'm not going to push hard for this. My main goal is just to simplify the configs.
|
yes - it perfectly fits there |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle stale |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
@mm4tt: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/reopen While we may not want to do this in the short-term, I'd like to keep this one open. The issues, you've mentioned means we avoid the problem, but not solving it. Where the problem i lack of functionality in clusterloader. We've added test for tokens recently, which is measured separately. We may add more in the future. We should be able to do measurements across the suite. |
@oxddr: Reopened this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
/remove-lifecycle rotten |
Currently we measure all SLOs per-test. We think about measuring it across whole testing suite (density + load).
Measuring SLO across performance suite will increase number of windows (as defined in SLO description). Single bad request will have a smaller chances to sink the whole tests. Currently we see test flakiness cause by a single request (e.g. kubernetes/kubernetes#82377). Also this would put us closer to intention behind the two-level SLO, which is defined per cluster-day.
Implementation wise, this would involve merging density and load test into a single test and moving some measurements to the very end of it.
/area slo
The text was updated successfully, but these errors were encountered: