Skip benchmark run depends on the result

[akiya-nagatsuka]

I want to exclude some benchmark runs if they are invalid.
For example

@State(Scope.Benchmark)
class BenchmarkClass {
    private lateinit var isValid: Boolean

    @Benchmark
    fun benchmark() {
        isValid = runBenchmark()
    }

    @TearDown
    fun tearDown() {
        if (!isValid) throw error("Invalid run")
    }
}

For JVM It works properly. But on MacOS the Gradle task fails with the error thrown in tearDown.
Can I make MacOS runner skip the benchmark run like it's done on JVM. Or I can send any message to benchmark runner that this run should not be counted.

[fzhinkin]

The current plugin's behavior for JVM is bogus (#186) and should be fixed. If a benchmarks throws an exception - that's an error on the benchmark side and should be explicitly reported as such. So the plan is to align the behavior with how it works for K/N.

@akiya-nagatsuka could you please elaborate on your particular scenario? What makes a benchmark invalid and why such benchmarks could not be excluded from a suite prior the run, or could not be excluded from results when you post-process them?

[akiya-nagatsuka]

The API I want to benchmark fails sometime and it pollutes the benchmark result. I would prefer don't include such benchmark runs in the final result.

when you post-process them?

If there is any way I can define that the benchmark has been gone wrong...
From the result json I only see the measurement time like 500ms, 450ms, 10ms, 520ms...
I can hack it by checking if the time very small comparing to other like 10ms, then it didn't run properly and I need to manually exclude it and recalculate the average time and procentiles.