tl;dr: you can try this rspec wrapper and here is a working example regarding how to ticket your flaky tests running on CircleCI using GitHub issues.
One of the common ways of dealing with flaky tests in your test suite is slapping rspec-retry on top of your test suite. Unfortunately, most of the time that’s the end of a solution. Without any sort of logging around it, you’re completely in the dark. Do you know how often things are failing and why? Are you introducing new flaky tests? Are you solving those?
On the other hand I only partially agree with the radical approach regarding simply removing or fixing such tests right away. As usual in real-life scenarios - it depends. It’s all about context, you might have a myriad of other issues at hand, you might lack context, or you might be new to the codebase altogether. Just blindly chopping random code down doesn’t seem like a reasonable approach. On the other hand, Hiding from the problem on the other hand and just letting tests retry silently seems like a good technical debt-collecting recipe.
Recently I have been working on a project riddled with flaky specs, and the majority of those specs were backed by the convoluted, unclear, and obscure implementation of a given feature. You’re not going ‘just fix or delete’ everything in such a case.
Moreover - flaky tests can be a valuable source of potential bugs buried in the codebase. It’s not always a bad test code, quite often it’s a sign of bad architecture and unhandled edge-case scenarios.
I have decided to extract one common approach that doesn’t require 3rd party gems and integrations. It just relies rspec persistence file and just spins a fresh rspec process again.
One key benefit is that you start with a clean env and that can sometimes reveal really interesting findings regarding the mess underneath; sometimes just re-running a given test/s (as rspec-retry does) is not enough as your global state is messed up somewhere, somehow already.
It doesn’t try to be smart and does not try to fail to build when ‘new’ flaky test is introduced, because in the end - you never know if that’s a truly new flaky spec or some new issue emerged because the test was run on 11:23 PM Friday, during a full moon. Especially with dynamically typed languages like Ruby - correlating new code/new tests with a new flaky test reports seems hopeless.
Such approach gives you basic visibility and creates an actionable item in form of an issue in your issue tracker - which should be just enough to get you started.