As eBay accelerates its tech-led reimagination, having a way to quickly vet and release code changes is key. Building on a foundation of 25 years of code requires thoughtful integration and detailed testing, which can be time-consuming – especially for intermittent errors which are difficult to replicate. To help developers focus on creating features which customers will love rather than troubleshooting regression test errors, eBay engineered a new way to quickly identify bugs so the code can be routed for updates.
eBay’s new open source Swift package, dubbed “Targeted Auto Retry,” is a simple, lightweight, but powerful solution which provides resilience against flaky test steps. Mobile teams at eBay that have adopted it have seen their flaky test problems virtually disappear overnight – with up to 99% of flakiness eliminated from their entire regression test suites.
Unique Advantages
This targeted approach has a number of advantages over the more common “global” auto-retry (a heavy-handed solution which automatically retries the entire test if it fails anywhere):
- By only focusing on specific steps prone to flakiness, Targeted Auto Retry greatly increases the chances of catching important intermittent bugs, instead of glossing over them. This immediately unlocks significant potential value from existing automation suites without any effort.
- It’s always an explicit and intentional choice to include Targeted Auto Retry for a test step.
- It’s surprisingly lightweight and easy to use. The package is hosted on eBay’s open source page for Swift. Most Swift developers / testers should be able to get it working and see immediate benefits within a single afternoon.
- Console logging is already baked into the solution through XCTContext.runActivity, so relevant, actionable information about Auto Retries will be easy to find in any test reporting system without needing to sort through multiple test run logs. These logs also provide flags set in the test steps, which can be used in test reporting solutions to pull out actionable metrics around the retry attempts:
○ [RETRY INFO: \(retryAttempt)]…
○ [NEW RETRY ACTION: \(retryAttempt)]…
○ [SUCCESS CONDITION: \(retryAttempt)]…
○ [RESET STEPS: \(retryAttempt)]…
○ [FAIL]…
- Since it’s targeted, the execution time is faster than global solutions. Also, the solution is more configurable / optimizable for potential flakiness in each test step.
When To Use Targeted Auto Retry
Targeted Auto Retry is a powerful solution. By design every use is intentional, so it’s unlikely that it will accidentally gloss over an important intermittent bug in the code. However, if misapplied there’s still a chance it could re-try a code segment with an intermittent bug until it works. Be sure to only use Targeted Auto Retry in cases where:
- You have exhausted attempts to fix the flakiness directly.
- The test step being auto-retried is part of the setUp / tearDown for the main focus of the test.
- If there is a real intermittent bug in the test step, you don’t want your test to catch it. (Or, there is a separate test focused on this test step without auto retry, which will catch real intermittent bugs.)
How It Works
Instead of retrying entire tests on failure, Targeted Auto Retry focuses on retrying just the steps which are most likely to cause issues with flakiness.
- You identify a test step which is prone to flakiness (e.g. app launch).
- The mainAction for the test step runs (e.g. XCUIApplication().launch()).
- A successCondition validates whether the mainAction step was successful (e.g. XCUIApplication().buttons[“My Awesome App”].exists).
- If the successCondition is true, you’re all set and you move on to the rest of the test.
- If the successCondition is false, that means the mainAction test step failed (or was flaky), so you automatically circle back to retry the mainAction test step.
- In some cases, you may need additional steps to reset the state of the test to just before the mainAction was attempted before retrying it, so an optional resetSteps code block is available to handle that if needed.
- If the mainAction continues to fail consistently (meaning the successCondition continues to be false) eventually it stops attempting retries and just fails the whole test.
The Impact
Since adopting Targeted Auto Retry, mobile teams at eBay have virtually eliminated flaky automation tests and have seen a number of improvements to their testing:
- Code coverage numbers increase substantially when flaky tests no longer artificially drag them down;
- Less engineer time is spent maintaining and troubleshooting failing flaky automation;
- Existing automation catches critical (but difficult to reproduce) intermittent bugs early in the process, and the results are taken seriously; and
- QE / Developer trust in the automation tooling and results is enhanced.
How to Integrate Targeted Auto Retry Into Your Swift Project
Having proven its worth across eBay’s mobile teams, Targeted Auto Retry has just been released publicly to eBay’s Open Source platform as a Swift Package, where it can be easily integrated into any existing Swift project. Step-by-step instructions for adding the Swift package are included. Most Swift developers or testers should be able to get it up and running and start seeing added value to existing automation in a single afternoon. Check it out, and discover what virtually eliminating flaky tests can do for your team.