Backscatter's Strobe Test: A Deep Dive Into Underwater Photography's Most Ambitious Data Project

In this article from The Underwater Photography Show, we take a close look at Backscatter's newly published in-depth underwater strobe test — a project covering more than 20 strobes currently on the market.

This test took Backscatter well over a year to put together, and it shows. We were genuinely surprised at how little attention it's received in the community so far, which answers questions the community has been asking for years. About the only place we've seen it discussed at all is in a small thread of less than a dozen posts on Waterpixels. It should be generating more conversation!

Part of the apathy is probably timing. Three years ago, a huge portion of underwater shooters were still using previous-generation strobes — older Inons, Sea & Seas and other strobes — and in the years since, it seems to us that the majority of serious shooters have upgraded. Having spent the money, there's less appetite to go looking for reasons a newer strobe might be better. I know when I buy a new phone or laptop, I instantly stop reading the reviews!

But this is different. We'd encourage anyone with an interest in strobes to read through it. You don't need to focus on the comparative data, the written explainers are clear and genuinely improve your understanding of how strobes work and the issues that shape their performance.

The scope of the project is enormous. Backscatter measured strobe brightness and beam pattern, including the underwater guide number for each strobe, alongside the actual test shots used to calculate it; recycle time; color temperature, measured on land using their own color-measurement equipment; weight, both in water and out of water — a detail that's often published individually but rarely compiled, how evenly spaced the power settings are — and just as importantly, the total range those settings cover; and flash duration.

And the reward that often comes from rigorous testing are interesting revelations. We found the consistency-of-power-levels data in particular was surprising — it's not something that gets tested casually, and the variation across different strobes turned out to be far wider than assumed. Flash duration was another area that, on paper, doesn't sound like it should matter much. Surely all strobes flash fast enough for typical flash synchronization. Not so. Check it out.

Whatever criticisms follow, the scale of the undertaking deserves real credit. Most of us have only ever done head-to-head comparisons — this is the first time anyone has tried to build the full picture.

The obvious elephant in the room is that Backscatter is itself a strobe manufacturer, and it's very much in their interest for their own products to come out looking good. They're not hiding this — the test is published on Backscatter.com, and their strobe is called exactly what it's always been called, the Backscatter Hybrid flash or Atom, depending on the model. So the relationship isn't being hidden.

Backscatter said to us throughout that the aim was to provide a consistent dataset and not to declare a "best" strobe. They just wanted to put data out there. But we do feel that there is a marketing angle to it all. After all their social media post announcing the tests starts with "What is the best underwater strobe?" above to a photo of their own Hybrid flash! And the way the data are presented plays to the strengths of their own products.

Our read is that the motivation when they were testing was genuine. There's been a real lack of objective, industry-wide testing, and Backscatter felt frustrated that some rival manufacturers — particularly several from China — were making claims about their strobes that weren't close to the truth. That's a legitimate problem: retailers get customers every day saying "I read online that this strobe does X, why doesn't yours?" when the claim simply isn't true. And we understand their desire to set the record straight, but also feel the article would have benefited from another editing pass to strip out some of that framing.

One of the most useful aspects of the test are the photos of the strobe beam spreads — all taken at the same exposure. But that exposure was clearly chosen to show off their strobes — and anything less bright — underexposes and looks poor by comparison.

A few things stood out as genuinely new information, even to people who've been shooting and testing strobes for years, which is why we think it is such essential reading.

First beam spread can differ significantly topside versus underwater. Backscatter highlights this with an extreme example from Sea & Sea. But it is worth noting they showcase the strobe with the biggest topside-to-underwater difference to make the point — most strobes don't actually vary that much between the two environments.

Flash duration matters more than most of us assumed and could be even more significant in the future if global shutter sensors become more widespread. The way some strobes achieve high power output is by letting the flash tube burn longer — and occasionally this is longer than the shutter speeds we typically use underwater. If you're shooting at 1/200th or faster, you will not be getting the full output of a few strobes on the market. This matters because the time you are most likely to need those high powers is when balancing bright ambient light, and this is also when you will be using faster shutter speeds.

Power settings aren't always evenly spaced. Most strobes do a decent job, but a few showed power dials where certain clicks make almost no difference to actual output, while others make a big difference. Clearly not something you'd want when trying to correctly expose a photo.

We don't know whether Backscatter tested multiple samples of each strobe or just one of each type. It is likely the latter, which raises a fair question about unit-to-unit variation. So while we find the revelations in the data exciting, we have no idea about the error bars on these findings.

One theme running through the whole test is the gap between manufacturer-claimed specs and what Backscatter actually measured. Unsurprisingly, the Backscatter strobes' claims line up neatly with Backscatter's own measurements, because they're using their own methodology. As our friend Dave Hicks already pointed out on Waterpixels, this is "methodology bias" — every manufacturer believes their own way of testing is correct, and any other manufacturer's numbers that don't match are assumed to be inflated.

We understand Backscatter wanting to stress these differences, but including so much claimed (or perhaps more correctly measured under different conditions) data on the same graphs as Backscatter's consistently measured comparison muddies the picture. On some graphs it stretches the axis so the real difference between strobes are lessened, or just makes graphics overly busy with too much to take in. The Guide Number graph is one example — and a simple ranking table of UW measured Guide Number would be easier to follow. The color temperature comparison graphic also suffers, where genuinely important differences of 1,500–2,000 Kelvin between strobes end up squeezed into the middle of the chart because the claimed numbers push the x-axis scale so wide on either side. We'd have preferred a separate section addressing claims-versus-measurements, with the main article showing only Backscatter's "measured in the same way" data for each strobe.

This bleeds into a broader issue: the aesthetics of the graphs themselves. Most readers are going to glance at a chart on their phone and simply note which bar reaches furthest right, without checking the actual numbers, or appreciating what are significant differences. On the guide number chart, for instance, the difference between a strobe rated 45 and one rated 43 looks enormous on the page — even though as a photographer if you stop and think it is just the difference between using an aperture of f/45 or an aperture of f/43 to get the correct exposure — less than a click on your camera and hard to even notice. That's partly a function of using a logarithmic scale rather than an f-stop-based one. The log scale properly represents the energy and makes the most powerful strobes (like Backscatter's own) stand out, but most photographers think about exposure in terms of clicks on their camera — and a one click-one space scale would be more useful.

We believe the data were collected as reliably as possible, but do feel the objectiveness of the article is undermined by some strange choices on how the data are presented, which always seem to be to highlight a strength or disguise a weakness of Backscatter's own strobes. And we feel this is a great pity — because we believe Backscatter made their measurements to the best of their ability and we believe it. But some of the presentation choices, clearly designed to lead the eye a certain way, start to make you sceptical.

For example, we really rate Backscatter's Hybrid flash, but one criticism we had was it is heavy in the water. The review shares in water weights, but the data is plotted on the same scale as weights in air, so in water weights seem small and insignificant by comparison. When you are neutrally buoyant underwater a small negative weight at the end of your body is anything but. Our only other criticism of Backscatter's Hybrid was the light was noticeably cold. Backscatter again recorded this faithfully, but the Colour Temperature graph is plotted on a wide axis, so significant differences between strobes are compressed together and less obvious. Furthermore, the Beam Pattern comparison shots are presented without colour, so the colour differences between strobes is hidden. And as we already mentioned that the beam comparisons are set to be correctly exposed by the most powerful strobes, making all others look "incorrect" by comparison.

In isolation each of these choices could be justified, but take them together and they feel like a narrative that undermines the feeling of objective testing. Which, again, we feel is a pity, because we believe the underlying data.

For us, the single biggest gap in the test is a clear attempt to look at quality of light — that we'd argue is the thing that matters most about a strobe's actual output. I (Alex) rarely shoot modern strobes at full power; so power itself isn't the priority. What matters far more is the evenness of the beam, how consistent the brightness is from the center of the frame out to the edges, particularly for wide-angle work.

Backscatter agree with this, writing in the review "For wide angle, a beam with a wide and even spread is king. Not only will it light the scene more evenly than a more spotty strobe, but it also makes strobe positioning way more forgiving compared to one with a tight beam. We'd gladly take a less powerful but wider and more even beam strobe than one that is bright and spotty for wide angle."

However, the way they choose to present the data does not allow you to examine this. The Beam Pattern comparison shows all strobes fired at full power, captured at the same exposure (the correct exposure for the brightest strobes). This means that weaker strobes are incorrectly exposed, look dim, look wrong, especially so away from the centre. And a more powerful strobe will look brighter at the edges too, simply because it's brighter overall — that's not the same as telling you how evenly the light falls off.

A far more useful test would be to have exposed every strobe correctly — so that they are the same brightness in the center of the frame and that would let you compare how the light falls off. So that you could "gladly take a less powerful but wider and more even beam strobe" as they advise.

We'd also question the test distance. Backscatter tested from one meter, which makes sense for guide number comparisons since that's a common standard for on-land measurements. But one meter — just over three feet — isn't a realistic shooting distance underwater for most wide-angle work; try setting your focus to three feet on a wide-angle dive and see how many photos you get in focus. And this will bias the test towards strobes set up to throw light far. And the real challenge for underwater strobes is producing a high quality of light when working up close. We understand the choice of one meter for measuring Guide Number, but not for the beam spread.

While we found it important to cover the critiques here, we doff our caps to the Backscatter Team for their hard work in creating something truly valuable. We'd strongly encourage everyone to read through the feature and learn more about strobes and lighting. This is surely going to become a genuinely central resource for underwater photography — and a long overdue one.

We'd like to see a bit less emphasis on claimed specs vs measured and suggest that feeling of objectivity would be enhanced without some of the strange choices on graph presentation. Some data tables of numbers would be valuable as would a table of current prices (we know these change, but you could put a date next to it).

Overall, the sheer scale and scope of the review deserves real credit, and just as we have, we feel everyone will learn something from reading it. The review is packed with detail too and this is very much our first look — we'll be returning to look through the data in more detail future episodes for sure.

You can watch the full conversation below.