I wrote this some time in 2009. Since then I have changed my opinions on a number of
things, and it's possible that I no longer agree with what I have written here. But I
haven't had time to check.
DxO Labs have published a web page which, amongst other
things, claims to measure the accuracy of the ISO settings of cameras. It would be easy to
believe it is as broken as the web site itself is, but in fact the details are quite
interesting.
First, the web site breakage. Clearly DxO is a Microsoft-based shop. It seems that they
bring out their software for Microsoft first and Apple only a few months later. They're
apparently not interested in other operating systems at all. It shows in the rendition of
their web pages. Here's firefox on FreeBSD, Safari on Apple Mac OS
X, and firefox on Microsoft. For all images, clicking on them will make them
progressively larger. The text boxes below the photos
are shifted to the right, and they overlap the general text below:
So: if DxO software were of the same quality as the web software, I wouldn't give them a
second thought. This is serious breakage. But what about their measurements? I was told
that my camera, an E-30, was off by nearly 1 EV at 800 ISO, and indeed, that's what the graph shows.
The person who mentioned this (on a German web forum) compared the
results with the results for the Olympus E-3, which the DxO analyser found correct. I tried
it with cameras from other manufacturers, the Canon 50D and the Nikon D90. Result: DxO
claims that they all have effectively between 0.5 and 0.8 EV inaccuracy at ISO
sensitivities beyond 100 (or 21°). And they all have too high a sensitivity
at ISO 100, so that the real sensitivity is the same at ISO 100 and ISO 200 (24°). Well,
that's what DxO is claiming:
Only see two different colours on the graph? Yes, that's because they're all on top of each
other. Here the Nikon appears to be missing; try again and one of the others may be missing
instead. Now it's quite possible that all manufacturers have made the same
mistake—doubtless a conspiracy?—but this kind of coincidence points to a single
cause, and the obvious single cause is DxO's analyzer.
It's easy enough to test, though: take some photos. The following were taken with my E-30
with manual exposure. In each case, the aperture was f/8. In my experience, the accuracy
of the aperture is lower than that of the shutter, so for the first two I set the shutter at
1/250 s, and for the third I set it to 1/500s. The first was set at ISO 100, and the other
two it was set to ISO 200. If the DxO claims are correct, the first two should look the
same, and the third one should be noticeably darker:
Clearly that's not the case, and that should enough evidence for me to discredit DxO's
claims. But the truth is stranger.
First, the third one is marginally darker than the first. Is there maybe a grain of
truth in their claims? These photos were taken with combined JPEG and raw images. The ones
above are from the in-camera JPEG. I put the raw images through UFRaw and ran “auto adjust exposure”
against them. The results:
The first image was set to a compensation of EV +0.58. UFRaw is wildly inaccurate with
its claims of absolute exposure, though the relative values seem to be OK. Based on my
experience, this suggests that the photo was overexposed by about 0.6 EV.
The first image was set to a compensation of EV +0.60, in other words pretty much the
same.
The third image was set to a compensation of EV +1.73, in other words pretty much 1 EV
more, as you would expect from DxO's claims. In fact, compared to the first image, it's a
compensation of +0.15 more, about the amount by which the third image is darker than the
first. That may be an inaccuracy in the shutter speed, which I'd need to check more
carefully.
The strange thing here is that the raw images don't match the in-camera JPEGs. There's
obviously more here than meets the eye. Obviously the images converted with exposure
compensation all look the same, and the histograms are pretty similar:
This is the effect that I would expect from DxO's claims, so at least the shape of the graph
seems to be correct: there seems to be almost no difference between ISO 100 and ISO 200.
Why? Clearly the ISO ratings of raw images are only marginally of interest; the real issue
is what comes out when they're converted into JPEG or other open formats, and there the
results are pretty much what you'd expect. But it seems that all manufacturers do this. I
checked a couple of other cameras, the Canon EOS 1Ds Mark II, the Nikon D3, the Pentax K20D
and the Sony Alpha 900, and the results were similar (though the Canon starts at about
80/20° ISO). The Hasselblad H3DII 50 is given as ISO 50/18° in all cases (from 50/18° to
400/27°). Only the Leica M8 is exactly on the line. It's interesting in this connection
that the default ISO rating for the E-30 is 200, and that may be related.
In summary, then, the results are interesting, but it's not clear to me what they say about
the cameras or the DxO analyser.
