We all know that some lenses are ‘sharper’ than others. We all know that there is some variation in the performance of different examples of the same lens. We also know that some lenses are rumored to suffer from greater sample to sample variation than others i.e. some manufacturers have better quality control than others. The challenge is ‘unpickling’ this and determining how relevant it is to you. The problem is that there is very little hard data available. Instead, we have anecdotal information available on various forums and blogs, but this can be misleading. Sometimes a few disgruntled users will create a stink about a product and you’d be forgiven for thinking its awful (when perhaps only a small batch was affected). However, I do believe that with care one can glean a lot of useful information from the internet about which lenses are predictably wonderful and those which may require you to try a few copies before getting one that performs up to expectations. What I am talking about here is resolution and consistency of that performance. Other optical characteristics are another topic entirely and won’t be covered by this article.
Moving on to MTF charts (to understand what MTFs are and how to read them, have a look at the Nikon explanation here or the Photography Life explanation) – they are normally used to show the performance of a given lens model. They can be either computer predicted or generated after actually testing a lens ‘in the lab’. The problem with them is that you do not really know how closely the MTF charts resemble the lens you purchase. Thankfully, lensrentals.com investigates this as part of their testing and makes the data available to the public.
Most testing sites only test single samples of a given lens, giving the potential for their ratings to relate to anything from an exceptional example to an awful one. These tests results are then posted without the end user having any idea what it represents on a scale of best to worst case scenario. Unfortunately, this means that a website/blogger can test a poor sample of a generally superb lens and leave a diabolical rating on their results page for years to come, leading readers to conclude that it is an inherently bad lens (when the opposite is true). Lensrentals is different. They purchase batches of lenses for their rental business and check them for serviceability and performance before leasing them out. Their data for a given lens usually represents an averaging of the data for up to ten individual lenses, so it is therefore much more representative than the single sample data used by sites like DxO.
As you will see if you click the first link in this post (and hunt around for more tests from them), lensrentals.com shows the level of variation across a number of samples of a given lens model. This gives you a sense of the best, worst and average scenarios you might face as a buyer, which is very useful information indeed. What’s more, they show trends associated with different manufacturers. The general impression one gets from reading reviews and users comments extensively is that Canon and Nikon are a little more consistent than the likes of Tamron and Sigma, which tends to suffer greater sample variation. This is reflected quite well in the data provided by lensrentals and both mesh quite well with the assumptions I have built up over the last few years (of personal experience) with regard to two manufacturers: Canon and Sony/Zeiss. Please note that I am talking about Sony and Sony designed and manufactured ‘Zeiss’ branded lenses (such as the 35mm f2.8 FE Sonnar and 55m f1.8 FE Sonnar), which are sometimes nicknamed ‘Zony’ lenses. I’ve had some fairly awful experiences with Sony and Zony lenses, having returned a number of examples for serious optical misalignment issues. I went through three copies of the 35mm Sonnar and two 55 Sonnars before finding one of each that was obviously not significantly misaligned and did not bother to replace my 28-70mm kit lens (which has a soft lower right corner). Thankfully my first Sony 70-200 f4 G was a good’un; however, overall, I received more badly misaligned (decentered) Sony/Zony lenses in a few months than I have all other manufacturers combined in ten years. I am, however, very with the performance of the lenses I ended up with, its just that getting there was a hassle.
You may also have noticed that the lensrentals testing shows that not only is the sample to sample variation of lenses like the Sony Zeiss 35mm f1.4 Distagon poorer than, say, the Canon 35mm f1.4 L II or Nikon 24mm f1.4 G, they detected significant variation in results when comparing performance the four corners of the same lens (this being caused by the lens elements not being perfectly aligned, which is called ‘decentering’. What’s more, they found that this variation in corner performance of some models was, erm, quite consistent. Translated, this means that samples of the Sony Zeiss FE 35mm f1.4 Distagon consistently show some degree of decentering. This also means that exchanging one lens sample for another was quite likely to give another sample with the same problem, perhaps featuring a different combination of good and bad corners. It does not mean that anyone owning this lens should be unhappy with it and its possible that the lensrentals team was unlucky with the lenses they tested, but these are the results that they got.
While the bad news is that Sony/Zony came out relatively badly here (a little worse than Sigma), Nikon did pretty well and Canon did astoundingly well with their most recent models. Although it is not mentioned in the linked article, they commented at how consistent their samples of the Canon EF 11-24mm f4 L were. Canon might be trailing the imaging universe when it comes to their sensors’ dynamic range, but they seem to be leading the pack when it comes to the consistently high quality of their lenses. This should mean that if you are a critical user and expect consistent performance, buying Canon lenses should prove less frustrating than some other brands (notably Sony/Zony) – one can buy with relative confidence.
In terms of sheer resolution, another point perhaps worth thinking about, if you surf the web looking at these things, is this (if you are not already aware) is this: DxO lens tests (probably the largest database of lens ‘performance’ to be found on the web) give results that are dependent on the number of pixels of the camera the lens was tested on. Some would regard this as a weak system (and I would be inclined to agree), because it gives lower scores to lenses tested on lower resolution cameras. This means lens A might actually resolve much more detail than lens B in sheer optical terms, but if lens A is tested on a 20MP body and lens B on a 40MP body, lens B will get a much higher score. Exactly; its a silly system.
To be more brand specific, the result is that in DxO’s lens rankings, Canon optics trail Nikon and Sony/Zony lenses because the Nikon and Sony/Zony FE lenses have been tested on 36MP bodies and the Canons only on the 22MP Canon 5D Mk III. According to ten samples of each, lensrentals’ results (derived from testing the optics on their own) suggest that the Canon EF 100mm f2.8 IS L Macro that I just reviewed is a somewhat better overall performer (and more consistent from sample to sample) in terms of resolution/contrast than the Sony G 90mm f2.8 FE Macro…. yet DxO gives the Sony FE 90mm f2.8 OSS G a sharpness score of 40 and the Canon EF 100mm f2.8 IS L Macro only 28! The internet is awash with reports of how great the Sony FE 90mm macro is, but there’s not much of a din associated with the older Canon lens, which aside from in control of Chromatic Aberation (CA), appears to be easily the equal of the Sony. The DxO hierarchy will be turned on its head when DxO retests its Canon lenses on the super-high resolution 50MP Canon 5DSR, assuming they do so.
Of course outright resolution is only one of the qualities a person may look for in lens, but consistent optical performance is arguably one of the most crucial. Strong asymmetries in resolution across a frame is often noticeable in large prints and can spoil an otherwise great image. I’d rather have a well aligned lower resolution lens any day of the week and this is why I continue to love Leica (and true Zeiss) optics. They’re expensive as hell, but if Sony Zeiss is making consistently visibly decentered $1500 lenses, they seem better value all of a sudden. Oh, and Canon may just be the best overall lens manufacturer in the world at the moment. They may not seem to be innovating like Sony at the moment, but to working pros, consistency and reliability is everything.