Raw vs. Compressed Raw

Jarrell,

I've come to the same conclusion although I certainly haven't done an in-depth test. I'm also interested in the conclusions of other forum members.

 

I have not been able to detect any differnece hence I shoot in compressed RAW.

 

Partly, it depends on the camera and the level of compression. For example, the D100 compressed RAW is more compressed than the D70s RAW (which is compressed only, no straight RAW available). From an image quality point of view, I can't see much difference, but from the write speed of the D100, I'd never shoot compressed RAW, I can't afford 20 sec/image write times. In fact, I can't afford the RAW write times anymore, so I have gone back to shooting fine jpegs. Maybe with a D200 I can get back to RAW, but frankly, the jpegs seem to be just fine for the print sizes I am doing. The image degradation that I am getting is due to conditions, not the shortcomings of jpegs. Bouncing on a boat trying to shoot erractically moving whales doesn't lend itself to maximum sharpness, even without being elbowed by other passengers!!

 

Dear Jarrell,

From what I've read the differences are mainly in the highlights, and you increase the chance of something called posterisation especially if your exposure was a mite off and you correct it in post-production. It also seems to depend upon camera too as the D100 as has been pointed out is too slow to gain much whilst the newer D70s and D50 hardly notice the processing yet save a lot of space, not sure about the top-end cams like D2Hs and D2x, presumably you'd buy enough memory with those to not need compression, anyway? I think the phrase is YMMV (your milage may vary) isn't it?

 

I don't think file space is the real issue. If there are issue its with the highlights. A photo can be "properly" exposed and still carry a wider dynamic range than the ability of the camera to fully record. That is when you may get some clipping in either the highlights, shadows or both. I haven't made conclusive tests myself yet, but I've generally trusted the research from others which show slightly more clipping at the high end under compression. Sorry I dont have the addresses but you should be able to find some of that research through Google.

I've begun shooting uncompressed when I know I'll have to balance a larger dynamic range, or when I expect a lot of spectral highlights or light traces on an object, such as under controlled lighting conditions. Part of the compression seems to cut off the white point just a little sooner. This can be troubling with a lot of highlight data, because half of the image data is being stored in the brightest stop of information. I guess it's better to be safe than sorry, but it shouldn't matter for most average subjects.

 

I don't think uncompressed NEF increases the maximum DR, that's fixed. What it supposedly gives you is finer granularity in the highlight tones. I recall a thread in the Adobe User Forums where people like Bruce Fraser and Jeff Schewe claimed that the extra tonal information in the highlights would give you greater editing headroom for things like highlight recovery or curve manipulations. Based on what we know about how compressed NEF works, that makes sense. However there was also a recent thread at DPReview where this issue was discussed, and the conclusion from people who actually tried to test this theory was that there was no real difference in the ability to recover highlights or manipulate the highlights with curves. Now, maybe the people who tried to test this did something wrong, but it definitely got me to wondering whether it's really worth the increased file sizes.

It's a shame Nikon felt the need to use this "imperfect" compression scheme, because I think most of us would have been happier with a truly lossless compression. I find it interesting that I can take a 20MB NEF and convert it to a 9-11MB DNG using true lossless compression.

 

Well, I'll tell ya what...this discussion has encouraged me to spend the extra time experimenting on my own. I can't do it this week because my camera is in the shop (argh!) so I'm using film right now, but I'm very curious what I might find on my own. I do believe there is a difference, because as cryptic as Nikon may be, there is no way they would offer an arbitrary choice of wether we simply like big files or small files! It also comes to my mind that if a compressed NEF were truly lossless it would take a little longer to open while it unpacked all that extra compressed data. Things like zip compression works by stripping away all the "blank" fields of data, but the ratio for that isn't the same as compressing an NEF. JPG works by eliminating data, and that ration is closer to what we get with our compression. It's no use getting very technical about it because we don't really know what Nikon is doing, but the bottom line to me is that there is something that's giving in this compression.

 

RE: compression time, maybe the true lossless compression is more computationally expensive than Nikon's "visually lossless" version. However I can tell you that opening a compressed DNG in ACR is faster than opening an uncompressed NEF (at least in the case of the D2x).

As to why the choice? Well, Nikon claims that the compression is visually lossless. If that were really true then the choice would come down to smaller filesize versus faster card-write times. With the large buffer on the D2x the latter isn't really an issue IMHO unless you're doing a LOT of continuious shooting. I do sometimes use 3-5 shot bursts, but with the buffer write time isn't really an issue for that.

 

I believe that 'visually lossless' will be just that, you will not see the difference in an image. However, if you edit the image, you will have less than the normal 12 bits of editability, so some artifacting may occur.

I wish Bjorn were here, because he would probably have done testing on this and would know.

 

I understand the write times, etc. But what I mean is....if there were truly no difference (if it was truly lossless) then there really wouldn't be an issue, the file size would simply be smaller. If Nikon is giving us that choice, we can be sure there is something being stripped away that someone might think is important some day. To call something "visually" lossless is a pretty good admission that it's not really lossless. It just depends on wether or not you're going to notice whats missing in the end result. I'm sure there's only a certain criteria that would make it noticeable, or maybe only noticeable under a comparison basis. It seems apparent though, that whatever we are losing is happening in the highlights. In terms of the end points, it could be analogous to the difference between ProPhoto and RGB color spaces. Without further conclusive testing, that's generally the way I tend to think about it. There always seems to be a lot more room to play with the highlights than there is in the shadows, and Ive always read that there is more data is stored in those higher ranges. This type of clipping makes complete sense to me.

But we can't really discuss it without knowing the NEF proprietary data, or conclusive testing. Once I'm able, I will perform a series of tests and post my results, including all criteria used and every step performed. Maybe I can even host the compressed and uncompressed NEF's on a server somewhere for others to perform their own tests. I have the perfect source image in mind. It will include spectral highlights, light "traces", and bracketing with +/- exposure compensation. That would result in test comparisons between 3 uncompressed and 3 compressed RAW NEF's. This is about as conclusive as I can think of. This thread has inspired me.

 

That's the theory, the question I think is whether artifacting actually does occur in real-world shots.

Yeah I'd also love to know what Iliah has to say on the matter (his work on RML would certainly give him some insights, I think) but apparently he's been pretty swamped lately.

 

Wait a minute...a compressed RAW is still a RAW file. Its still 12 bits, isn't it? I don't think that's what they are stripping. That would be fairly extreme.

 

Ok well...I found the following. It's rather technical, but it's a good explanation of how we lose data/resolution in the highlights with compressed RAW. Thom Hogan seems to agree, which had spawned the research. I think it still depends on certain conditions as to wether its relative or not. The complete article is at
http://www.majid.info/mylos/weblog/2004/05/02-1.html
--------

The D70 only offers compressed NEFs as an option, but mercifully they have improved the performance. Ken Rockwell asserts D70 compressed NEFs are lossless, while Thom Hogan claims:

Leaving off Uncompressed NEF is potentially significant--we've been limited in our ability to post process highlight detail, since some of it is destroyed in compression.
To find out which one is correct, I read the C language source code for Dave Coffin's excellent reverse-engineered, open-source RAW converter, dcraw, which supports the D70. The camera has a 12-bit analog to digital converter (ADC) that digitizes the analog signal coming out of the Sony ICX413AQ CCD sensor. In theory a 12-bit sensor should yield up to 212 = 4096 possible values, but the RAW conversion reduces these 4096 values into 683 by applying a quantization curve. These 683 values are then encoded using a variable number of bits (1 to 10) with a tree structure similar to the lossless Huffmann or Lempel-Ziv compression schemes used by programs like ZIP.

The decoding curve is embedded in the NEF file (and could thus be changed by a firmware upgrade without having to change NEF converters), I used a D70 NEF file made available by Uwe Steinmuller of Digital Outback Photo.

The quantization discards information by converting 12 bits' worth of data into into log2(683) = 9.4 bits' worth of resolution. The dynamic range is unchanged. This is a fairly common technique - digital telephony encodes 12 bits' worth of dynamic range in 8 bits using the so-called A-law and mu-law codecs. I modified the program to output the data for the decoding curve (Excel-compatible CSV format), and plotted the curve (PDF) using linear and log-log scales, along with a quadratic regression fit (courtesy of R). The curve resembles a gamma correction curve, linear for values up to 215, then quadratic.

In conclusion, Thom is right - there is some loss of data, mostly in the form of lowered resolution in the highlights.

 

I just wanted to make it clear, that was an excerpt from a more complete article at:
http://www.majid.info/mylos/weblog/2004/05/02-1.html

 

Excellent analysis Christian. Any way that you could make a .jpg of the Excel graph and post it here so that we can see the magnitude of the compression?

Thanks!!

 

Jarrell,

I don't worry about most of it. If you're happy with your pictures that's more than enough.

Personally, I come from a technical background, as I suppose many of us do, at least enough of us to argue about things and perform various tests and compare results like these. And this whole thread should have taken place on the Technical Forum anyway.

But in general, I think we have enough information to say that there is at least some clipping of the highlights that will occur under current Nikon NEF compression algorithms, some sacrifice in editable space (from making the 12 bits into 9 bits), and that you will either notice it or not depending on what type of highlights you have and how deep you edit it, either at the RAW or PS converted level. So in the end you still just have to do your own testing, because nobody else is going to shoot identical to you or edit exactly the same.

I would still reccommend uncompressed NEF for compound editing or highlight critical situations.

 

Sorry but it was not my analysis. I should have put end quotes at the bottom of my post. At the beginning I stated the following was an excerpt from a complete article that could be found at http://www.majid.info/mylos/weblog/2004/05/02-1.html.

Sorry for any confusion on that.

 

I think you hit on a key point Jeff. I have read a number of pundits pontificating on the loss of granularity in highlights but I have yet to see an example to demonstrate this, and I have been unable to see it myself.

If anyone does have an example that shows the difference I would be very interested in seeing it.