The Open RAW Initiative - response to Nikon and others

[Retief]

I, as many others on this forum, have been involved to greater or lesser extents with a group which is proposing that all RAW formats be openly documented. The first press release went out today from this group and I highly encourage all to look to this site:

http://www.openraw.org/

Much good informtion to be found here and the more support the better.

One nice thing about this initiative is that it is not just Nikon specific, but across the board.

Thanks for looking......

 

[bpetterson]

Bill;
Yes I have been reading what many people have been saying and then repeating themselves on many other forums.

I'm getting tired of reading the same individual complaining.

Now some people are solving their own problems by updating their
computers.

I also have written Nikon urging them to stand firm.

Now I am going to take pictures and enjoy life without reading
about any problems with anybody.

Birger

 

[Retief]

Bill;
Yes I have been reading what many people have been saying and then repeating themselves on many other forums.

I'm getting tired of reading the same individual complaining.

Now some people are solving their own problems by updating their
computers.

I also have written Nikon urging them to stand firm.

Now I am going to take pictures and enjoy life without reading
about any problems with anybody.

Birger

Birger, I completely agree with you, which is why I think this is important, especially as it turns out that this is not just a Nikon issue. Luckily, I am too cheap to be affected right now :lol: , but as I have learned more and realized how this is across the manufacturers I do think it is a good thing if we can push for more open documentation.

And it is quite nice to be dealing with a group of folks who don't simply complain, but actually look for solutions. Except for some of the "comments", there is hopefully much less "problems and complaining" there than on the more open, pun intended, forums.

 

[bpetterson]

Bill;

If I were younger, I would fund a new company, find some smart programmers, call on Nikon [a visit not an Email] get the Nikon SKD and see what I could produce to meet the problems of the individuals who seem to take 2500 raw files per day or any day.

Subscribe to see EXIF info for this image (if available)

Birger

 

[Man-Fai Wong]

Bill,

I can understand the concern and desire for open format, but I do wonder though whether now is the right time for such a thing. Forcing an open format on the camera makers will mean restricting their ability to advance their technologies, and their technologies are not matured to the point where it makes good sense to apply such limitation, IMHO.

Forcing an open format on the camera makers would be almost like forcing every software maker to use the same SDK/API provided by the camera maker. So which one is really better?

And the reality is that it's much harder to get the camera makers to agree to an open format than for the software makers to simply keep up w/ the camera makers, IMHO. To wit, just look at what goes on in the audio and video industries. Look at what's been going on w/ SACD vs DVD-Audio, HD-DVD vs Blu-ray, and all the previous battles along those lines.

OTOH, I suppose it's debatable that it's best to go for an open format *before* the differing technologies mature and stances become intractable as in the cases w/ the audio and video worlds. But no matter what, getting these longtime rival competitors to sit down and cooperate is not much more than mere pipe dream. Even when faced w/ the serious threat of Microsoft and NT, the Unix community never got together well enough to get very far. Maybe if Microsoft bought out Adobe and one of the smaller camera makers (perhaps Olympus and push hard for 4/3 format or perhaps Fuji and develop around their SuperCCD) and started taking big market shares w/ DNG, then that might be enough to force the issue on Canon, Nikon, et al.

Birger,

I agree w/ you to some extent. I do find it a bit odd w/ those complainers who complain about the WB issue for dealing w/ tons and tons of RAW files. It's not like ACR ever got the colors right so that you can just use the in-camera WB setting and let things fly. I thought those folks just use a gray card for WB reference anyway.

Sure, it's a concern that Nikon *might* lock out other developers, but seems like most of them are just complaining for the sake of complaining.

_Man_

 

[Retief]

Man, I would suggest that you peruse the site and look back at some of the archived material and arguments. I, for example, argued much the same as you, and to be honest I am not at all sure this will result in in this open documentum in the short term, but I do think that having the documentation is going to be needed in the long term. Remember as well that this is not by any means an attempt to force a specific format on any manufacturer, simply to ask them to make available the specs for how they write the RAW file as it comes from the camera. If they do this we will still get the complaints from some that they didn't get the "new" specs soon enough, but you can't fix everything for everybody :wink:

If this were a push to force the manufacturers to use the same standard, DNG for example, as the RAW format "for all time", I would be in complete agreement with you.

I also agree with both you and Birger regarding a large number of the complainers and complaints we are hearing. I think that some Adobe proponents, and it is certainly debatable how much influence Adobe may have had, have made this a much bigger issue than needed. Look at Bibble for example. Eric made it work and I didn't hear him going public with huge complaints.

As to the timing, I don't know if there ever is s "right" time, but I do know that if the formats are documented and made available, there is a better chance of availablity of software into the future which will still support the old bits. Now, as I stated in my comment on the site, this also makes it incumbent on 3rd party developers to not screw up the data while they are doing "great and wonderful" things.

 

[DavidM]

It seems to me that lots of people confuse "open format" with "open access". I really don't care if the format is open, so long as I can achieve what I want with the pictures I take.

I happen to prefer Capture's rendering over ACR's, probably because recovering highlights is less of an issue to me than overall image fidelity (I've heard ACR is better at recovering highlights, and I accept that at face value). For that reason, I'm not feeling the angst that a lot of other folks are right now, yet I still respect their feelings on the matter.

My point is simply that Nikon designs the system as a whole, and they have chosen to designate their API (as part of the SDK) as the "public" gateway to the data of a raw image; if that gateway doesn't provide the data that users need and want, then Nikon should be pushed to enhance the API instead of pushing them to publish the internals of a RAW file. I believe an open RAW format would ultimately create more problems than it's worth (my opinion of course, but based on a software development background).

I've never bought into the theory that Nikon is trying to sell more Capture boxes with all of this. But I do believe it is pointedly trying to establish the dividing line between where their system stops and the image data is made available to the user.

 

[Chris101]

Thanks for the pointer Bill.

I am a fan of neither the FUD coming from Adobe nor the SDK lock-in advocated by Nikon. An open RAW format may indeed be the way to the future. I'd much rather that the owner of an existing format, such as NEF open it up, but I suspect all the growth of the format has been hardwired and kludged, hence the secrecy.

The open software movement, with few exceptions, has not produced the stellar successes it promised though. I would not mind an owned format that is open to inspection and extensible enough to be viable for future developments. I don't know the DNG is such a format, and it will be interesting to see what comes from Open Raw.

 

[Chris101]

... they have chosen to designate their API (as part of the SDK) as the "public" gateway to the data of a raw image; if that gateway doesn't provide the data that users need and want, then Nikon should be pushed to enhance the API instead of pushing them to publish the internals of a RAW file. ...

Whenever you work through an API you are limited to the parts the API designers think are important, thus limiting things that are currently outside the concept horizon of the designers.

An open format for something as complex as full color high resolution imagery will benefit from minds outside of the camera's manufacturer designing tools for manipulating the data. If outside designers are bottlenecked by too narrow of an api then the tools available to us users will be limited. I would like to be able to pick and choose which tools from different programs I would have in my toolbox. Unfortunately we are quite a distance from that type of modularity.

 

[Iliah]

As a first step, I would suggest to compile a list of tags that allow for basic converter functionality, and ask camera-makers to "open" them.

This will also allow photographers to see how few of those tags are actually used by major third-party converters, and how irrelevant are "archival" fears.

Most important, that will allow everybody suggest the tags they want to have and think of how those tags will be used in practice.

 

[Man-Fai Wong]

Sounds like a good approach, Iliah, especially since the current atmosphere seems to be a bit too hostile towards the camera makers and not placing enough responsibility on the 3rd parties involved, especially Adobe, or even many of the complaining end-users, who seem not to want to understanding the situation -- there is definitely a consumerism mentality going on in the latter.

_Man_

 

[Man-Fai Wong]

Well, DPR just did an interview w/ David Coffin. It's an interesting read. Nice reference to the economic principle of "complements". Certainly, helps get a clearer big picture on what's going between Adobe and Nikon, et al. even though most of us probably had some ideas along similar longs already.

http://www.dpreview.com/news/0504/05042701davecoffininterview.asp

_Man_

 

[papafoxtrot]

The Problem is not thinking the Problem Through...

The problem currently is that camera manufacturers have not thoroughly vetted the issue of digital film.

Historically, for example, Nikon is a mechanical and optical engineering firm (That's what all their cameras were up until the early 80s). As time progressed, they had to start resorting to software to operate their cameras. (In the Aviation industry this has become known as fly-by-wire).

As they moved to digital, more and more software has been required to accomplish what a camera does. This software (aka firmware) is integral to how the image is accomplished in-camera and delivered to your Flash card.

So now, Nikon is a Mechanical/Optical Engineering (Hardware) AND Software Company! My guess, though, is that most of the Leaders of the company still are grounded in Mechanical Engineering and Optical backgrounds...and that the troops are the ones fighting the software issues. It takes a long time for a company grounded in hardware to understand the importance of software.

In the software business you have to define and account for the workflow that will be required to use the end product and you need to consider the full lifecycle of the resulting document. You also have to understand how to market it! It is my opinion that Nikon and probably all other current camera manufacturers have not done their homework here.

Best example: Flagrant disregard for the use of DEE as Nikon moved forward with its new NC feature of D-lighting. I have photos that are less than one year old in NEF formats that cannot be recovered with their DEE settings, NOW. I'm not even sure which NEF files those are!

Another example is simply the poor programming that has been accomplished with NC. It is in many ways outstanding and provides very good editing capability on your RAW files...but it still crashes way too often and it certainly is way too slow for many photographers who are trying to make a living. It is not a product that any software manufacturer would be proud of. It is a product that Nikon Management allows to be shipped because they do not understand the Software side of their business, yet. And these are the managers who are affecting your digital photo future.

By the way, the above problem would never have happened with a mechanical or optical issue.

Canon 300D users no longer have Manufacturer support for their RAW files. They are now totally dependent on 3rd party software vendors.

Unlike any problem we've ever faced in photography before, the integrity of your original is at stake! This is no small issue and should be the concern of every digital photographer who thinks "The Image Matters".

This should be your concern based on the fact that you take pictures and want them to last. This should be your concern no matter how young or how old you are. Now, if you are not bothered by this, at least make sure that you are archive your post-processed files in a TIFF format. The Tiff is one format that should protect one version of your image while the RAW issues get worked out.

I've tried to present my thoughts in a level headed manner. Am I passionate about this subject? Yes. Am I emotional - probably - but if you look at each of my statements I think you'll note they are mostly grounded in good logic. I'm typically a pretty happy-go-lucky kind of guy...but this is a serious subject that should concern every one of us here. (And I know that this forum is sort of a refuge away from the madness of DPReview...but some things you cannot totally get away from.)

Bottom line: Write Nikon a letter, please. You don't need to even advocate an open raw standard. Just ask them to think through the entire lifecycle of the photograph...and to consider the different use cases - From a fine arts photographer to an event photographer. If they did this we would not be worrying about the long term access to our precious soft copy negatives and we wouldn't be worrying about clumsy workflow with Nikon Capture, or inability to fully use other prominent 3rd party tools that we need to get our work done. If you are of a mind...tell them you'll spend $200 for Capture if that is what it takes to get it right!! I'm sure I lost $100 in productivity just writing this letter, alone.

With regards and respect -- Paul



Write to Nikon in Japan:
Makoto Kimura
President of Imaging Company
Nikon Corporation
Fuji Bldg., 2-3, Marunouchi 3-chome
Chiyoda-ku, Tokyo 100-8331
Japan

 

[Retief]

I'm glad to see this thread getting a bit of traction, and one thing that I want to reinforce, over and over and over again, is that this is NOT a Nikon specific issue. Read the David Coffin interview, look at Canon cameras no longer supported, even look at the claims that Sigma has "opened up their format" only to find that they produced a document that describes bits and pieces, but not the bits and pieces to deal with the image.

I agree this affects us all in the long run, which is why I have been involved with the OpenRaw folks since inception, but I think we have to be cautious as well that the baby does not go down the drain with the bath water. Do I think for a moment that suddenly 2 months from now the Nikon NEF will be fully documented and made public? Heck, I wouldn't even bet your money on that, but the only better time to have begun this push would have been 4 or 5 years ago. Look at the Open Raw site, this is getting traction all over the place now.

Paul, I don't disagree that a letter to Nikon, specifically regarding the DEE issue that was plain dumb, would not hurt. But we do have alternatives and I have a lot of data archived for other applications that is no longer supported by the newer version, this is nothing new in the industry either, and it sucks no matter which app and data is affected.

Man and Iliah, great points on how to begin. If neither of you folks want to make this suggestion at the Open Raw site, I'll be happy to do it for you, but I'd rather see you get direct credit. Let me know.

Chris, you are right, and the FUD has been flying from every direction. I find it quite disengenious for Thomas Knoll, I am not saying Adobe as noted before it was Thomas and not offically Adobe who complained, to ***** about Nikon and not note the others, see the list from David Coffin. Then again this whole issue has pushed things to more of a head. I'm not sure if there is a list on the OpenRaw site yet of all the media contacts, but I can tell you it is incredibly extensive, and some very large photo groups are getting involved as well. In the end the worst that happens is that a bunch of folks waste time, at best the formats get documented. Open Raw is not saying "Same Format For Every Camera", that is Adobe DNG, but is asking that the format be made known so that 3rd parties don't have to "figure it all out" every time. I agree with how it may have grown, which may make things a bit ugly, but hey, if that is what is, the dev folks can deal with it.

David and Chris, the API/SDK is whatever Nikon chooses it to be. The complaints are that it is "slow", well duh the rendering is which just may be why many think it is better, and that you don't get access to the raw data. The second point is exactly what an API/SDK is for, to shield you from that. I suspect that even with the format open Nikon will still do a better job simply because they have more intimate knowledge of the hardware/firmware/software together.

So, in the end I don't see how tossing a comment out on the Open Raw site can do any harm, and I do think there is a better than even chance of doing some good. I am amazed to see the number of new members as well as the variety of gear showing up. Couple that with the inquiries coming in now and I am encouraged.

 

[cmpalmer]

I think most of Paul's remarks are right on the money. The biggest benefit to opening standards (either by using a shared standard or just publishing the specs on internal standards) is that you know that in the future, it will be possible to read your original files.

I have a handful of old word processing files from a company that is long out of business. The only converters I could find (until recently) would, at best, extract the basic text from the files but lose all of the formatting. For some reason, after five or more years, I managed to find a free importer last week that successfully converted those old files to MS Word.

Now, I don't think Nikon is going out of business soon and I do understand their desire to protect their intellectual property, but in 10 years (which is a loooong time in technology years), will the latest software still read your original files?

A valid alternative is an intermediate format that preserves *all* of the information present in the original file, even if it requires a licensed (purchased) copy of software.

 

[Man-Fai Wong]

Paul (and others),

The thing is that this fear is in large part based on the assumption that 3rd party support would eventually cease just because of certain assumptions about what's going on now.

I would submit that's a very unlikely scenario unless the camera makers actually go to the extreme of properly using strong encryption to lock out 3rd parties *AND* effectively throw away the "key" themselves (by ceasing support themselves for whatever reason).

If there is actual $$$ to be made, somebody will more than likely pick up the slack. If there is no $$$ to be made, then no amount of letter writing will help. Now, I don't mean it has to be $$$ made directly from offering support, but could be indirect also. For instance, if Nikon went to the feared extreme and then ceases support for a lot of iterations of their NEF format while other makers do not do that, then Nikon will lose marketshare due to customers losing faith, which means $$$ lost.

We should recognize that much of what's happening right now is probably more about posturing between the Adobe and Nikon than Nikon necessarily heading toward that feared end. Even if Nikon decides to go the extreme route tomorrow for all future bodies, your current existing NEFs will still have 3rd party support and will still be accessible w/ today's Nikon software. Just don't buy Nikon anymore if it comes to that. You know the "funny" thing is that all this is coming back to bite all the Canon shooters who want a new model every 6 months because now Canon can't keep up w/ support -- today, it's D30, tomorrow 10D/20D/etc.

Of course, 3rd party support might not last forever if the costs involved are too great, but then again, if the costs are too great, then it probably means not enough people are willing to pay enough for the support. That would probably be the same underlying reason why a camera maker might cease support itself.

Sure, let all the camera makers, not just Nikon, know your fears, even if you only own Nikon gear -- afterall, you're a potential customer for non-Nikon gear. But I'd submit that most of the fears are a bit overblown and not accounting for market realities.

Also, if it really comes to that, it's a virtual impossibility for your RAW files to become so obsolete so fast that you have no opportunities to convert them to some other high fidelity format anymore, eg. 16-bit TIFF as you pointed out or maybe Adobe's DNG (ack! ). The only threat here regarding existing RAW files, and a somewhat remote one at that, involves future advancements of RAW conversion of your existing RAW files, not so much whether you can still convert them at all. Again, if and when support ceases permanently, just do a final conversion w/ the best latest available software. At least your RAW files and whatever other final format you choose are digital and won't wear away from errosion over time like film does.

RE: the example of word processing documents and such. I assume most of those cases either involved small companies w/ relative small customer base or are simply niche markets. Otherwise, somebody would be persuaded to make $$$ by offering support of some sort. Consider the computer programming field itself. Who's really writing new code in Cobol now? But you can bet there's still $$$ to be made just for the maintenance as long as the legacy stuff are still needed. If you know Cobol, go get that $$$ -- maybe I should go for that when I'm too old to keep up w/ the field. If you don't know Cobol, you can learn it.

Finally, I think we have to remember that these are all business entities w/ $$$ as their bottomlines (for the most part). If we believe there's a problem and want to be proactive to help solve it, we need to give proper consideration to the big picture for all involved, not just for our own fears of obsoleted photographs or whatever inconveniences or lack of choices. It does little good to merely yell at somebody to simply "do better" afterall.

_Man_

 

[Iliah]

Bill,

I think I made that suggestion in OpenRAW list. Maybe it really deserves some discussion.

I know from experience that for now only few tags present in raw files are used in 3rd party raw decoders. Camera quality is often judged upon the results produced by those independent decoders; and it seems to be all the rage at the moment comparing cameras using same converter. But those decoders often level the cameras, using small subset of tags; and demosaicing/colour management/noise reduction (or lack of it, like the case of infamous banding in 20D) found in those converters do not make justice to the cameras. For camera makers this can become an inconvenience, as it is increasingly difficult to compete under given situation.

Paul,

Is it that previous versions of Nikon Capture supporting DEE are removed from Nikon site?