Home > Standards Blog

Advanced Search 

Welcome to ConsortiumInfo.org
Wednesday, September 03 2014 @ 12:10 AM CDT

Email Article To a Friend View Printable Version

As Go Document Formats, So Goes Video

Standards and Society

Back in March of 2006, I interviewed Alan Cote, the Supervisor of Public Records in the Public Records Division of the Massachusetts Secretary's office.  Alan had testified back in October of 2005 in the hearing where Peter Quinn had been called on the carpet by Senator Marc Pacheco, the Chair of the Senate Committee on Post Audit and Oversight.  At the Pacheco hearing, Alan had professed neutrality about ODF, but also doubts that document formats could provide a useful tool for document preservation.

What struck me most forcefully at both the hearing as well as the interview was that Alan presumably should have been one of the biggest proponents of open formats, rather than a doubting Thomas.  Why?  Because the process he now follows to preserve electronic documents seems almost comically cumbersome and tedious.  Briefly summarized, it involves recopying every single electronic document every five years or so onto new media (electronic media degrade surprisingly rapidly) in multiple formats (because formats are regularly abandoned).  Shouldn't someone stuck with such a chore be desperate to find a better way?

Apparently, preserving documents is child's play compared to preserving modern movies, especially those created initially in digital form.  How bad - and expensive - is that?  According to a 74 page study released by the Academy of Motion Picture Arts and Sciences (AMPAD) to a limited audience in early November, preserving a full-length digital movie can cost  $208,569 (dramatic pause) per year.  The reasons are exactly the same as for digitized documents, and the currently available means of preservation are the same as well.  The amount of data involved, however, is vastly greater - and no commitment to remain faithful to a single standard (yet) exists to ensure that future technologies will be able to display movies created using today's techniques.

The name of the report is aptly called "The Digital Dilemma."  An article based on the report appeared in early November in The Hollywood Reporter, and a longer article is in this morning's New York Times.

The issue is of greater urgency, perhaps, to the film industry than document preservation has been to most commercial interests, because much of the ultimate profit in video properties is reaped downstream in time.  And according to the report, the risk of "digital extinction" can arise for a given property in as little as 18 months.  The hazards of digital storage are summarized by the Times article as follows:

To begin with, the hardware and storage media — magnetic tapes, disks, whatever — on which a film is encoded are much less enduring than good old film. If not operated occasionally, a hard drive will freeze up in as little as two years. Similarly, DVDs tend to degrade: according to the report, only half of a collection of disks can be expected to last for 15 years, not a reassuring prospect to those who think about centuries. Digital audiotape, it was discovered, tends to hit a “brick wall” when it degrades. While conventional tape becomes scratchy, the digital variety becomes unreadable.

In contrast, the traditional process for archiving video is reliable and inexpensive - a methodology developed after learning the economic lesson of carelessness the hard way.  That happened when the ability to deliver dozens of channels via cable television created a demand for old video that in most cases turned out to be either lost, destroyed, or degraded beyond use.

Now the same danger has arisen again, and AMPAS has gotten on top of the issue if not in advance of the problem, at least more quickly than occurred in the days of unstable celluloid film.  Milt Shefter, the project leader on the AMPAS Science and Technology Council's digital motion picture archival project, is quoted in The Hollywood Reporter article as follows: 

"We are already heading down this digital road ... and there is no long-term guaranteed access to what is being created.  We need to understand what the consequences are and start planning now while we still have an analog backup system available." ... Shefter noted that a requirement for any preservation system is that it must meet or exceed the performance characteristic benefits of the current analog photochemical film system. According to the report, these benefits include a worldwide standard; guaranteed long-terms [sic] access (100-year minimum) with no loss in quality; the ability to create duplicate masters to fulfill future (and unknown) distribution needs and opportunities; and immunity from escalating financial investment.
Today, 100 years of storage of a film master of a movie in a salt mine would cost about $105,900, in today's dollars, while the digital master could cost as much as (where's that dramatic pause again?) $20,856,900.  Assuming you could find a movie worth spending that on.

If this sounds familiar and worrisome, it should.  We are rushing pell mell into a future where we only focus on the exciting benefits of new technologies without considering the qualities of older technologies that are equally important - such as preservation - that may be lost or fatally compromised by migrating to a new whiz-bang technology.  I wrote about this some time back in an article that I would recommend reading to make this phenomenon more real, called Clay Tablets, iPods and Evo/Devolution.

The moral of the story is that we should look before we leap, and not transition from one technology to another until we have figured out how to hang on to what we've got.

For further blog entries on Standards and Society, click here

sign up for a free subscription to Standards Today today!

As Go Document Formats, So Goes Video | 11 comments | Create New Account
The following comments are owned by whomever posted them. This site is not responsible for what they say.
As Go Document Formats, So Goes Video
Authored by: Anonymous on Sunday, December 23 2007 @ 01:31 PM CST
Which brings up the interesting question of why we are assigning copyrights for terms much longer than current technology can deliver?  If works entered the public domain sooner then they would be preserved in the public domain indefinitely in a multitude of places and forms.  This is a good argument for reducing the length of copyright and specifically allowing consumers to migrate copyrighted works to new formats so they won't be lost to society forever.

Doug Webb
Powell River, BC
Canuckistan
[ # ]
$208,569 - Apples and Oranges
Authored by: Anonymous on Sunday, December 23 2007 @ 01:43 PM CST

I agree that long term digital storage is a huge problem that everyone is ignoring. However, I would like to point out a couple of small things. For the $208,569 per year, you get a lot more than the film put down a Salt Mine. I also think that the $208,539 is an optimistic figure because the format changes in the next 30 years are likely to be immense (see below).

So firstly, if you want to add a brand new scene to Alien, the salt mine doesn't help too much because you also need the rubber alien costume, the spaceship set, a ginger cat (Sigourney Weaver doesn't age).

For a modern CGI and computer-rendered Sci-Fi film, you will have all the data and programs used to render the film. So if you want a new scene, you can rerun the alien program and the giant spaceship program planet program.

Assuming of course all this has been done in open and standardised formats with programs that you have the source code to; so you can make the programs and data work on whatever computers we might have in 30 years time. So this is where the extra cost comes in, most 2037 people probably will not be very interested in flat screen with moving picture on. So the labour and expertise in turning a 2007 film into a 2037 holodeck feature (or whatever they have) is what will really cost.

Zeth

P.S. Andy, can I humbly request that you consider OpenID as an alternative to creating an account? It is slowly emerging as a standard (a random geeklog user talks about OpenID here).
[ # ]
As Go Document Formats, So Goes Video
Authored by: Anonymous on Sunday, December 23 2007 @ 07:44 PM CST
I was at the public hearing (though it was not called that at the time), and I too was struck by what seemed like a huge waste of tax payers money. Why doesn't Allen Cote want a better system? I relayed this question to my mother (who was a town nurse dealing with similar issues at the time). It really comes down to two things: money and power. At the heart of the hearing was whether or not the IDT really has the power they felt they had to make the changes they wanted to make. They stepped on many toes in other parts of the executive branch, including the Department of the Secretary. Here is a lowly IT department with no elected official encroaching on the fiefdom of an elected official who needs to justify expenses. In the short term ODF will make more work for the Department of the Secretary (work which will eventually need to be done anyway), but will reduce overhead drastically over time. The archival group has its own IT department with their own chain of command, up to the Governors office. The moves the IDT was making were a threat to this other entrenched and very influential group. Each town and municipality needs to communicate with this Secretaries office. Immunization records, birth and death records, housing, fire permits, town budgets, etc. The Secretaries office is extremely powerful and well connected. Spring something on them like 'We will recommend this format for digital archival purposes in the Executive branch'; well you will be squashed and fast.
[ # ]
Digital storage vs film
Authored by: moo on Sunday, December 23 2007 @ 09:05 PM CST
(sarcastic) I never knew that cost of 100 racks of films in salt lake is much cheaper than half rack of DVDRAM that replace it. 

Let face it, somebody simply use plain accounting to do the calculation without thinking the factor of the technology.  The digital medium has the potential of shrinking exponentially.   However, the film medium will expand in linear forms.
So 10,000 films mights just take 2 racks space with terabytes digital storage media. How many space does it take to store 10,000 raws film reels?  500 perhaps? 
[ # ]
What exactly is included ???
Authored by: Anonymous on Tuesday, December 25 2007 @ 01:33 PM CST
For $208,569 you can buy over 160TB of storage! That's not just raw price for raw storage on HDD: it includes backup, regular integrity checks, online access, etc. Why the film includes so much data ?

And if it's not just question of information of preservation then what else is included ? You can easily inflate film storage price by including unneeded operations there too...

P.S. And if analog preservation is so cheap and painless then why we original version of Astro Boy no longer exist ?
[ # ]