Back in March of 2006, I interviewed Alan Cote, the Supervisor of Public Records in the Public Records Division of the Massachusetts Secretary's office. Alan had testified back in October of 2005 in the hearing where Peter Quinn had been called on the carpet by Senator Marc Pacheco, the Chair of the Senate Committee on Post Audit and Oversight. At the Pacheco hearing, Alan had professed neutrality about ODF, but also doubts that document formats could provide a useful tool for document preservation.
What struck me most forcefully at both the hearing as well as the interview was that Alan presumably should have been one of the biggest proponents of open formats, rather than a doubting Thomas. Why? Because the process he now follows to preserve electronic documents seems almost comically cumbersome and tedious. Briefly summarized, it involves recopying every single electronic document every five years or so onto new media (electronic media degrade surprisingly rapidly) in multiple formats (because formats are regularly abandoned). Shouldn't someone stuck with such a chore be desperate to find a better way?
Apparently, preserving documents is child's play compared to preserving modern movies, especially those created initially in digital form. How bad - and expensive - is that? According to a 74 page study released by the Academy of Motion Picture Arts and Sciences (AMPAD) to a limited audience in early November, preserving a full-length digital movie can cost $208,569 (dramatic pause) per year. The reasons are exactly the same as for digitized documents, and the currently available means of preservation are the same as well. The amount of data involved, however, is vastly greater - and no commitment to remain faithful to a single standard (yet) exists to ensure that future technologies will be able to display movies created using today's techniques.
The name of the report is aptly called "The Digital Dilemma." An article based on the report appeared in early November in The Hollywood Reporter, and a longer article is in this morning’s New York Times.
The issue is of greater urgency, perhaps, to the film industry than document preservation has been to most commercial interests, because much of the ultimate profit in video properties is reaped downstream in time. And according to the report, the risk of "digital extinction" can arise for a given property in as little as 18 months. The hazards of digital storage are summarized by the Times article as follows:
To begin with, the hardware and storage media — magnetic tapes, disks, whatever — on which a film is encoded are much less enduring than good old film. If not operated occasionally, a hard drive will freeze up in as little as two years. Similarly, DVDs tend to degrade: according to the report, only half of a collection of disks can be expected to last for 15 years, not a reassuring prospect to those who think about centuries. Digital audiotape, it was discovered, tends to hit a “brick wall” when it degrades. While conventional tape becomes scratchy, the digital variety becomes unreadable.
In contrast, the traditional process for archiving video is reliable and inexpensive – a methodology developed after learning the economic lesson of carelessness the hard way. That happened when the ability to deliver dozens of channels via cable television created a demand for old video that in most cases turned out to be either lost, destroyed, or degraded beyond use.
Now the same danger has arisen again, and AMPAS has gotten on top of the issue if not in advance of the problem, at least more quickly than occurred in the days of unstable celluloid film. Milt Shefter, the project leader on the AMPAS Science and Technology Council’s digital motion picture archival project, is quoted in The Hollywood Reporter article as follows:
"We are already heading down this digital road … and there is no long-term guaranteed access to what is being created. We need to understand what the consequences are and start planning now while we still have an analog backup system available." … Shefter noted that a requirement for any preservation system is that it must meet or exceed the performance characteristic benefits of the current analog photochemical film system. According to the report, these benefits include a worldwide standard; guaranteed long-terms [sic] access (100-year minimum) with no loss in quality; the ability to create duplicate masters to fulfill future (and unknown) distribution needs and opportunities; and immunity from escalating financial investment.
Today, 100 years of storage of a film master of a movie in a salt mine would cost about $105,900, in today’s dollars, while the digital master could cost as much as (where’s that dramatic pause again?) $20,856,900. Assuming you could find a movie worth spending that on.
If this sounds familiar and worrisome, it should. We are rushing pell mell into a future where we only focus on the exciting benefits of new technologies without considering the qualities of older technologies that are equally important – such as preservation – that may be lost or fatally compromised by migrating to a new whiz-bang technology. I wrote about this some time back in an article that I would recommend reading to make this phenomenon more real, called Clay Tablets, iPods and Evo/Devolution.
The moral of the story is that we should look before we leap, and not transition from one technology to another until we have figured out how to hang on to what we’ve got.
For further blog entries on Standards and Society, click here
Which brings up the interesting question of why we are assigning copyrights for terms much longer than current technology can deliver? If works entered the public domain sooner then they would be preserved in the public domain indefinitely in a multitude of places and forms. This is a good argument for reducing the length of copyright and specifically allowing consumers to migrate copyrighted works to new formats so they won’t be lost to society forever.
Doug Webb
Powell River, BC
Canuckistan
I agree that long term digital storage is a huge problem that everyone is ignoring. However, I would like to point out a couple of small things. For the $208,569 per year, you get a lot more than the film put down a Salt Mine. I also think that the $208,539 is an optimistic figure because the format changes in the next 30 years are likely to be immense (see below).
So firstly, if you want to add a brand new scene to Alien, the salt mine doesn’t help too much because you also need the rubber alien costume, the spaceship set, a ginger cat (Sigourney Weaver doesn’t age).
For a modern CGI and computer-rendered Sci-Fi film, you will have all the data and programs used to render the film. So if you want a new scene, you can rerun the alien program and the giant spaceship program planet program.
Assuming of course all this has been done in open and standardised formats with programs that you have the source code to; so you can make the programs and data work on whatever computers we might have in 30 years time. So this is where the extra cost comes in, most 2037 people probably will not be very interested in flat screen with moving picture on. So the labour and expertise in turning a 2007 film into a 2037 holodeck feature (or whatever they have) is what will really cost.
Zeth
P.S. Andy, can I humbly request that you consider OpenID as an alternative to creating an account? It is slowly emerging as a standard (a random geeklog user talks about OpenID here).
"if you want to add a brand new scene to Alien, the salt mine doesn’t help too much…"
Well, perhaps that’s part of the problem. Filmmakers (studios, directors, producers) are all too eager to go back and re-write existing films. As a prime example, in the last of the original Star Wars Trilogy, the three figures that appeared in the funeral pyre were Yoda, Obi-Won, and Anakin Skywalker. George Lucas seemed to have no problem changing the ghosts of those three, changing the actor who played Anakin from Sebastian Shaw (an older Anakin/Darth) to Hayden Christensen – a much younger man. For some bizarre reason, he didn’t change Obi-Won from Alec Guinness to Ewan McGregor, so there’s some weird disconnect there.
To me, it altered the film for the worse. Yes, the digital special effects were better on the DVD release than they were in the original 1977 – 1983 films, but we accepted Star Wars for what it was, warts and all. The same is true for other films and TV shows. One other one that comes to mind is Star Trek (TOS) which seems to have taken a digital makeover on many space shots. I first noticed it on the episode "The Doomsday Machine" (the one with the aluminum foil-like unrolled cornucopia). Yeah, the effect was corny, but we accepted it. The "revised" version has a CGI doomsday machine and the Enterprise makes lovely arcs around it. Better effects, yes, but it doesn’t "fit" the whole lovable campiness of the series.
Why can’t filmmakers leave their work alone? For goodness sakes, everyone ELSE copies each other in Hollywood, why can’t filmmakers just copy themselves if they think they can do better? The reason is they usually CAN’T. No posthumous added scenes are going to make the film "new", and they only serve to annoy those who appreciated the originals. If George Lucas thinks that the original Star Wars trilogy stinks, for goodness sakes, make a new one, using the actors that were used in the last three. Just don’t screw up a classic, okay?
Zeth,
Part of the cost, according to the Times story, is that they want to hang on to quite a bit of stuff besides the original master (bloopers, even the rubber suit outtakes). They didn’t go into any detail, though, to explain exactly how you could get to such an enormous cost per year, even if you were transposing all that data annually.
On OpenID – sounds like a good idea, but it might be awhile before I can get around to it. If you browse around this site, about the most tactful descriptor you might use would be "retro." I’m in the middle of doing a site-wide update that will take a year and, since we’re using an outside Web developer, is going to cost a lot of money. You can see what the new look will be like by visiting the Standards Today pages. We’re working on moving the MetaLibrary over to the same look and feel (this will go live in February), the Home Page after that, and so on.
– Andy
If you browse around this site, about the most tactful descriptor you might use would be "retro."
Well I think no one will mind that, a lawyer with a highly popular blog that can be read by geeks too is still very much ahead of the curve by several years (decades?) not retro at all.
>You can see what the new look will be like by visiting the Standards Today pages.
Very nice, I’m a big fan of the Silicon Valley clinical look, as used by Google and others. It makes things very easy to read.
Merry Christmas.
I was at the public hearing (though it was not called that at the time), and I too was struck by what seemed like a huge waste of tax payers money. Why doesn’t Allen Cote want a better system? I relayed this question to my mother (who was a town nurse dealing with similar issues at the time). It really comes down to two things: money and power. At the heart of the hearing was whether or not the IDT really has the power they felt they had to make the changes they wanted to make. They stepped on many toes in other parts of the executive branch, including the Department of the Secretary. Here is a lowly IT department with no elected official encroaching on the fiefdom of an elected official who needs to justify expenses. In the short term ODF will make more work for the Department of the Secretary (work which will eventually need to be done anyway), but will reduce overhead drastically over time. The archival group has its own IT department with their own chain of command, up to the Governors office. The moves the IDT was making were a threat to this other entrenched and very influential group. Each town and municipality needs to communicate with this Secretaries office. Immunization records, birth and death records, housing, fire permits, town budgets, etc. The Secretaries office is extremely powerful and well connected. Spring something on them like ‘We will recommend this format for digital archival purposes in the Executive branch’; well you will be squashed and fast.
There is absolutely no question that there was a lot of turf protection going on. The ITD appears not to have been as tactful and communicative as they might have been with Cote and his boss, and Peter Quinn and his staff thought that the reverse was true as well. But even beyond that, when I spoke one on one with Alan Cote (who was the first to admit that he had no technical training), it was clear that he either didn’t really "get" the concept of open formats, or simply didn’t believe that they would provide the benefits that they promised – which, to be fair, remains to be proven, because people have to faithfully implement them over the long term in order for them to deliver.
– Andy
(sarcastic) I never knew that cost of 100 racks of films in salt lake is much cheaper than half rack of DVDRAM that replace it.
Let face it, somebody simply use plain accounting to do the calculation without thinking the factor of the technology. The digital medium has the potential of shrinking exponentially. However, the film medium will expand in linear forms.
So 10,000 films mights just take 2 racks space with terabytes digital storage media. How many space does it take to store 10,000 raws film reels? 500 perhaps?
I think, though, that an reel of film, kept away from heat, light, and moisture, only costs to place it into the salt mine, where a digital medium (tape, DVD, or ???) needs to be pulled up regularly and copied from one medium to another (either of the same kind or of a successor). In addition, newer media are often sensitive to electromagnetic energy.
If we accept this as fact, then the traditional media have a front-loaded one-time cost followed by whatever it costs to bring it back out in fifty or one hundred years. The digital media will require all the same front and back costs, repeated every two to five years whenever the copying operation takes place.
That’s exactly the point of the article, although since it’s not yet public it’s not possible to see how they came up with the specific numbers they did. Presumably you might be able to automate some the updating, I would think, especially if it would result in savings of any magnitude, given their projections, which might have been based on mostly manual management.
– Andy
For $208,569 you can buy over 160TB of storage! That’s not just raw price for raw storage on HDD: it includes backup, regular integrity checks, online access, etc. Why the film includes so much data ?
And if it’s not just question of information of preservation then what else is included ? You can easily inflate film storage price by including unneeded operations there too…
P.S. And if analog preservation is so cheap and painless then why we original version of Astro Boy no longer exist ?