PGA President Urges Producers to Finish in 4K and HDR

[Alexandros Angelopoulos Apostolos]

Perhaps this will result in some sort of change:

 

http://www.hollywoodreporter.com/behind-screen/pga-president-lori-mccreary-urges-899766

 

Apparently, there are 30 Alexas 65 now in use, with 40 more being built.

[Tyler Purcell]

I have no idea what 4k and HDR has anything to do with "longevity".

 

The problem with have now is that our digital movies are 50 - 200TB worth of data, which is expensive to store.

 

The other problem is, unlike film which can be seen by the human body without translation, digital can't. Also, digital is not impervious to degrading. The difference is obsolescence becomes greater and extrapolating more from a fixed-pixel digital image is nearly impossible, when compared to formats without fixed pixels. Still, the cost issue is absolutely the highest priority. Remember when films were melted down after use to recoup some money from their cost? That's what digital is like today. Studio's will eventually spend hundreds of millions maintaining digital archives and they will pick and choose which content is stored online or nearline. I don't know if the PGA president knows this, but since the digital revolution started, we have lost many original masters to degrading tapes caused by poor storage and bad recording methods. That will not change, there is no 100+ year magic bullet for digital media. The problem is, it's the little guys who will hurt the most. Guys who make a great product, but maybe didn't get great distribution. Those wonderful products will most likely be lost because those filmmakers can't afford to keep backing up tape archives year after year to insure they're ok. A small mistake can loose everything.

 

Sure, storage technologies are getting smaller, faster and less expensive. However, camera files are getting bigger/fatter and more difficult to work with. We have yet to see any storage breakthroughs, it's the biggest reason why we rely so heavily on the cloud for dealing with our media. The amount (cost) of server storage necessary to deal with the overhead of the next 20 years of high resolution digital masters. To this date, almost all of our data is 1080p OR LESS and we struggle to hold onto it all.

 

Unfortunately, the only 'solution' are scanned out separation prints. They will last 300 years easily and don't cost nearly anything, sitting on a shelf.

 

As a side note, a recent 2014 study stated, the previous decade was the least recorded since the early 1900's. Why? Because we create more content then ever before, yet very few people organize and backup what they've done in a way that's future proof. If the internet went down tomorrow, nobody would have anything. Where during the analog days, we'd have pictures stored in books, in our attic's, spread around the family on our mantle pieces, in our home movies, etc... All of that still happens, but a slight slip-up computer wise and everything you've done vanishes.

[Aapo Lettinen]

I think LTO tape is the only viable tape storage medium at the moment for large amounts of digital media, it is so widely used in wide variety of fields that the world can't afford losing those archives and thus considerable effort is made to preserve the systems and to ensure reliability. It is quite different with niche tape formats like D6 which is not so old standard after all but very few recorders were ever made and most of them are lost already, so you are basically quite screwed if you have mastered anything to D6 and need to read those tapes in the future.

 

I would say that LTO in the most future proof format like original camera files + TIFF + prores and combined with film separation prints would be the most reliable option by now. Cloud based systems are great but need much more maintenance than tapes generally and may cost more per TB depending on configuration and how much you use the tape system

[Aapo Lettinen]

mostly talking about LTO archives generated with for example BRU, the LTFS standard is not future proof even in 2 year time scale so it is not a good idea to use it for archiving <_<

[Alexandros Angelopoulos Apostolos]

I think that when she mentions longevity, she means that with the 4K TVs getting ever more popular, 4K content will have a longer life than anything in HD, which is obsolete even now and will be wildly unsuitable to be shown on 4K screens. I.e. 4K TVs will become obsolete in, perhaps marginally, more time than HD ones, which are outdated even now. I know that even the 4Ks will be out-of-date pretty soon, too. But that “soon” will came after the “soon” of HD screens, whose “soon” has already happened.

 

The storage problem and the longevity of digital data has stayed with me as a serious problem ever since you mentioned it not too long ago in on of the other threads. I think it was in The Hateful Eight discussion?

 

Veritas Technologies is predicting that currently 33 % of stored data in the world is ROT – redundant, obsolete, trivial. The firm predicts that if firms don’t face up to this, by year 2020, we will be spending 2200 billion dollars on storing unneeded data.

 

http://www.datacenterjournal.com/veritas-global-databerg-report-finds-85-percent-stored-data-either-dark-redundant-obsolete-trivial-rot/

Edited June 6, 2016 by Alexandros Angelopoulos Apostolos

[Brian Drysdale]

That forgets that it's the content which is king, not the resolution. For a long life it has to be something that worth watching, there are hgh resolution feature films which have been forgotten, while old 405 line B&W Dr Who still has a commerical life.

[Aapo Lettinen]

some viewers may refuse to watch a program if they know it is lower than UHD or 4K resolution.

If nobody tells them, they probably won't notice at all and will gladly watch it and enjoy it ;)

 

just like with 3D, people think they will get something more out of it all the time and thus choose the 3D version even if it's worse than the 2D :blink:

[David Mullen ASC]

People are still buying 480P DVD's over 1080P blu-rays unfortunately...

[Tyler Purcell]

Some of you may not know this, but most movie-specific content distributors require 4k source material today.

 

Also, a lot of the UHD BluRay's aren't mastered in 4k. They're a lower resolution master, upres'd to UHD. So the whole UHD BluRay thing is a total scam in my opinion.

 

In terms of LTO as the industry standard for backup, it works great if you make two copies, stored in entirely different facilities, in completely different libraries and duplicate each tape at least once every 2 years. LTO tapes aren't expensive either but the formats are always changing. So the cost to keep migrating libraries from LTO 6, to LTO 7 to eventually LTO 12, is crazy. Libraries large enough to handle a catalog worth discussing, have a hefty foot print and are expensive to keep running. Unlike film which happily sits on a shelf, digital needs electricity and technological support to keep it working. Worst part is, if an LTO tape does fail, you can't splice it back together and magically get all the data like with film. One's and zero's don't work that way, though I will admit, data recovery from LTO is MUCH better then spinning disks.

 

The great thing about film is that you could take the IP and store it at one facility, take the IN, store it in another and take your original negative and put it somewhere else. None of that costs you any extra money since those three items are already struck during the photochemical finishing process. All you have to do is be smart on whose basement you store them in! Just kidding! :)

 

Still I agree with David, most people still buy standard DVD's and are OK with them. As I've said in MANY threads, everyone was forced to buy HD TV's and until those TV's fail, the vast majority of people aren't going to be upgrading to UHD. There are cinema geeks (myself included) who will eventually migrate to UHD content and display, but outside of the people who always need new tech, the UHD market is very small. 1080p BluRay's look amazing for most people (if done right) and it's far less costly then UHD today. Now we all know that will change and there is already a provision in place for 8k BluRay, which is very interesting. I think all the "tech" will exist, I just don't think it will mean anything substance wise. There are less than 100 feature film titles on UHD today, compare that to the beginning of BluRay and DVD, it's an atrocious number. It just proves how "soft" of a launch UHD really has been, mostly because the content just doesn't exist. The big question in my mind has more to do with internet distribution and getting 4k to look good streamed into houses. That's the big breakthrough technology which hasn't happened yet because currently Netflix UHD content looks like poop, they had to lower the bitrate too much and it ruins the image quality. Satellite and cable providers are screwed as well, they have so many active channels, it would be nearly impossible for them to make UHD look good, even if they could get it to work at all. They'd have to upgrade everyone's decoder box for free? Yea, that ain't gonna happen anytime soon.

 

So call me a skeptic, but I just don't see any of this happening anytime soon. All I know is that every time I work on a big show shot in 4k, I ask myself, how are these guys backing up their movie. They've got two sets of spinning disks and that's it. I'm like, guys you've gotta pay someone to put this on tape. They're like; tape, I thought that was long gone. I'm like yea, well... it never left the IT world. People just don't know any better and they can't afford to do the proper procedures either because labs charge a lot of money. In fact I've done the math and to this day, if you account for high-end 4k camera rental/storage cost through keeping your product alive for 100 years, shooting on film is still cheaper then digital and fixed-resolution agnostic.

[Kenny N Suleimanagich]

Amy Taubin had an excellent story in Artforum Oct. 2015 about film archiving, and the huge costs of digital archive upkeep. (paywall, sorry)

Edited June 6, 2016 by Kenny N Suleimanagich

[Landon D. Parks]

I really think many people are giving whole archival issue too much difficulty. Yes, a film print can last hundreds of years (or so we are told, but so far we only have a little over a 100 years of actual proof). However, that film print is also not entirely safe from destruction. Any number of disasters from Fires to poor treatment can render a film reel useless. It's happened in the past.

 

I have been working with computers since I was 9 - going on 20 years now, and I can tell you I still have CD's that I burnt with WMV files back in the mid 90's that still play perfectly fine today. In fact, properly cared for a stored, none of my digital discs like CD's or DVD's have ever stopped working. I still have a CD collection I started from the early 90's at age 5 and all of them still play - though now I have digitized all that content into a hard drive for easier access.

 

File formats do not simply 'stop working', at least in general. Therefore, the whole saying 'you need to transcode to the greatest format every 5 years' is kidding themselves.

All the PNG, BMP, TIFF, and JPEG files I collected over the years still work fine, and always will in my opinion.

 

Do you want a pretty secure codec for your film master? Store the damn thing as a series of TIFF images. I'll be a flying pig the day computers loose compatibility with TIFF images. Yes, image sequences are sometimes larger than compression-based video codecs, but not by much - and those files are universal and can be loaded into pretty much any program you desire for editing that can load still images. Don't archive in a video codec format and you won't have to worry about trans coding.

 

The other side of the argument is that digital storage methods are unreliable in terms of quality and are prone to breaking down. That much is true, everything is prone to breakage and failure. A feel reel can be destroyed or it can break as well.

 

The final complaint I hear all the time is that 'it takes too much space to store a film in digital compared to a feel reel'.... Yeah, how about no. A single film in TIFF image sequence can be stored on two hard drives (one primary and one backup), which take up all the space of about a hardcover book - much smaller than the storage requirements for a feature length film reel. With a hard-drive based image sequence master, it is so small and inexpensive that you could, in theory create a 4 or 5 backups and store them at different locations for extra security. You know that ONE feel reel master you have stored? I hope the place it's stored doesn't catch on fire. If that happens with digital hard drive masters, I can simply pull one of the other master.

 

Hard drives are small and dirt cheap compared to film reels. A print costs what, $3,000 or so just for the print itself (not including the required processing)? $3,000 would buy me 60 hard drive backups. As long as you store those in a image-based format and in many different places where they are CARED FOR (just like a film print would need to be), you should be able to plug that hard drive in 50 years in the future and have it work just fine. If one doesn't work, go down the list of your 60 until you find one that does. Mechanical hard drives fail due to two reasons: mistreatment and wear. As long as it's cared for in a properly climate controlled environment and not used much after original burning, there is no reason that drive should ever stop working.

 

And yes, there is also the issue with compatibility of the drive itself, but this is not that big a deal as people want to make it out to be. I can go to micro center a buy a PATA and/or IDE board for around $20, plug that into my computer and still play any of my old hard drives sitting around my house from the 90's and before. Generally speaking, even WHEN hardware changes, it's still possible to buy old connection types.

 

That was a very long post, so in summary: Storage of digital is not that big of a deal. You don't need 200TB to store a single film. Hard drives are a lot cheaper and small than reels, and the reliable and compatibility factors so often screamed about are overblown.

 

 

In fact, I have a proposal for a perfectly viable storage means for digital film files for anytime in the future: A company specializing in digital-based film storage. There are already companies and archives that store film prints, so it's not a new idea. They need maybe THREE archive locations around the world that are climate controlled. Each film sent them is sent in as a TIFF image sequence on a single hard drive. that company then make a total of SIX backup clones of the hard drive on high quality drives ($100 each or $600 for the lot). They store TWO copies of each film at each of their locations in a warehouse fashion. Total space needed to store a film at each location is about the size of a hardcover book. They charge, say. $10 a year to store each film you send them. Over a 100 year period, this service has cost you $1,600 total for each film. Heck, let's say for the sake of argument we make a new master and copies every 25 years (just to keep up with modern technology in drives and such - but it is certainly not required), that adds another $1,200 to your total (assuming hard drives prices drop for an equal amount of space, which they have done). So now you're up to $2,800 for a 100 year period. Still cheaper than a single film print. Finally, the company itself would need to maintain a computer system capable of opening those files and drives at any point in the future, and have the ability to transcode at the studios request. This would also not be hard, and would mainly just require keeping a computer around from the same era as when the drives where made, or being careful and proactive in upgrades to ensure compatibility.

Edited June 6, 2016 by Landon D. Parks

[JD Hartman]

Longevity has nothing to do with file format, anything stored on magnetic media, floppy, magnetic tape, magnetic cartridge, disc drive platter, etc., will eventually lose it's magnetism and become unreadable. Fifty copies made all at the same time will all start to degrade as soon as they are created. The point at which they become unreadable, will be different, but that point will eventually be reached by all fifty copies. http://www.storagecraft.com/blog/data-storage-lifespan/

[Shawn Sagady]

Luckily before all those films stored digitally go bad we will be moving into new areas of digital storage including crystaline based binary storage. You will be able to store all your films in a completely uncompressed format byte for byte created in a crystaline matrix that will be nearly impossible to destroy and suffer no decay or data loss over geological time spans.

 

In the meantime its important to take great care with your storage methods, and always have backups. As for the copy of copies, that only really comes into play if you are re-compressing every time. A checksummed or CRC32 copy checking will ensure data integrity, and you'd fare much better working with raw data as apposed to compressed since a messed up byte in a raw file will change the color of a pixel vs risking serious damage in a compressed format. That of course requires more space but well get there.

 

Fun facts, storing a two hour 4K 24fps film in uncompressed RGB 16bit will take just over 9.1 terabytes of data. A 1080p film stored the same way is about 2.1 Terabytes so you could partition the film across multiple SSD drives which are more reliable than magnetic storage and good for 100+ years with careful storage and no use. Intel is actually about to release a 10 Terabyte SSD so you could store your feature films on that.

Edited June 7, 2016 by Shawn Sagady

[Shawn Sagady]

Little addendum, the SSDs would only last for that amount of time if powered occasionally or constantly. Actual cold storage a magnetic drive would be better. Proper optical drives would be the best but they do not exist in reasonable sizes. Also keep an eye out for sales. At one point BM reduced the cost of BMPCC by half to get it into more peoples hands.

Edited June 7, 2016 by Shawn Sagady

[JD Hartman]

This: http://physicsworld.com/cws/article/news/2013/jul/17/5d-superman-memory-crystal-heralds-unlimited-lifetime-data-storage

[Tyler Purcell]

There is one small misunderstanding here. I don't think anyone is referring to ONLY the final output. The problem ISN'T storing the 2 hour movie, that's a piece of cake. The problem is storing all the camera original data, which can be hundreds of terabytes per project. This is because, most productions are not finishing in 4k, even if they're shooting in 6k. It's FAR cheaper to finish in 2k or even 1080p. So the only way to really future-proof your digital product, is to store every single frame of camera original and prey in the future, there will be someone who can re-compile everything and bring the resolution up to the spec it was shot at. None of this is a problem in the film workflow, but it's a HUGE problem for a digital workflow, even if it was shot on film and finished digitally.

 

I work for some reasonably sized post houses and most of them are freaking out about this whole 4k nonsense. It takes longer to do everything, it costs more money to do everything and the infrastructure necessary has put some companies out of business. The one's who survived, are charging more money because they put so much into their infrastructure. Using higher compressed consumer based media formats like MPEG, are not solutions either. They are simply a waste of time for any professional shop as it's not up to the standards most clients want today. Most companies in the industry with finish in Pro Res XQ or some sort of tiff/jpeg based format, with an average file size of 25-40MB per FRAME (600MBps-960MBps)

 

CD's were a great medium, but not all disks were created equal. If they are stored perfectly, never touching anything, they should last 50 years or more. However, that's because the data is physically large on the disk, so the likelihood of error is much lower then DVD's for instance and burned BluRay's have to be treated very carefully. So there are digital formats out their which are pretty good, but nothing that is guaranteed to last over 50 years. Honestly, modern computing hasn't been around long enough to test these theories yet.

 

Spinning disks are an atrocious storage medium, they're very expensive per GB compared to LTO and their life span is around 5 - 7 years. Sure, we all have stories about 20 year old hard drives still running today, but most drives will fail. Either you keep them spinning to prevent bearing failure, which causes board failure or you keep them in a safe place not-spinning and the motor/bearing system fails. I've done A LOT of data recovery, there are petabytes of information being lost every year thanks to consumer storage systems that suck and people loosing/dropping/damaging their consumer devices.

 

SSD's are also an atrocious storage medium. They use multiple flash style storage IC's to create a small RAID ZERO internally, that's how they get the speed up. The problem is, if any one of those IC's goes bad, you loose everything. Also, if they aren't used on a regular basis, they can "forget" the directory block data. I'm very good friends with a designer of SSD's who works for Seagate and he's told me how hard they're working to fix some of those issues, but at this point, SSD's are really only good as constant use volumes. When SSD price per GB are in line with tape backup solutions, which may happen, we COULD see people building small SSD raid 5's and trying to make them work. Right now, they're not considered an option.

 

The great thing about a 35mm camera negative is how robust it is. You can submerge it in salt water and the image will still be fine. If stored in metal can's, you can subject it to quite a bit of heat, moisture (humidity) and beat the living snot out of the container physically and you've still got an image. Plus, film is permanent, it can't be erased, you can't misplace it (big/heavy reels are hard to loose) and most importantly, it's fixed-pixel resolution agnostic. So you can always get more from it, especially large format's like 8/35, 5/70 and 15/70, something impossible to do with today's digital medium's. Stick an original camera negative in metal boxes for 100 years and when the world has changed, you can always find some sunlight to shine through a frame and look at the past. People who think about preserving history, understand this... people who are just looking to make money, don't. This is what separates the archivists, from the people looking for easy/cheap solutions to storing their movies.

[Tyler Purcell]

One more little side note... digital storage will always be a failure without a culture surrounded by the technology. If something happened to our culture as we know it today, if the paradigm shifted somehow, the likelihood of loosing digital material is far higher then something that can be physically seen by the human eye or physical heard through a needle/stylus.

 

My comments aren't a film vs digital thing, in my eyes they are just common sense concerns. Does anyone honestly care about some stupid studio movie lasting 300 years? No... I sure don't. But what I DO care about is people understanding that digital technology hasn't been around for a long time and neither has this modern world we live in today. We are reaching a precipice, as no other civilization has survived to the level we have. It only takes a small incident to start unraveling everything and our history, what we are as a culture, will be the first things destroyed.

 

People may not care about this, but these are the kinds of discussions archivists have on a regular basis and it's important to understand. New "technology" doesn't solve the problem, it only complicates things even more.

[Shawn Sagady]

JD your link doesn't work :(

 

Tyler when you bring in the argument of all the footage recorded for the film etc its a much more compelling argument for sure. And resolution agnostic is great up to a point. But obviously scanning all your footage shot digitally out to 35mm is not going to be any more reasonable a solution, so its just something in favor of getting it in the can on celluloid the first time.

 

Very interesting discussion.

[Tyler Purcell]

They already make RGB separation prints for big movies, scanned out from digital to 35mm. That's the only REAL archive.

 

Honestly, I developed a machine years ago that stores one's and zero's onto film. The problem is, film fluctuates too much for digital information. So you couldn't fit ENOUGH to make it worth while. Theoretically, Kodak COULD produce an ultra-fine grain B&W stock that could be used for that purpose, but it kinda negates the whole point I'm making about being analog beings in a digital world.

 

Currently, we live in a world where everything we do needs "translation" of some kind. Which to me, is a real red flag and when the sky comes crumbling upon us, whoever inhabits this planet 500 years from now, will probably be looking at photochemical images wondering this current generation couldn't stick with the same long-term preservation format.

 

The sad truth is, it's not about money... it's about power. We are OWNED by our "machines" and technology.

 

It's an interesting subject for sure and if you study, you will learn that our culture is on the precipice of self destruction. So as someone who cares deeply for history, it's time NOW to figure out how to make things more technology/future proof then they currently are. Server farms and data centers, don't mean anything if there is no way to access them.

[Aapo Lettinen]

In terms of LTO as the industry standard for backup, it works great if you make two copies, stored in entirely different facilities, in completely different libraries and duplicate each tape at least once every 2 years. LTO tapes aren't expensive either but the formats are always changing. So the cost to keep migrating libraries from LTO 6, to LTO 7 to eventually LTO 12, is crazy. Libraries large enough to handle a catalog worth discussing, have a hefty foot print and are expensive to keep running. Unlike film which happily sits on a shelf, digital needs electricity and technological support to keep it working. Worst part is, if an LTO tape does fail, you can't splice it back together and magically get all the data like with film. One's and zero's don't work that way, though I will admit, data recovery from LTO is MUCH better then spinning disks.

 

You don't actually have to migrate every time when the LTO generation changes, you usually migrate over one generation (the system has backwards compatibility so a LTO7 drive can read LTO5 tapes and read+write LTO6 tapes) .

You may have to verify them every now and then but that is not labor intensive, it just takes lots of time to read all the tapes in the archive.

 

LTO is serpentine writing media, meaning that it has many tracks (depends on the standard how many in total and how many are used for read/write in per pass, "wrap") on the tape which means it writes for example 8 tracks in one direction in high speed, when the tape ends it write the next 8 tracks back to the other direction, and so on. this is to reduce seek times. for example with LTO5, it takes about 3 minutes for the drive to read or write about 830m of tape to one direction, then it changes direction and write back to other direction, making 80 passes in total if writing all the tracks. If you'd have a tape breakdown a specialist would probably still be able to restore most of the data because the tape addresses are in the catalog file and the servo bands are otherwise intact than at the breakdown point. If it's a large file you would lose lots of small fragments of it but most of the data could still be read probably.

 

with film you would also lose data, usually worth from one to three frames which of some could maybe be somewhat repairable with very careful splicing and intensive post processing of the scan

[Aapo Lettinen]

if the goal is to make a future proof image archival format which does not need any regular migrating work the only possibility would be to store actual human readable analog image on film, not any kind of digital data which needs translation or nifty equipment or codecs. always someone is stupid enough to lose everything we know about compression methods and formats and then they can't figure out how to read a file even if it was well documented in the beginning. You can make a format SOMEWHAT future proof but you can't make it idiot proof, thus it is always a good idea to make a archive format that is as easily readable as possible. one can make a film scanner by oneself relatively easily compare to for example a VTR so someone handy will always figure out how to get the image out in usable form from the film :lol:

[Aapo Lettinen]

btw is the any possibility to switch the auto correction off from user settings, it always fails with my posts and leads to more typos than what where in the original text :blink:

[Tyler Purcell]

Auto correction is in your computer, not the BB. :)

[Tyler Purcell]

with film you would also lose data, usually worth from one to three frames which of some could maybe be somewhat repairable with very careful splicing and intensive post processing of the scan

If you took a 12 inch chunk out of an LTO tape, it would be missing data on every track, thus it wouldn't be missing data from ONE file, but several different files. This is part of the reason I don't like LTO. Not saying that could ever happen, but the tapes are SUPER thin and it's still a mechanical system prone to magnetic field interference and mechanical failure.

 

With 35mm film, if you took 12 inches out, you'd still have a movie! Maybe a funny little cut, something you see a lot in classic movies where the camera negative has been lost or destroyed.

[JD Hartman]

JD your link doesn't work :(

 

Tyler when you bring in the argument of all the footage recorded for the film etc its a much more compelling argument for sure. And resolution agnostic is great up to a point. But obviously scanning all your footage shot digitally out to 35mm is not going to be any more reasonable a solution, so its just something in favor of getting it in the can on celluloid the first time.

 

Very interesting discussion.

 

Which one? I just tried both.

Here are some others. http://www.coolinfographics.com/blog/2012/8/14/the-lifespan-of-storage-media.html

http://www.livescience.com/53783-superman-memory-crystal-data-storage.html