Jump to content

Recommended Posts

I wanted to share a problem I recently encountered on a feature I shot in May/June of 2017. This website is a great resource to search for solutions to film specific problems and I feel my experience with G Drives, partition maps, formatting, and Red Undead can help others who may end up in the same situation. I will go in depth on what I learned and end with a simple tutorial on how to fix the problem.

 

Detailed Story

 

The production I was shooting decided to have post-production be a Windows OS based workflow. Per usual, I checked in with the post-team to figure out their OS and together advised the production on which types of hard drives to get. Remember, a FAT32 file system is largely unsuitable for video production because of the file size limitations. The production was opening formatting all of the hard drives FAT32 so they can work on either a Windows or Mac OS, but I advised against it for the above reason. I haven't had much experience with ExFat, which holds larger file sizes and is cross platform. We decided a conservative approach and felt formatting the hard drives for Windows was our best solution.

 

The film was low budget and slightly inexperienced, so we received our hard drives on the first day of production. We used G-Drive branded hard drives with 3TB each. Our camera system was the RED Epic with normal SSD magazines.

 

The first night after shooting we started to format hard drives and discovered that G-Drive is specifically designed for Mac. It can work on Windows, but G-Drive's workflow for formatting hard drives on and for a Windows environment is convoluted. Instead of having the hard drive in a FAT32 format that both systems can recognize for initial formatting, the drives came pre-configured for Mac. To format them, we had to use disk utility and command prompt (see instructions here: https://support.g-technology.com/KnowledgeBase/answer.aspx?ID=14080)

 

A key thing to note about this process: command prompt uses the clean command that erases the information that tells the computer the layout of the file structure of the hard drive. In a nutshell, every hard drive has a file that is a roadmap of all the data it stores; where each physical gate is, what row, how it's fragmented, etc. Windows and Mac have very specific structures, hence why Mac hard drives are largely unreadable by Windows and vice versa. [TANGET] Software exists that allows the OS's to read hard drives for opposing OS's [TANGENT] The clean command erases that file, but nothing else.

 

When you're in a Windows environment, a Mac formatted hard drive will not even be recognized by the OS. The only way a user can see it is through the Disk Utility program; at that point a user can only see that a hard drive is connected, but nothing else (because Windows sees nothing else either!). When you use clean in command prompt on the Mac hard drive, it creates the drive as an "unallocated" or unused hard drive. You can then create a new volume out of this unallocated hard drive and format it to a Windows structure. From there you have a hard drive that Windows can read/write to.

 

What if you accidentally use a clean command? What if that command was used on a Red SSD that has undownloaded footage? That happened to my production.

 

There is no way to undo a clean command; the partition file that describes to the OS what the structure is becomes completely unique once content is written to the hard drive because the partition file is updated to reflect where that new information is written. It's like a dynamic rosetta stone; without it, there's no way to decipher the data written. It is better described as a partition map.

 

With the partition map erased, the SSD had undownloaded footage with no way to find it. This became extremely stressful for me. However, I did find a solution.

 

Red Undead is a Mac based program developed by Red. It's designed for recovering data from accidental formatting of SSD's. Essentially, when an SSD is formatted in camera, the partition is re-written to display the structure and all of the data is made available for re-writing. But, none of that data is changed; physically the bits are still in the position they were last in (either a 1 or 0). Red Undead takes advantage of these "old states" and reads the data. From there, based on how Red's own R3D files are constructed, Red Undead can re-create the lost footage.

 

To recap, we had accidentally erased the partition map of a Red SSD. There's undownloaded footage sitting on the magazine, but there's no map available to tell the computer a) that a hard drive exsists and B) what and where each file is.

 

So, we decided to try Red Undead. We formatted the Red SSD in camera (to write the Red partition file) and then brought it into Red Undead. The program then spent an hour or so reading every gate, eventually spitting out nearly all of the lost footage. We only lost a portion of the first clip, which is a lot better than losing the entire card. NOTE: do not perform a secure format of the SSD in camera. A secure format rewrites the partition file and writes 0's across the entire SSD. A secure format truly erases footage.

 

There are a few things to learn from my experience:

  • Hard Drives must be delivered during prep. Be adamant about it as a DP and DIT.
  • Erasing a partition map is part of the formatting process; the damage is done.
  • An unsecured or simple format doesn't erase the already written data, but makes it available to be written over
  • A secured format erases the partition map and writes 0's across the entire card/hard-drive, fully erasing footage.
  • Red Undead reads old written data that hasn't been written over yet
  • Red Undead is only available for the Mac OS. (we should keep pestering them for a Windows port)
  • The obvious one: DO NOT FORMAT HARD DRIVES WITH CAMERA MEDIA CONNECTED AT THE SAME TIME

Of course, this technology may change over the years, but the concepts should hold.

 

Tutorial: How to fix a Red SSD Mag that has its partition map erased

  1. If the partition map has been erased, do nothing else to the SSD mag if it's still connected to a computer.
  2. Insert the SSD mag and do an unsecure format in camera.
  3. Reconnect the SSD mag to a Mac computer.
  4. Run Red Undead in the computer.
  5. Verify which clips were saved and which were not.
  6. Ensure recovered clips are backed up before formatting the card again.

---

 

I hope my experience can help others! I'm in no way an engineer (software or hardware), so please correct me if my terminology or concepts are wrong.

 

Has anyone else had experiences similar to this? Does ARRI, Canon, or Panasonic offer data recovery software like Red Undead?

  • Upvote 1
Link to comment
Share on other sites

  • 7 months later...
  • Premium Member

just recovering an old thread... I just wanted to add that there IS a way to use other file systems than FAT/FAT32 or exFAT when working at the same time with Windows + Mac based workflow.

 

You need a separate commercial plugin for that, for example the Paragon's HFS+ for Windows software. I personally edit with couple of desktop macs and a windows laptop with Paragon installed on the windows laptop and can read and write the Mac formatted edit drives natively on the Windows machine, edit the Premiere projects located on in, even use Shotput Pro on the Windows machine to backup to the HFS+ drive (though I'm not sure how safe it is for the data to use a translating software between the operating system and the backup drive, will test it later) . the Paragon makes the Window Explorer to see the hfs drive natively, there is no user interface or anything in the plugin.

 

This type of plugins are worth a try if you want to use multi operating system workflows. that "HFS+ for Windows" cost something like 16 euros and paid itself back in an hour or so ;)

Link to comment
Share on other sites

  • 2 weeks later...

That's a great point. I personally own Paragon's software as well and love it.

 

However, it's a software acting as a middle man translating between two different file structures. Not a huge problem for editing, but offloading footage from cards becomes a problem in regards to speed. The software creates a bottleneck if one is downloading footage onto a Mac formatted drive through a windows OS. (I've experienced that for 5 years now on my Windows laptop trying to offload footage through USB3 at slow speeds)

Link to comment
Share on other sites

  • Premium Member

That's a great point. I personally own Paragon's software as well and love it.

 

However, it's a software acting as a middle man translating between two different file structures. Not a huge problem for editing, but offloading footage from cards becomes a problem in regards to speed. The software creates a bottleneck if one is downloading footage onto a Mac formatted drive through a windows OS. (I've experienced that for 5 years now on my Windows laptop trying to offload footage through USB3 at slow speeds)

that's true that it can create bottlenecks when using translating software...

if having a multi platform/filesystem workflow one should use the operating system's native format for the offloading drives and could use the other filesystem with the translating softw for the edit proxy drives. For example, on mac one would use hfs+ drives for all the offloads and use ntfs for the edit drive.

It would also not be a huge problem if having only one of the BU drives in different filesystem as long as the offloading is fast enough

Link to comment
Share on other sites

that's true that it can create bottlenecks when using translating software...

if having a multi platform/filesystem workflow one should use the operating system's native format for the offloading drives and could use the other filesystem with the translating softw for the edit proxy drives. For example, on mac one would use hfs+ drives for all the offloads and use ntfs for the edit drive.

It would also not be a huge problem if having only one of the BU drives in different filesystem as long as the offloading is fast enough

 

I completely agree!

 

The original problem was the G-Drive brand of hard drives were pre-formatted as HFS+ and not FAT32. Because they're HFS+, a Windows OS (without any software that we've mentioned) won't be able to recognize the hard drive which means the user will have to go through a complicated process to format an HFS+ hard drive for Windows. If the hard drive was formatted as FAT32, then either OS could recognize it and then could format the hard drive to their specific OS file structure.

 

But, this is all great info for any google searcher trying to find a solution to their problem. :)

Link to comment
Share on other sites

  • Premium Member

 

I completely agree!

 

The original problem was the G-Drive brand of hard drives were pre-formatted as HFS+ and not FAT32. Because they're HFS+, a Windows OS (without any software that we've mentioned) won't be able to recognize the hard drive which means the user will have to go through a complicated process to format an HFS+ hard drive for Windows. If the hard drive was formatted as FAT32, then either OS could recognize it and then could format the hard drive to their specific OS file structure.

 

But, this is all great info for any google searcher trying to find a solution to their problem. :)

 

I use Paragon Hard Disk Manager on Windows for that, it can also format drives to hfs+ and ext file systems on windows if needed. it is slow to boot up etc. and has its quirks but it has been quite handy so was worth buying anyway.

 

if you have the HFS+forWindows plugin you can also format them because the windows then recognizes the hfs drive correctly.

 

(not sponsored by Paragon Software :lol: )

Edited by aapo lettinen
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...