Jump to content

Data Wrangling with a MacBook Pro... eSATA via Thunderbolt?


Recommended Posts

Hi,

 

I'm primarily an Editor dabbling in motion graphics. I'm wanting more on-set work and have been offered the opportunity to do some AC and data wrangling on a local short in Leeds.

 

I'll be using a MacBook Pro 15" (late 2011) 2.5GHz Intel Core i7, 16 GB RAM.

Shooting RED scarlet and backing up on two G-DRIVE's - Yet to have confrimation whether or not the drives have eSATA.

 

As my mac has one FireWire 800 port and one Thunderbolt, I'll be running into some problems on the way in working out how fast I can transfer data on the budget I have.

 

I've heard and read of the problems of trying to daisy chain two FW800 drives and a RED station, all through one little FW port. I've also read a lot of positive things about the Sonnet Echo Pro Thunderbolt adapter. http://www.sonnettech.com/product/echoexpresscard34thunderbolt.html

 

I'm hoping to get the eSATA extension, but if the G-DRIVE's are not compatible the FW800 will do for now.

 

For £250 I'm wondering if this is the best option I have for transferring data as quickly as possible, while linking two G-DRIVE's through one eSATA and using the second eSATA for the RED station?

 

 

Thank you.

 

 

 

Luke Wheldon.

 

 

Link to comment
Share on other sites

I've just read that eSATA cannot be daisy chained. Duh.

 

How well would the set up work if the eSATA drives each had they're own port and then I used the FW800 for the RED station? Or one eSATA for one G-DRIVE and the other for the RED station... then the FW800 for the second G-DRIVE?

 

Thanks.

Link to comment
Share on other sites

  • Premium Member

I've tried to use Red cameras twice and on both occasions had problems getting the red card reader to work via anything other than USB. Which is part of why we avoid red. But I digress.

 

My approach is to try and keep things on private buses as much as possible, and I'd prefer to have symmetrical performance from the hard disks. You will need some sort of software to do a double copy and checksum compare for real reliability.

 

Ideally you want to own an LTO drive if you're going to be doing any serious amount of this. Hard disks fail way, way too easily. And a macbook is not really the ideal tool. Laptop processors are comparatively rather underpowered, and by the time you've got this stuff all patched up it can become a reliability nightmare with all the cables and connectors. I like to have everything racked up in a server case and ready to go.

 

P

Link to comment
Share on other sites

Hi Phil,

 

Thanks for your post. For double copy and checksum I have been advised to use ShotPut Pro for now. At some point in the future the idea is to be doing a lot more of these jobs and to expand into DIT work. As I am very new to the role I have to work with what budget I have and the drives I'm provided with. The job I mentioned is unpaid, which is fine for now as I am still learning and would like to build up some relationships - I can't go around giving myself a new job title until I'm a lot more confident and experienced under the role.

 

I've been doing a lot of reading on the role and it seems that many people have different ways of working, mainly due to budget and production. Just out of interest what sort of set up do you have for data wrangling?

Link to comment
Share on other sites

Okay, first up be careful with the firewire stuff. NEVER hotplug firewire, it's easy to burn out your firewire ports that way as they aren't well designed in that respect.

 

Secondly I agree with Phil that it's best to keep things on separate buses as much as possible. So if for instance you can only get the red card reader to work from USB then just put it on the USB bus and make sure nothing else is on that USB bus.

 

If you can get eSATA working then it would be great to have the RED station on eSATA so that you can make the most of the really fast pipeline. The Red SSD's should be able to pump out data very quickly. If the hard drives have the capability to work with eSATA then that might be good but there is something to be said with having them on the firewire 800 bus anyway as they are slower devices and they will be on a separate bus that way.

 

You might want to experiment a bit and see what works fastest based on those principles.

 

I assume your laptop doesn't have a cardbus slot? Just thinking that might make getting eSATA a bit cheaper.

 

Freya

Edited by Freya Black
Link to comment
Share on other sites

Great! Thanks, Freya. Unfortunately I don't have the expresscard slot on my 15" so I've just ordered the Sonnet with eSATA.

This all may seem very basic to most but it's essential info for me right now.

I'm really appreciating the time and advice from everyone.

 

Luke.

Link to comment
Share on other sites

You need to assess the weakest link in the data flow. The G-Drives and FW are the definite ones in this case.

 

G-Drives are safe for storage but horribly slow for transfer, they peak at 100 MB/s on a good day. FW peaks at 80 MB/s. Add to this the checksum, which doubles your time, and you're looking at 40 MB/s total in the very best of all possible scenarios.

It looks like the wrong setup, to be honest. You need to ditch the G-Drives for G-Raids, and go either USB3 and/or eSata. Those drives are so slow they won't be able to actually play the footage.

It will all, however, work as you have planned it, but it'll be dreaduly slow. Make sure you get plenty of redmags to keep the camera rolling and arm yourself with patience.

How many USB3 does your laptop provide? Belkin has a nifty little thunderbolt hub that provides 3 USB3 and another thunderbolt. It's the cheapest option around, and with TBolt being a direct connection to the PCI Bus you won't have the slightest problem, bottleneck or conflict with it. Patch an esata through the belkin via tbolt and sonnet's expresscard reader. This way the weakest link shall be your drives, with esata2 (what the redmag reader has) being the second slowest at 3 gigabits, and usb3 the third slowest at 5 gigabits.

It's what happens when producers buy the drives!

  • Upvote 1
Link to comment
Share on other sites

use ShotPut Pro for back up and check-sum verification. I have managed over 8000 hours of footage for two seasons of a TV show and the software rocked. We used G-Speed Qs in RAID 5 hooked up to Macbook Pro 15" via an eSATA to USB3 converter. USB 3 is much cheaper and there are more devices made with it rather than Thunderbolt. Remember that the drives to which you are backing up to are the bottleneck. You can have a 10,000 RPM drive hooked up to a computer, both by Thunderbolt and then again by USB 3 and you won't see a difference in speed. Not until you use SSD as your storage medium will you see a speed boost. FW800 is only going to be marginally slower than either USB 3 or thunderbolt with spinning drives which are not striped.

 

With the set up I used and using a San Disk Imagemate USB 3 card reader (they are the fastest, faster than Lexar) I could back up a 32gig CF card with verification in 8 minutes. If I were you I would look into a RAID 5 or RAID 10 solution, going to two non striped drives is a little scary.

  • Upvote 1
Link to comment
Share on other sites

  • Premium Member

I've long felt that Raid5 is a poor choice for... well, a lot of things, with the only real exception being where you want the absolute minimum cost per megabyte and don't care too much about speed, which is not your situation. Raid10 is cheaper if you want it to be even slightly fast, less likely to fail, recovers more quickly if it does fail, and remains more usable while degraded.

 

I wrote a lengthy article on why I think this is the case: http://www.redsharknews.com/technology/item/877-raid-everything-you-need-to-know

Link to comment
Share on other sites

I've long felt that Raid5 is a poor choice for... well, a lot of things, with the only real exception being where you want the absolute minimum cost per megabyte and don't care too much about speed, which is not your situation. Raid10 is cheaper if you want it to be even slightly fast, less likely to fail, recovers more quickly if it does fail, and remains more usable while degraded.

 

I wrote a lengthy article on why I think this is the case: http://www.redsharknews.com/technology/item/877-raid-everything-you-need-to-know

 

Thanks Phil. Just going to get stuck into your article now!

Link to comment
Share on other sites

  • Premium Member
Your only real solution is a Thunderbolt PCI expansion chassis

 

Or, you know, a real computer.

 

It is a significant source of frustration to me that a large proportion of people trying to do real, serious data handling work leap instantly and inexplicably to the conclusion that a laptop is the way to go. And that laptop has to be made by Apple.

 

An Apple laptop is just about the single most expensive way to buy computer capability. I have no bone to pick with Apple in particular, but they're not particularly cost-effective, and they no longer really have a properly expandable solution for heavy lifting. Laptops are expensive no matter who you get them from, as well as difficult to expand (as this thread demonstrates). Compound these factors and you're making a genuinely crazy decision to base a DIT cart around a macbook.

 

Get a 4U rack case with a lot of drive bays and load it up. It doesn't have to be expensive to begin with and you can update it as your needs grow. You can drop it in an off the shelf flight case for transport, too.

 

P

Link to comment
Share on other sites

 

Or, you know, a real computer.

 

It is a significant source of frustration to me that a large proportion of people trying to do real, serious data handling work leap instantly and inexplicably to the conclusion that a laptop is the way to go. And that laptop has to be made by Apple.

 

An Apple laptop is just about the single most expensive way to buy computer capability. I have no bone to pick with Apple in particular, but they're not particularly cost-effective, and they no longer really have a properly expandable solution for heavy lifting. Laptops are expensive no matter who you get them from, as well as difficult to expand (as this thread demonstrates). Compound these factors and you're making a genuinely crazy decision to base a DIT cart around a macbook.

 

Get a 4U rack case with a lot of drive bays and load it up. It doesn't have to be expensive to begin with and you can update it as your needs grow. You can drop it in an off the shelf flight case for transport, too.

 

P

 

Phil, I do this for a living, and trust me, TBOLT expansion chassis works perfectly fine. I use it on commercials and long form. Even for RED transcodes as long as you've the latest Retina and a Redrocket. The Belkin Hub is spreading like wildfire at the moment. Jigsaw 24 simply doesn't have enough stock to keep up with the demand.

 

The only situations in which I'd recommend bringing the big one are these:

 

- Multicamera transcodes. In 99% cases best go big one, however, I've gone retina MBP with up to three Alexas at prores 2k 4444 and it was absolutely fine. The intermediate drive was an 8 bay RAID 6, and I had my assistant operating it whilst I attended the three camera crews. The name of the game was "feed the monster". We managed to ingest, verify and transcode with a one light grade close to 1 tb of 2K prores 4444 per day. Staying only 30 minutes after wrap. We were real proud of this feat.

 

- Anything RAW, with the exception of a single Red camera, in which case red rocket on expansion chassis goes just fine.

 

What I've discovered is that the number of cores does matter greatly in the retina. The moment you have 4 verified copy processes in the background plus one transcode the GPU stops maxing its FPS due to the processor not being able to feed the GPU.

 

The retina is a real beast. And the moment apple finally (as in they'll never do it) decides to release thunderbolt GPU drivers, even single camera RAW will be doable with an external chassis.

 

But for data wrangling... even an i5 Macbook Air with the right expansion chassis and cards, or hub, is overkill.

 

And yes, I'd rather have a Linux or Windows going on in a bespoke computer. But ShotPutPro, the only pro verified copy appication in Windows is slow slow slow indeed. Silverstack beats it to a pulp every single time. And the list of OSX only DIT apps is larger, way way larger then the alternatives. Not what I personally like, but it's the status quo. :(

 

My humble opinions.

Edited by John Miguel King
Link to comment
Share on other sites

  • Premium Member

John, I shall defer to your more recent experience - last time I did this for money, the 5D mark 2 was new and we were barely on LTO4, although I've done a lot of related stuff in the meantime and always feel underdressed without properly racked up gear. Laptops become a hydra of flimsy cabling, in my view. Anyway.

 

I'm a bit concerned about ShotPut. Back when I started doing this stuff - more or less around the time anyone started doing this stuff - I just wrote my own scripts to do double copies and MD5 sums, and the like. I had thought of writing something more formalised to do copies and transcodes (via ffmpeg) but assumed ShotPut had the market sewn up. I've never been an Apple guy. Might this be worth revisiting if it's flawed?

 

P

Link to comment
Share on other sites

No problem Phil. We're seeing paradigm shifts on this area at a rate of once a year, at least! I´m forced to keep up cos I like to pay my rent on time.

ShotPutPro has been replaced by Pomfort Silverstack. MD5 is now legacy, with MHL being the algorithm of choice. MHL signs and seals the copy, which I can then use to tell my clients "what's on that folder is exactly what was on the card". Silverstack also has advanced metadata capabilities, useful for naming rules, logging, transfer to ALE, quality control... It's a Data Wrangler software solution in one single place.

I agree, cables are abundant on a laptop solution. However, because Thunderbolt is the PCI bus, you can end up with only two cables, then take these to your rack of choice, and expand from there. I prefer laptops because they clutter my cart less, allowing a big 24" Grade monitor and a waveform to sit comfortable, so I can actually do real DIT work. Also, up until the new Mac Pro, the old Mac Pro was a bit of joke. Huge but underpowered.

Sadly, very sadly, software on Windows is scarce and the laptop hardware lacks crucial expansion slots. I've been building my computers since my teens, and it enrages me not being able to keep doing it now. A hackintosh just wouldn't look right... and there's bound to be conflicts with every OSX upgrade :(

Link to comment
Share on other sites

I had thought of writing something more formalised to do copies and transcodes (via ffmpeg) but assumed ShotPut had the market sewn up. I've never been an Apple guy. Might this be worth revisiting if it's flawed?

 

P

Shot Put Pro does the job. Not the best, but it works.

 

And Davinci Resolve Lite is awesome, and free, for anything short form, so no, unless you're going to give it a grading interface... :D

 

For everything else, there's Scratch, which is just perfect. I'd bring it to every job, if only clients understood the extra cost.

 

Although, a little app that could automate it all, and was cross platform (no need for grading, or colour correction) would definitely help a lot of people out there.

Edited by John Miguel King
Link to comment
Share on other sites

  • Premium Member
And Davinci Resolve Lite is awesome, and free, for anything short form, so no, unless you're going to give it a grading interface

 

Eugh, I looked at Resolve once and decided it was a nightmare. Not bad as a grading app, just the workflow...

 

I'm thinking more of something that would do automated transcodes - with cubes - and burn-ins and the like. I keep needing that even now, but I hate to put the amount of time into engineering something just for my own use. Edit: and it wouldn't be four hundred quid. Jesus Q. Christ. Maybe you work on big American action movies all the time...

 

P

 

Edit - oh, and, doesn't MHL actually use MD5 as the per-file hashing algorithm? It's just an XML file with the folder contents. This sort of thing worries me as it then requires you to keep the files and XML in sync, which can break accidentally and give rise to false positives. I always used to avoid this sort of separate-file metadata.

Link to comment
Share on other sites

HAHAHAHA I wish.

Well, big budgets are now being shot in London by default. We're all talking of a wave of nonstop work for the next ten years! It's going completely insane like nobody's ever seen it before. So the cost is not really an issue when you pass it on to the client. If it gets the job done, and it's fast, it's a no brainer.

I think you had a bad experience with Resolve. I'm reading your post and there are a couple of misunderstandings there. I'll post them there if you want. I'm off today after a very long day on set yesterday.

As to the app, there are a few doing that. Mind you, something simple, free and or cheap, that transcodes to the usual offline flavours could be helpful for those on no budget. But then, there are options out there for those that research. A simple rsync command line will get you a good copy, then resolve for offline.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...