Jump to content

DIT / Data - Trying to speed up offloads


Recommended Posts

Hi, I've been asked to help out as a DIT for an upcoming student short film, although I'll be mostly just offloading and transcoding files I've tried to investigate since this will be my second time doing it and I'd love to speed things up a bit from last time. I'm given a macbook pro (2017/2018) to work with but have recently gotten access to a Stardom DR5-WBS3 which I think may help.

 

Here's the two main issues I'm running into:

- No dedicated DIT software, I'm offloading with Davinci's Clone Tool which keeps me from doing anything during offloads. I'm trying to fix this by getting Silverstack installed on the machine.

- I back up everything to lacie rugged disks, they run quite slow and average 70-80MB/s transfer speeds, I need to backup 2-3 drives per card and tend to always make backups from the card which makes freeing them take much longer.

My questions are:

- will building something like a RAID 0 or RAID10 on the Stardom connected through a USB3.0 protocol be faster than the lacie rugged (also connected through USB3.0 protocol)?

- Is cascade copying a good option to free card faster? and, in a worst case scenario, is it possible to do without Silverstack?

 

I'm still unsure if the Stardom DR5-WBS3 is going to be that much of an upgrade, but I believe Silverstack might be. Any other recommendations would be appreciated since I have very little experience in the field but would love to make my job and the camera department's job as easy as possible!

Link to comment
Share on other sites

  • Premium Member

Your biggest bottleneck tends to be the backup drives which are much slower than the camera card. A slow computer may cause additional delays when using verified transfers like on silverstack.  But generally it is much faster to make at least one copy to raid or ssd very fast and then just manage other fast copy so that you dare to reuse the card. If you are shooting normal amounts of material then the 1 raid configuration should be no problem. Normal means something from 400 to 700GB per day. If it goes over about 1TB per day then I like to use dual raids which are 4 or 6 drive thunderbolt configurations. This way you can do one copy from card to raid at full speed the card can read which may be for example 400 or 500mb per second, and then just do a second copy from raid to raid with speeds of over 1 gb per second, even unverified if you need to get the card back very fast. If not in a hurry, you can so second copy to a normal drive and the third when you have time. It is very important that the first copy is verified and generates the mhl files with the material so that you can save manual work when further backing up the material. Remember to quickly manually check the material too, silverstack can rarely mess up a transfer and claim that everything was ok. The corrupted files will have .pfncopy file extensions so you should be able to spot them easily on finder

Link to comment
Share on other sites

I would keep it simple .. I use a Mac book pro 2017 with Hedge software ,down loading to 3 x Lacie drives .. down load to 3 drives at the same time from the one card.. as long as you start off with decent battery power you can afford to use up all the 4 USB C ports ..about 40 - 50 mins to all 3 from about 200GB card .. 4K  24p..

Hedge is very good and easy to use.. it has pre sets for folder naming etc.. and alot cheaper than Silver stack ,which has a ton of features you will probably never use ..

  • Upvote 1
Link to comment
Share on other sites

  • 2 weeks later...

Hey professional DIT here

Don't cascade copy, that makes multiple backups pointless on one level (sure, you're protected if you physically lose one, but if there's a bad copy on your first generation, that gets copied to all other drives). I know that Hedge etc will verify that one copy, but it's not worth taking the risk with any footage, IMO 

The quickest solution would be:

• throw any Lacie Ruggeds in the rubbish bin. 

That's an extreme statement and I appreciate that you might not have much in the way of budget. But the bottleneck of those drives is MASSIVE. 

Get G-Tech USB-C drives. You need drives with a power supply, because Ruggeds or similar run power and data through the same cable. Massively slow. 

The even quicker version (for transcoding) would be to buy 2x G-Techs for your backups, and have 1x SSD scratch drive (say a 1TB drive). 

Back up to all 3, then run your transcodes from and to the SSD. 

Agree with Robin – Hedge is excellent. Would definitely recommend it over Shotput. Silverstack is amazing but even I'm not sure if I can afford it right now (and it's a steep learning curve)

Hope that helps somewhat. Shout if you need more!

Link to comment
Share on other sites

@Davo McConville Thanks a lot for the advice, I'm currently halfway through the project and have been cascade copying onto Lacie drives with Silverstack since I had no other choice to adapt to the production's budget and try to keep offloads from slowing it down. I've been made aware of the first copy's potential to corrupt the rest of drives by other DITs but unfortunately any faster alternative was not possible and I was told my only alternative was to trust the verifications at that point.

I never heard of Hedge but Silverstack has been my go to whenever I've done any offloads/transcodes because of the ease of use once set up and the daily reports I can generate with it but I'm curious to try it out sometime!

Link to comment
Share on other sites

3 minutes ago, Kian Rahnema said:

@Davo McConville Thanks a lot for the advice, I'm currently halfway through the project and have been cascade copying onto Lacie drives with Silverstack since I had no other choice to adapt to the production's budget and try to keep offloads from slowing it down. I've been made aware of the first copy's potential to corrupt the rest of drives by other DITs but unfortunately any faster alternative was not possible and I was told my only alternative was to trust the verifications at that point.

I never heard of Hedge but Silverstack has been my go to whenever I've done any offloads/transcodes because of the ease of use once set up and the daily reports I can generate with it but I'm curious to try it out sometime!

Ah I'm so sorry, I didn't see the date on your post. Thought you were still pre-production!

Sorry I couldn't say anything sooner. Lacies are the main bottleneck so for next time, you know!

Silverstack – you finding it good? I've not bitten the $900 bullet yet, did you get a short term project license?

 

Link to comment
Share on other sites

1 minute ago, Davo McConville said:

Ah I'm so sorry, I didn't see the date on your post. Thought you were still pre-production!

Sorry I couldn't say anything sooner. Lacies are the main bottleneck so for next time, you know!

Silverstack – you finding it good? I've not bitten the $900 bullet yet, did you get a short term project license?

 

It was still helpful info to consider for future projects!

AFAIK the filmschool had the DIT station's Silverstack licensed and that's what I was given, I couldn't afford to license it myself that's for sure ?

Link to comment
Share on other sites

Just now, Davo McConville said:

Ah cool – yes Silverstack is a $$$$$$$$$ beast

Message me if you ever need any other help! I don't look at the forums too much but would see a message

It definitely is ahaha

Will do, thanks!

Link to comment
Share on other sites

6 hours ago, Davo McConville said:

Ah cool – yes Silverstack is a $$$$$$$$$ beast

Message me if you ever need any other help! I don't look at the forums too much but would see a message

Thanks for the knowledge sir .. yes unfortunately the BUS powered HDD,s are slower but Im often  in a hotel room downloading and sometimes they don't have many power points .. like 2 at opposite ends of the room ! ..so not having to power the drives is easier if slower .. I have a shoot next week where they said their insurance policy actually stipulated Lacie rugged ..!  because they need to be shipped .. which of the BUS powered drives do you think are the fastest .. or all about the same due to the power not being separate ..  occasionally production have sprung for SDD but I didn't find them that much faster ..?

Link to comment
Share on other sites

5 hours ago, Robin R Probyn said:

Thanks for the knowledge sir .. yes unfortunately the BUS powered HDD,s are slower but Im often  in a hotel room downloading and sometimes they don't have many power points .. like 2 at opposite ends of the room ! ..so not having to power the drives is easier if slower .. I have a shoot next week where they said their insurance policy actually stipulated Lacie rugged ..!  because they need to be shipped .. which of the BUS powered drives do you think are the fastest .. or all about the same due to the power not being separate ..  occasionally production have sprung for SDD but I didn't find them that much faster ..?

Ah then a power stop is your friend! DM me and I’ll send a photo of an inexpensive little setup. 
All bus drives will be about the same, sadly

SSD - should be a lot faster, especially on USBc and even better thunderbolt 3!

Link to comment
Share on other sites

1 hour ago, Davo McConville said:

Ah then a power stop is your friend! DM me and I’ll send a photo of an inexpensive little setup. 
All bus drives will be about the same, sadly

SSD - should be a lot faster, especially on USBc and even better thunderbolt 3!

Yes true , but TBH just having BUS power is simplest and easiest in a hotel room in south east Asia .. .. although longer down ,loads .. I have USB C Mac Book Pro .. not sure why it wasn't faster .. ,maybe the Sony SxS card reader then was the block.. the cards were new ..  I have XQD / fx9 now should try it out on an SSD .. trouble is production will seldom spring for 3 X large capacity SSD,s.. until they get even cheaper I guess.. they love those Rugged drives .. even as far as demanding them for their insurance.. some ones on the payroll for sure .. !

Link to comment
Share on other sites

  • Premium Member

I am always slightly alarmed by discussions of computer performance which describe a computer as a "macbook pro (2017/2018)". That's not a specification that actually helps anyone understand what the underlying computer actually is. If one were to buy a 16" MacBook Pro right now, one could choose a Core i7 or Core i9 processor, 32 or 64GB of RAM, Radeon Pro 5500M or 5600M with four or eight gigabytes of graphics memory, and anywhere from one to eight terabytes of storage. I know that's propellerhead stuff, I know it's not cool, I know it's not something Apple encourages anyone to care about, but depending on the task at hand, these differences will have a significant impact on performance.

This is why full-time propellerheads like me tend to sigh and facepalm every time someone says they've got a "sixteen-inch MacBook Pro!"

Lots of DITs, who are camera people first and computer people fifth, talk like this. It is depressing, because things could be so much better.

There is no connection between a drive being bus-powered and its performance, beyond the blindingly obvious issues of not purchasing 3.5" spinning disks and expecting a Thunderbolt port to power them. I mean, that is blindingly obvious to everyone, right?

  • Upvote 1
Link to comment
Share on other sites

21 minutes ago, Phil Rhodes said:

I am always slightly alarmed by discussions of computer performance which describe a computer as a "macbook pro (2017/2018)". That's not a specification that actually helps anyone understand what the underlying computer actually is. If one were to buy a 16" MacBook Pro right now, one could choose a Core i7 or Core i9 processor, 32 or 64GB of RAM, Radeon Pro 5500M or 5600M with four or eight gigabytes of graphics memory, and anywhere from one to eight terabytes of storage. I know that's propellerhead stuff, I know it's not cool, I know it's not something Apple encourages anyone to care about, but depending on the task at hand, these differences will have a significant impact on performance.

This is why full-time propellerheads like me tend to sigh and facepalm every time someone says they've got a "sixteen-inch MacBook Pro!"

Lots of DITs, who are camera people first and computer people fifth, talk like this. It is depressing, because things could be so much better.

There is no connection between a drive being bus-powered and its performance, beyond the blindingly obvious issues of not purchasing 3.5" spinning disks and expecting a Thunderbolt port to power them. I mean, that is blindingly obvious to everyone, right?

Hi phil, with respect the computer’s year and therefore generation is enough for the discussion.  Processor and ram aren’t in play here in this discussion: types of ports are. 
nothing is blindingly obvious, we’re trying to help someone who may not be a propellorhead. Bus powered drives *do* have a major performance cap compared to their psu-powered counterparts. Compare a LaCie 7200rpm drive inside a rugged and inside a psu drive and you’ll find a huge performance difference 

Link to comment
Share on other sites

  • Premium Member
4 minutes ago, Davo McConville said:

Compare a LaCie 7200rpm drive inside a rugged and inside a psu drive and you’ll find a huge performance difference 

That's fine but which 7200rpm drive? 

With how much cache? Designed for what market? Lifetime? Guarantees?

"A PSU drive" is not a specification.

I'm not sticking on this to be a pain, I'm sticking on this because this stuff matters and it's not enough to wave our hands at it and assume certain manufacturers are just magic.

The way a device is powered does not (necessarily) affect its performance and it is misleading to suggest otherwise.

Link to comment
Share on other sites

Im certainly no propeller head , Im just happy if the thing starts up, especially after installing Big Sur.. blank screens abound .. but Ive always heard that an AC powered HDD will be faster down load than a BUS powered one.. although for me the BUS power convenience is greater than the speed .. Im not a DIT I just have to down load 1 or 2 cards at the end of the day with Hedge to check and generate a report .. 

Link to comment
Share on other sites

  • Premium Member
9 hours ago, Davo McConville said:

If you can show me a bus powered drive that offers speeds close to a wall-powered drive I would be very happy to know about it!

 

More to the point, I can very easily show you a lot of "wall-powered drives" that are very slow.

The DIT world in general has got to stop relying on these sloppy assumptions and actually use specifications. The fact that a laptop is made by Apple does not make it fast. The fact that a hard disk plugs into the wall does not make it fast.

We express storage speed in megabytes per second, not number of plugs. Yes, I know that's boring and uncool and forces people to spend actual time learning actual facts but jesus.

If I have to spend another hour waiting around while someone uses an Apple laptop to transcode something slower than my phone could do it, on the assumption Apple is always the best thing around when the only thing they know about it is its year of manufacture and screen size I am going to set that person's head in concrete and throw him in the river! Learn your trade!

9 hours ago, Robin R Probyn said:

Ive always heard that an AC powered HDD will be faster down load than a BUS powered one.. 

It's true to an extent in certain situations. It largely comes from the days where everything was spinning disks, and those disks tended to spin at either 5400 or 7200rpm. Naturally, faster is better, but it draws more power, so you can't, generally, run a 7200rpm 3.5" hard disk from a USB (or, now, thunderbolt) port. These days you possibly can as things have improved but it's often not done.

Very often now you'll be using external flash devices anyway and it becomes completely irrelevant, but companies like G-Tech have some devices based around 2.5" laptop-sized disks which are generally very acceptable. If you want the absolute zenith of speed from spinning rust, it may still demand an external power supply, but people make these claims about flash devices and tiny laptop drives and... the mind boggles.

P

Link to comment
Share on other sites

Your attitude is incredibly rude here phil. You make sweeping generalisations and assumptions that are not justifiable or helpful. How dare you tell a fellow user to “learn your trade”. 

your own post contains a number of inaccuracies or unhelpful notes so you’re not exactly showing how it’s done. 

Link to comment
Share on other sites

  • Premium Member
9 minutes ago, Davo McConville said:

Your attitude is incredibly rude here phil. You make sweeping generalisations and assumptions that are not justifiable or helpful. How dare you tell a fellow user to “learn your trade”. 

your own post contains a number of inaccuracies or unhelpful notes so you’re not exactly showing how it’s done. 

Sorry, you're feeling the sharp edge of my tongue after a week of negative experiences around all this stuff. I'm aiming this at the not-insignificant subset of DITs who have their eye on a DoP position, don't really take the job as anything more than a stepping stone into the camera department and really could do with a bit more book learning, and I think that's a legitimate concern. I'm sure we'd all rather spend less money, go home sooner and have fewer problems.

If there's anything technically wrong with what I'm saying then I'd be very pleased to hear about it.

  • Upvote 1
Link to comment
Share on other sites

If you have something to say about the specifics of which drive to use, contribute to the conversation that way. 
Your lazy slandering in lieu of helpful constructive advice speaks volumes about you. 
I’m not even going to get into the specifics of any of it, but you freely admit your own prejudices are showing. 
You know nothing about my or the others’ qualifications or backgrounds. No place for this on this forum. 

Link to comment
Share on other sites

3 minutes ago, Mark Dunn said:

Phil Rhodes

  • sustaining.png.f580000f0737f1b5b3dca64ff
  • 13043 posts
  •  

Davo McConville

  • 53 posts

Just a thought.

Ha!

?

Well, I bow down to the great professional metric of number of posts on a cinematography forum. 

If you didn't read the thread properly, take a look at where Phil completely hijacks a thread in which we were trying to help someone with a question. 

If you don't have anything constructive to say and you rant about what's bothering you instead, that has a name: trolling. 

Link to comment
Share on other sites

  • Premium Member

I would never claim post count as a measure of correctness.

As I say if anyone wants to have a technical discussion about anything I'm up for it. I have a bunch of storage devices here right now that I'm supposed to be writing about, including:

- G-Tech G-Drive Mobile Pro SSD 1TB

- Samsung 870 QVO 2TB

- Samsung 970 EVO Plus 1TB

- Samsung X5

- Samsung T7 Touch

Doesn't seem like anyone's interested in courting coverage of their spinning disks anyway!

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...