Premium Member Tyler Purcell Posted December 8, 2022 Premium Member Share Posted December 8, 2022 (edited) 6 hours ago, Perry Paolantonio said: You keep saying this, but it's simply not the case. Please provide documented proof (and not anecdata from some random forum) that this is truly an issue. This was a problem with Windows XP, for sure. 15 years ago. You wouldn't believe me even if it came out of a Microsoft's engineers mouth. So why should I bother. My company has built the top facilities in the country from Fotokem to Technicolor. Guess what our techs have said over and over and over and over again, the same damn thing about bandwidth, system resources and small files. When you actually sit down with storage vendors and Microsoft system engineers, to do a final proposal for a multi million dollar facility, these questions are always brought up. It's a HUGE issue in the audio industry where they'll have millions of small files (caches mostly) on raid arrays. This is one reason so many facilities work on mac's exclusively. They DO NOT have this issue at all. An example of this is very easy to produce at home. Take a set of 4k 10 bit DPX files, write them to a limited speed volume, lets say a 1tb NVME over USB-C via NTFS formatting. Then take that DPX folder and play it back on both a Mac and Windows system. You'll get somewhere around 12fps on BOTH systems. Now take that same file, put it on a Mac OS Journaled volume and play it back on the Mac, you'll get full 24fps playback. This is why so many storage solutions DO NOT USE NTFS formatting. They have proprietary formatting and use a program or plugin on the client/server system which does the translation. This is why Stornext has been so popular and why companies like DFT have sworn by it for so long. It eliminates all of the issues. 6 hours ago, Perry Paolantonio said: Most VFX houses are using image sequences of some kind (DPX, EXR, etc), and not having these issues. They are using shared storage without NTFS formatting. I only know two SAN providers who use Windows, all the ones we sell use Linux FOR THIS EXACT REASON! 6 hours ago, Perry Paolantonio said: We work with DPX sequences all the time, reading and writing them, on Windows machines. Our ScanStation is a Core i9 with a built-in 24TB RAID 5 that can easily do north of 1GB/second. We capture 4k DPX to this system at 15fps all the time. That's the max scan speed of the ScanStation in that resolution. We can do faster than that with lower resolutions. Bro, you don't understand the issue. It's overhead. NTFS has horrible overhead issues. You need to have 2x the bandwidth on NTFS systems to do the same thing you can do with Mac OS Journaled or extFS for instance. So sure, if you throw a bunch of drives at something, it will work. But the only reason you need to is because of how bad NFTS formatting is. Another thing that is programmable, is turning off indexing. When you write a lot of new files, the automatic indexing system will gobble up system resources. This is easy to turn off on the writing system, probably off on the Scan Station solution, but not magically off on any other system. It could be the reason why Lasergraphics sells a computer with the system, maybe is because of these issues. They have probably tweaked the OS quite a bit to get it to work properly. 6 hours ago, Perry Paolantonio said: On our Phoenix system we have an internal RAID-0 which we use as a scratch disk for caching. Caches are 10bit DPX at the native res of the job being done - typically about 4.5k. Again, we're easily able to play back and write cache files to this drive at faster speeds than our ScanStation. Phoenix is much slower at processing than the system can read and write. I don't think that example means much. You need a minim of 32 core Threadripper to get Phoenix to work at any speed worth discussing. Even our little 16 core 3950X system, ran Phoenix horribly slow, it's not storage. 6 hours ago, Perry Paolantonio said: Our SAN arrays can do more than 2GB/s and when we copy massive DPX sequences off the ScanStations internal RAID to the SAN, there are no issues as you describe. The SAN is windows based. San's generally do not use NTFS unless you're using something junky, even our complete toy Qnap is 100% linux. Edited December 8, 2022 by Tyler Purcell Quote Link to comment Share on other sites More sharing options...
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.