How RoviSys deployed Stratus Edge Computing to eliminate downtime and data loss, converge OT/IT, and reduce FAT/SAT from weeks to days for a life sciences company
Eliminate downtime and data loss in batch manufacturing for a life sciences company
Ensure data reliability for FDA regulation, GMP, and VCAP
Standardize control architectures for three sites across a multi-site campus
Enable manufacturing operations team to maintain and support computing infrastructure without high-level IT support
Solution
Stratus ftServer® and Stratus ztC Edge™ for fault tolerance and data collection
Rockwell Automation PlantPAx® software for campus-wide production control
Benefits
Eliminated downtime and consolidated software workloads by replacing six hyperconverged server clusters with Stratus Edge Computing
Standardized and scaled computing across different life sciences facilities and use cases
Reduced the FAT/SAT process from weeks to hours; completed full validation in one week
Cut total cost of ownership (TCO) by 15-20% and projected duty cycle of 15 years for compute platforms
Bringing reliability to batch manufacturing in life sciences
A leading life sciences company with a portfolio spanning cell biology, DNA, RNA, and protein analysis, drug development, human identification, and molecular diagnostics needed to eliminate downtime and data loss in its manufacturing operations by selecting a more reliable and easy-to-maintain compute infrastructure for its batch manufacturing operations. The company had been relying on hyperconverged server clusters and software-based fault tolerance to manage process automation and data required for FDA regulation.
Consulting with RoviSys®, a global systems integrator specializing in automation, the company standardized on Rockwell Automation PlantPAx® software running on Stratus® Edge Computing for three manufacturing sites on the company’s campus. RoviSys deployed fault-tolerant Edge Computing platforms for process automation that dramatically simplified the life science company’s IT infrastructure. Using Stratus platforms, RoviSys deployed the control architecture in a week – from factory acceptance testing (FAT) to site acceptance testing (SAT) and validation while reducing TCO by 15-20% as compared to the prior cluster solution. Since the Stratus deployment, the life sciences company has experienced no downtime, and is able to service the compute platforms without dedicated IT support.
Server clusters insufficient for life sciences manufacturing
The company had been relying on server clusters to provide computing and data collection for its critical manufacturing processes. The team faced a number of costly challenges, from component failure to unplanned system outages, that resulted in data loss or downtime. This impacted their ability to pass FDA audits and GMP reporting and led to crippling manufacturing disruptions and product waste. If a cluster server failed, the team often did not have immediate support and knowledge to rebuild the node requiring additional recovery time. Additionally, the team needed to be able to track their VCAP (version change, audit control) process to map system changes to specific batches and vice versa.
Initially, RoviSys considered a hyperconverged cluster based on Cisco® HyperFlex that would reside in a corporate data center and would be managed by the end user’s team. However, that choice entailed several hurdles. The solution required a virtual SAN with 10 gig networking across the customer’s campus which required a lot of expertise and a lot of bandwidth. In addition, the highly customized and complex nature of the solution not only needs extensive licensing costs, but would also require a team of highly trained people. From a practical standpoint, this allowed too much margin for error in the setup and management of the cluster system.
Simple, Protected, Autonomous Edge Computing built for OT
To provide a simpler and easily serviceable availability solution, RoviSys recommended the Stratus ftServer and Stratus ztC Edge compute platforms. In contrast to the more complex hyperconverged cluster that required significant integration work, Stratus platforms are ready off-the-shelf to be installed quickly. The operational simplicity was a key factor making the Stratus platforms a solution that the OT team could use without needing specialized IT skills. Installation and end user training took about two hours – instead of at least two days for the HyperFlex solution. The team also calculated the end-user’s total cost of ownership (TCO) to be 15-20% lower with Stratus.
The Stratus platforms provided an exceptional level of data protection with 99.999% availability and no risk of costly downtime while enabling stronger cybersecurity. Due to the hardware-based fault tolerance, there was no failover delay or data loss. In the event of component failure, the customer’s operational team could easily swap out plug-and-play components with no impact to the batch application, process disruption, or data loss. With the uninterrupted operation of the applications and VMs running on the system, validation testing can be completed quickly, saving significant time and resources.
In comparison, with a cluster node failure, the team would have faced multiple steps to replace and validate a new node, from physical replacement and physical and logical validation to confirmation that all VMs across all nodes were functioning. The entire process for a manufacturing site might require as many as ten people on a customer’s validation team plus two or three consulting resources to run the testing and occur over days or weeks.
Production control architecture with Rockwell Automation and Stratus
For production control, RoviSys architected Rockwell Automation and Stratus solutions for three facilities – a large-scale protein manufacturing site, a site producing DNA and RNA reagents, and a small molecule site feeding other manufacturing processes. For the two larger sites, the team matched Rockwell Automation PlantPAx, including HMI, historian, batch, Asset Centre, Vantage Point, Thin Manager, Engineering Workstation, and SQL, running on a Stratus ftServer 6910 and 2900, respectively. The small molecule site did not require 5 nine plus reliability and ran the PlantPAx suite on a Stratus ztC Edge 110i platform. For network security, the platforms were segmented based on Rockwell Automation’s recommended cybersecurity practices.
The team set up seven VMs per production line to consolidate software workloads. The efficiency also translated into significant savings on software licenses, for example, costing only $600 for VMware for ftServer as compared to thousands of dollars per node in a cluster solution. Similarly, the team was able to realize cost savings with Microsoft data center licenses.
Shorter FAT/SAT cycles and quick validation
For deployment with the Stratus platforms, RoviSys accomplished the crucial FAT/SAT testing and validation processes in hours versus weeks with a cluster solution. The team was able to plug and unplug nodes and verify immediately failover without disruption, including running a library of Linux routines to test that various components and low-level software were up-to-date as part of the FAT/SAT process.
The rack-mountable, fault tolerant, and self-healing design of the Stratus platforms enable them to be deployed in production for a lengthy time. The team forecast a 15-year duty cycle for the platforms and concluded that with Stratus’ longevity, durability, and streamlined FAT/SAT verification, the life sciences company would have a top-performing, high-availability solution for the long haul.
OT/IT convergence, shortened FAT/SAT cycles, and fault tolerance a win in life sciences
Working with their life sciences end user, RoviSys quickly deployed Stratus Edge Computing and transformed the reliability of their customer’s manufacturing operations, VCAP process, and GMP reporting. The solution dramatically simplified IT infrastructure and enabled the OT team to deploy and maintain new infrastructure without on-site IT support. Importantly, RoviSys’ key to success was running a successful proof of concept (PoC) and using a solution that reduced FAT/SAT and validation testing from weeks to days. With the trust built from this success, RoviSys has expanded its end user engagement for additional projects.
