Deploy SCADA applications on resilient, fault-tolerant platform
Provide reliable automated instrumentation for dozens of sites and facilities throughout the DC Water system
Create real-time data for analysis and executive/regulatory reporting
Solution
AVEVA System Platform, InTouch®, and Historian
Rockwell PLC
Schneider Electric PLC
Stratus Edge Computing Platform
Benefits
Absolutely 0 unplanned downtime since install in 2015
Total solution failover time was reduced by 90% from 20 minutes to 2 minutes, resulting in improved service reliability
Fast access to timely data for KPI analysis and regulatory reporting
It’s the vital dual service that virtually all residents of any municipality rely on every day — but that hopefully goes unnoticed: drinking water consumption and wastewater disposal. Water utilities are the foundation of any city or town, but for the District of Columbia Water and Sewer Authority (DC Water), the challenge is magnified by that fact that it services some of the nation’s most important buildings, including the White House, the Capitol, national museums, and other vital federal buildings.
The scope and scale of the mission have led to a massive infrastructure that pumps more than 99 million gallons of water to 681,000 residents through a web of 1,350 miles of pipes, four pumping stations, five reservoirs, three tanks, 43,000 valves, and 9,500 fire hydrants.
With a total service area of roughly 725 square miles, DC Water also treats wastewater for 1.6 million people in the District as well as neighboring Montgomery and Prince George’s counties in Maryland and Fairfax and Loudon counties in Virginia. The centerpiece of those efforts is the 150-acre Blue Plains Advanced Wastewater Treatment Plant, the largest facility of its kind in the world. It treats an average of nearly 290 million gallons per day — with a peak capacity of nearly 1 billion gallons per day. Thanks to a sophisticated SCADA system operating on Stratus Edge Computing Platform, the agency can achieve the availability goals that its large water system requires. To make this all possible, InSource Solutions, a Digital Transformation company and Stratus distribution partner, facilitated DC Water’s new solution.
“Of course, we also have upper management who are looking at the KPIs, and regulators like the EPA want to see our data, so it’s crucial to have reliable and valid data that’s complete and timely. This all comes from Historian and our SCADA software that reside on our Stratus Edge Computing Platform.” Karen Green SCADA Manager DC Water
The Critical Importance of SCADA to DC Water
Operating one of the largest water utilities in the country, it’s no surprise that a strong supervisory control and data acquisition (SCADA) system is a non-negotiable requirement for DC Water. The agency relies on the AVEVA System Platform for secure, real-time automation, operations, and control of its water pumping stations, storage facilities (tanks and reservoirs), sewage pumping stations, stormwater pumping stations, and fabridams on the Potomac River.
The AVEVA implementation spans databases called Galaxy Repositories, two redundant AVEVA Historian instances, two application object servers, and two terminal servers that run AVEVA InTouch® HMI for visualization. DC Water operates these applications in three control rooms.
A Resilient Edge Computing Architecture
Unsurprisingly, with so much at stake, DC Water has designed a sophisticated Edge Computing infrastructure to minimize or eliminate any downtime. “The one thing we absolutely cannot tolerate is a failure in any of our servers,” said Karen Green, SCADA manager for DC Water. “If we’d implemented regular servers and experienced any sort of failure, we would be dealing with downtime — pumps and valves not working, water not reaching our customers. That’s obviously unacceptable.”
“If a server fails, the other one takes over and there’s no downtime. This wasn’t always the case. In previous iterations, it might take 20 minutes for the system to failover. Now, with a different configuration, our failover is negligible” Karen Green SCADA Manager DC Water
Each DC Water site — from storage facilities and tanks to pumping stations to sewage facilities — has multiple programmable logic controllers (PLCs) from both Rockwell and Schneider Electric that feed data to a central server through a control network running on private fiber optic. To build in high levels of system redundancy in its computing architecture, DC Water operates a separate management network driven by two physically separated Stratus Edge Computing Platforms. Optimized for distributed edge-in architectures, Stratus Edge Computing Platforms deliver the performance needed to support advanced processor- and data-intensive applications, while also providing the fault tolerance, security, analytics, and manageability DC Water requires.
“Our AVEVA Historian instances are for data collection,” Green explained. “We push data from our Tier 1 Historian to a Tier 2 Historian where users across the authority can access various data including pump statuses, water pressures, water analyzers and power information. We’ve configured them as store-and-forward servers, so that if there’s any interruption, the data will backfill when the connection is restored.”
The one thing we absolutely cannot tolerate is a failure in any of our servers. If we’d implemented regular servers and experienced any sort of failure, we would be dealing with downtime — pumps and valves not working, water not reaching our customers. That’s obviously unacceptable.” Karen Green SCADA Manager DC Water
Complete, Reliable, and Valid Data
Naturally, the frontline users are the dozens of people who operate the equipment at remote sites using AVEVA InTouch® HMI. SCADA analysts like Green configure the various SCADA nodes and analyze the data to optimize performance. “Our analysts and mechanical and maintenance teams also use that data,” Green said. “The maintenance team, for instance, uses the historical data and alarms for troubleshooting and to prevent recurrences. We configure the reports, and they are automatically emailed to engineers and managers.”
Members of the water quality team can monitor their equipment and receive alarms and notifications on their phones if variances or failures arise — an important consideration for people managing water and wastewater systems. “Of course, we also have upper management who are looking at the KPIs, and regulators like the EPA want to see our data,” she said, “so it’s crucial to have reliable and valid data that’s complete and timely. This all comes from Historian and our SCADA software that reside on our Stratus Edge Computing Platform.”
The Power of Stratus for Edge Computing and Support
DC Water operates a Stratus Edge Computing Platform, backed up by another (physically distant) Stratus Edge Computing Platform, creating two layers of dual redundancy. This configuration gives DC Water’s sprawling operation exceptional levels of reliability.
“With Stratus, we have fault tolerance and redundancy,” Green said. “If one server fails because of a power outage or communications disruption, the other one takes over from another location and there’s no downtime. This wasn’t always the case. In previous iterations, it might take 20 minutes for the entire system — AVEVA System Platform, InTouch®, and Historian — to failover. Now, with a different configuration, our software/network failover is negligible.
“We’ve found that virtualization is very helpful to our agency. For example, if we need to perform maintenance, updates, or repair work, we can easily move one virtual machine to another server without any interruption. In fact, the other day, we had a problem with some PLC software. We just loaded it on the Stratus Edge Computing Platform, and that made things very easy for us. With virtualization, we don’t have to maintain an individual box for each server. We just have, basically, one large box that can run many virtual machines. We get one level of redundancy because Stratus Edge Computing Platforms have dual processors working in parallel. We can also backup, duplicate, and restore the VMs whenever we need to.”
Green also underscored the importance and value of the support she’s received from Stratus. “That’s important — very important,” she said. “In our line of work, we need responsive support, and Stratus’s engineers are right alongside us on every issue. Even if it’s a non-urgent issue, they will call me to follow up until the issue is resolved. They’re very responsive and diligent — and that’s just what we want.”.
In our line of work, we need responsive support, and Stratus’s engineers are right alongside us on every issue. Even if it’s a non-urgent issue, they will call me to follow up until the issue is resolved.
