The Y2K issue is a unique and unprecedented problem that could threaten the mission of the university and greatly impact its ability to provide utility services to campus buildings and facilities. Predictions regarding the effect of the Y2K rollover range from no impact whatsoever to complete loss of all utility and computing resources and services.
In August 1998, a utilities department Y2K team was instituted to look at the utilities issues and potential impact to the university. This plan concentrates on the Central Plant and distribution systems that provide utility services to campus buildings. Any Y2K issues that exist within the buildings themselves, or in their associated control systems or components, are outside the scope of this plan. Three (3) separate functions were identified as needing to be accomplished prior to June of 1999. These functions are "Inventory," "Verification," and "Testing." Specifically, the team was charged with performing the following:
The team was charged with brainstorming and developing an inventory of all equipment, systems and components that had date/time functions and/or contained electronic embedded chip technology, and they developed a list using Excel.
The inventory list was then divided up among the team members, who were assigned the task of verifying the Y2K compliance status of each component. Team members could use any means necessary to verify status (i.e., phone calls to manufacturers, use of manufacturer web pages and matrixes, etc.). Each member was asked to ensure that a hard copy of the verifications was received for the records.
Next, the team developed a list of all mission-critical systems and organized the information under the headings "internal threats to mission-critical systems" and "external threats to mission critical systems." A plan to upgrade and/or correct non-compliant components, as well as a request for funding, was forwarded to the Campus Operations Director and Facilities Services Business Manager last November.
On December 18, 1998, the team developed a test plan listing the components and systems to be tested for compliance "on site." Based on testing results, the team will continue to modify its contingency plan, which outlines actions to be taken in the event of the failure of one or more mission-critical systems. The contingency plan also lists possible alternative actions to increase system reliability or provide additional backup for critical systems.
The following list consists of mission-critical systems that provide vital services to campus facilities and buildings, and briefly describes the impact of a loss of one or more of those systems. Included is a description of the "internal threat"(computer software, hardware, or embedded chips) that could cause a failure of that system, as well as a description of the "external threat" (outside service provider) that could cause a failure.
1. Central boilers and steam distribution. This system provides steam for heating, hot water, ancillary services, and internal power generation. The boilers are controlled through a Honeywell DDC electronic control system with date/time functions as well as embedded chips throughout the system’s components. The loss of this system during the winter months could wreak devastating financial hardships on the institution due to freezing/cold weather, as well as the loss of all heating, hot water, autoclaves, and environmental controls in the buildings. The loss of the steam system would also reduce emergency power capability.
Threats:
Honeywell Control Systems - Internal
Natural Gas Supply System - External
Compressed Air System - Internal
Electrical Power System - External and Internal
City Water System - External
The team completed an inventory of all internal Honeywell electronic and computer component parts that has been verified and checked with Honeywell Controls Company. A Y2K compatibility problem has been found with the supervisory computer control system and with the programmable logic controller cards. A plan is underway to correct this deficiency and should be completed by April 1999.
Northwest Natural Gas Company and IGI Resources have been contacted about Y2K compliance. Both companies say they have completed upgrades or have plans in place to ensure compliance with any known Y2K issues to the best of their ability. However, they don’t rule out the possibility of unknown or unforeseen conditions that could affect their ability to provide services. [Mission-Critical Systems]
2. Chillers and chilled water distribution. This system provides 42-degree water for cooling buildings, computer systems, and other process loads around campus. The loss of this system in the winter months would cause minimal impact to comfort on campus, but could cause serious repercussions to the computer and process cooling loads in buildings like Deschutes and the Computing Center.
The chillers are controlled through the use of Trane electronic control systems, which are tied directly into the Landis & Staefa DDC electronic control system. These systems have date/time functions as well as embedded chips located throughout the systems’ many components.
Threats
Landis & Staefa controls system - Internal
Trane control system - Internal
Variable frequency drive controls - Internal
Electrical power system - External and Internal
Compressed air system - Internal
City water system - External
All internal electronic and embedded chip technologies in the Trane controls and the VFDs have been inventoried, checked, and verified against the manufacturer’s data. There appear to be no compliance problems at this time. The Landis & Staefa control system verification is being coordinated by the UO's Facilities Services System Administrator. [Mission-Critical Systems]
3. Compressors and compressed air distribution. This system provides compressed air to operate central plant control systems, building control systems and other miscellaneous uses. If the system fails, it would shut down the central plant and cause loss of steam and chilled water services to campus buildings.
Threats
Electrical power system - External and Internal
Chilled Water - Internal
City Water - External
An inventory has been completed and the only device that contains an embedded chip is the Toshiba Controller on the Quincy Rotary Screw air compressor. Toshiba has verified that their controllers have no apparent Y2K problems. [Mission-Critical Systems]
4. Electrical power. Power is delivered to the UO at 12,470 V under a EWEB high voltage E-2 rate schedule. The electrical power is reduced to building use voltage and delivered to campus buildings through a university-owned electrical distribution system. Electricity is also co-generated in the Central Plant as a byproduct of steam generation during the fall, winter, and early spring months, and as such, can provide some emergency power to the campus in the event of an EWEB power outage. The Central Plant boilers must be online to provide this capability. A loss of electrical service will affect the majority of campus lighting, HVAC, controls, computers, and other miscellaneous devices, and can lead to potential flooding in tunnels and basements.
Threats
EWEB (Eugene Water and Electric Board) - External
BPA (Bonneville Power Administration) - External
COE (Corps of Engineers) - External
Woodward electronic governors - Internal
Electrical Feeder Equipment (ME Controllers, S & C Switchgear) - Internal
EWEB has developed a Y2K action team and completed inventories of company systems and components. We have received conformation that they are progressing through all noncompliance issues as they arise and are working with other agencies to develop contingency plans. However, EWEB acknowledges that unforeseen conditions or events may affect their ability to deliver electricity. [Mission-Critical Systems]
5. City Water Supply System. This system provides water to all potable services, fire mains, process cooling, and other miscellaneous services. Loss of this system would have a serious effect on the institution as well as the city.
Threat
EWEB (Eugene Water and Electric Board) - External
EWEB has formed a Y2K action team and completed inventories of company systems and components. We have received conformation that they are progressing through all noncompliance issues as they arise, as well as working with other agencies to develop contingency plans. However, EWEB does not rule out unforeseen conditions or events that may affect their ability to deliver water. [Mission-Critical Systems]
6. City sanitary sewer system. The city sewer system consists of pumping stations and treatment facilities for sewage removal and treatment.
Threat
City of Eugene - External
The City of Eugene is currently developing a written Y2K compliance statement, which it will distribute in the near future. As the city’s sanitary pumping stations rely on EWEB electricity, it is also writing in a disclaimer. [Mission-Critical Systems]
The contingency plan below outlines actions to be taken by Utilities Department staff in the event of a failure of one or more of the mission-critical systems that feed the campus.
A. Failure of Honeywell Control System
i. Safely shut down boiler systems and turbine generator
ii. Contact EWEB for activation of the Intertie system
iii. Call in personnel to troubleshoot the Honeywell system.Resources needed:
Extra staffing
Capability of EWEB assuming our load, or partial load
Availability of troubleshooting/controls personnelNote: Y2K-compliant controls should be installed by April 99. The commissioning of controls includes "time testing" by setting their internal clocks ahead.
B. Natural Gas System Failure
i. Shift boilers to #2 fuel oil (we have approximately 2-3 days’ fuel supply on hand)
ii. Shut down turbine generator, provided EWEB power is not affected
iii. Stay in contact with NNG Co.; restore systems when gas is availableResources needed:
Extra staffing
Verify availability of #2 fuel oil supplies/deliveriesC. Air Compressor Failure
i. Start Worthington "Y" Compressor (no electronic controls)
ii. Contact manufacturer for repair/restoration of Quincy Rotary CompressorD. Power Outage (EWEB)
i. Safely shut down boiler systems and turbine generator
ii. Stay in contact with EWEB. Once power is restored, re-energize campusResources needed:
Extra staffingE. Water Outage (EWEB)
i. Safely shut down boiler systems and turbine generator
ii. Stay in contact with EWEB. Once water is restored, restart plantNote: Monitor boiler water levels. If water levels drop to the low alarm limit, perform emergency shutdown. (Explosive consequences may result if not accomplished)
Resources needed:
Extra staffing
Alternative (this can be accomplished with additional funds): Purchase/install a pump and sand filter system to allow use of Millrace water for boiler systems.
A. Failure of Landis & Staefa Controls (panel 81)
i. Shift operations over to manual operation of chillers #1 or #2
ii. Shift over to campus pumps #1 or #2
iii. Shift Millrace pumps to manual mode
Alternative modes of operation: Restart chiller #3 or #4 and associated pumps in manual control. Campus pumps #3 & #4 can also be operated in manual mode.
B. Failure of Trane Controls
i. If Trane Summit BCU fails, start chillers #1 or #2 and associated pumps in manual. (Will lose Landis & Staefa communications to all Trane units)
ii. If Trane Chiller control panels fail, contact Trane’s emergency phone/pager
C. Failure of VFD Controls
i. Start Chillers #1 or #2 and associated pumps
Note: If Safetronic VFDs are affected, contact King Group, Inc. Campus pumps #1 and #2 are the only pumps with VFD bypass capability.
D. Power Outage (EWEB)
i. Place controls in safe mode; restart when power is restored
E. Air Compressor Failure
i. Start Worthington "Y" compressor
F. Water Outage (EWEB)
i. Monitor system pressure.
ii. If pressure drops to 20 psi or less, shut down system and chillers
iii. Restart system once city water is restored 3.
A. Power Outage (EWEB)
i. Follow shutdown procedures for Boiler Plant, Turbine Generators, and Chiller Plant.
Ii Restart systems once electrical power is restored.Note: Compressed air is the heart of all campus and facility control systems. Loss of this system would cause emergency shutdown of most campus infrastructure systems that provide heating, cooling and ventilation.
Alternative: This can be provided with extra funding. Rent a portable diesel-powered Air Compressor for emergency standby. Have 2-3 day’ supply of diesel on hand
B. Chilled water system failure
i. Shift to Worthington "Y" Compressor , which uses city water as cooling source
C. Water outage (EWEB)
i. Maintain operation of Quincy Rotary Compressor (uses chilled water as cooling source)
A. EWEB System Failure
Note: Prior to Y2K rollover, plant configuration will consist of Boilers #1 & #4 online, #1 T/G rolling, #3 T/G online paralleled to the 4160V bus.
i. If successful in saving Station Service with #3 T/G, power will continue to be provided to CPS, Computing Center and Pacific Hall. Emergency Gen Sets should start at OPS, Willamette, Huestis, Oregon Hall for Telecommunication Services, and Mac Court for emergency lighting.
Ii If NOT successful in saving Station Service, follow the system emergency shutdown procedures. Keep OPS informed as to status of EWEB power
iii. In the event of an extended outage, shift switch gear configuration to condition #4 and selectively re-energize vital loads up to 80% of T/G capability
iv. If partial EWEB power is restored, realign to condition #1 and selectively energize individual loads up to maximum available power
Resources needed:
ii and iii above will require a large amount of personnel resources and coordination to switch individual campus building loads. If too many loads are added, or load is added to quickly, it could cause the loss of the entire system.
A Plan will need to be developed to prioritize buildings and loads.B. Reclosure "ME Controller" Failure
i. Follow same procedures as listed above for EWEB Outage
ii. Notify EWEB of failure and request assistance
iii. Troubleshoot and restore systemAlternatives: These can be accomplished with additional funds. Rent a 750 KVA emergency generator to have on hand.
Design a quick connect to enable restart of Station Service and CPS.Proceed with a permanent diesel powered emergency generator installation at the CPS as per Balzhiser & Hubbard Engineers’ 1996 plan.
A. EWEB System Failure
i. If unable to quickly restore or access back-up Millrace water supply, perform controlled Plant and systems shutdown
ii. Stay in contact with EWEB and OPS, restart systems when water is restored
A. City of Eugene Treatment Plant or Pumping Station failure.
i. Stay in contact with City of Eugene and OPS