Skip to main content

Lessons Learned

Lessons Learned

CHECK OUT OUR MOST RELEVANT INCIDENT LISTINGS!

Disclaimer: The Lessons Learned Database includes the incidents that were voluntarily submitted. The database is not a comprehensive source for all incidents that have occurred.

Description
Characteristics:
Damage and Injuries: Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source:
Setting: Refinery
When Incident Discovered: During Operations
Lessons Learned:
  1. The dilution water for making a polythionic acid aqueous solution was changed from industrial water to pure water to lower the chlorine concentration in the piping.
  2. Water that accumulated in the drain was removed before heating began. Steam tracing was strengthened to prevent condensation of moisture.
  3. The gusset was remodeled into a structure in which thermal expansion is not restricted.

Careful management oversight of startup procedures is necessary to avoid stress corrosion cracking from occurring in hydrogen piping, since it can result in a serious accident. The chlorine concentration can be high in industrial water, and it is desirable to avoid treatment using water with a high chlorine concentration.

Email (Primary):
Description
Characteristics:
Damage and Injuries: Minor Injury
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source:
Equipment: Piping, Valve, Flange
When Incident Discovered: During Operations
Lessons Learned:
  1. The plant tried to increase production and decrease workload, but failed to notice the potential negative outcomes of this change.
  2. It is necessary to thoroughly simulate all potential negative outcomes of a change in procedures.
  3. Although a valve stops a flow, it is necessary to consider potential leakage. A valve might leak due to fouling, solid material between seals, etc.
Email (Primary):
Description
Characteristics:
Damage and Injuries: Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source:
Setting: Refinery
Probable Cause: Equipment Failure
When Incident Discovered: During Operations
Lessons Learned:
  • Operation and management of the hydrodesulfurization unit should be strengthened to account for abnormal phenomena such as local corrosion.
  • The safety review system for remodeling existing equipment and facilities should be strengthened.
  • All refineries with the same kind of desulfurization units should thoroughly check dead-end piping areas. They should also conduct a safety review of their operating and management procedures.
  • If a very careful study is not done, safety aspects of remodeling projects might be disregarded, even if the initial purpose of remodeling is achieved.
  • Dead-end piping is the cause of various problems, so careful attention to areas of dead-end piping is needed to ensure safe operation.
Email (Primary):
Description
Characteristics: High Pressure (> 100 bar)
Contributing Factors: Design Flaw, Inadequate Inspection
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source:
Probable Cause: Equipment Failure
When Incident Discovered: During Operations
Lessons Learned:
  • Verify that all PRDs contain fuse-backed adapters.
  • Explore elimination of rupture disk PRDs and substitution of spring-style relief valves.
  • Develop new hydrogen unloading procedure using a written checklist. * Add instrumentation to monitor the high-pressure hydrogen system.
  • Confirm that temporary offices/facilities are not co-located with hazardous chemical storage sites.
  • A procedure is now in place that requires a Job Hazard Analysis (JHA) to be done on unloading hydrogen and shared enterprise-wide.
  • A competent plant employee must be present during all hydrogen unloading activities.
Email (Primary):
Description
Characteristics: High Pressure (> 100 bar)
Contributing Factors: Design Flaw
Damage and Injuries: None
Incident Date:
Severity: Near-Miss
Leak: No
Ignition: No
Ignition Source:
Probable Cause: Equipment Failure
When Incident Discovered: During Operations
Lessons Learned:

Investigation determined that internal galling has caused the failure rendering the needle valve unusable. The galling was caused by a stainless steel stem acting against a stainless steel seat. This failure mode had been observed before and the manufacturer had been previously notified. The valve was replaced with a new needle valve to continue with the test program. Additional discussion of best practices for equipment maintenance and integrity can be found in the Hydrogen Safety Best Practices Manual.

Email (Primary):
Description
Characteristics: High Pressure (> 100 bar)
Contributing Factors:
Damage and Injuries: None
Incident Date:
Severity: Near-Miss
Leak: No
Ignition: No
Ignition Source:
Probable Cause: Equipment Failure
When Incident Discovered: During Operations
Lessons Learned:

For the use of mechanical fittings in hydrogen service, administrative controls should be in place, as in this case, to ensure that leak testing is conducted on a regular basis. It should never be assumed that every fitting is tight. Additional discussion of best practices for fittings and joints can be found in the Hydrogen Safety Best Practices Manual.

Email (Primary):
Description
Characteristics: High Pressure (> 100 bar)
Damage and Injuries: Property Damage
Incident Date:
Severity: Incident
Leak: No
Ignition: No
Ignition Source:
Equipment: Compressor
Probable Cause: Equipment Failure
When Incident Discovered: During Operations
Lessons Learned:

Failure of a diaphragm is not infrequent, but the seizure of the main nut threads is very rare. The manufacturer claimed this had never occurred before with this type of unit. The broken plunger is likely the result of poor instructions/communications between the vendor and the user. The detail and quality of drawings provided by the vendor were poor given the level of investigation and repair needed for this occurrence. It is possible that the proprietary nature of some equipment information may have been a factor.

Email (Primary):
Description
Characteristics: High Pressure (> 100 bar)
Damage and Injuries: Lost Time Injury, Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: No
Ignition Source:
Setting: Laboratory
Equipment: Ball Mill, Reactor
Probable Cause: Equipment Failure
When Incident Discovered: During Operations
Lessons Learned:

The visiting intern had several years of experience in research projects and tasks that were similar to those in progress at this laboratory. This led to the incorrect assumption that formal introductory training for this laboratory was unnecessary. Steps should always be taken to ensure that anyone working in an unfamiliar laboratory setting is properly trained.

It is important to have written operating procedures for the use of laboratory-scale equipment involving hazards or risks of this nature. Operating procedures should also document any equipment parameter limits. Such procedures should be reviewed with all personnel as part of their laboratory training before they perform any experiments.

More information on management of change can be found in the Lessons Learned Corner and also in the Hydrogen Safety Best Practices Manual.

Email (Primary):
Description
Characteristics:
Damage and Injuries: Facility Closure, Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source: Substandard Electrical Wiring
Setting: Laboratory
When Incident Discovered: During Operations
Lessons Learned:

After the aforementioned incident, a rigid cage was designed to protect the reactor from external conditions, and to protect the contents of the hood and any experimenter from the reactor, in the event of a pressure burst from the reactor cell. Additionally, the experimental setup was redesigned to include only electrical distribution that has been verified to be in compliance with National Electric Code and that is located outside the cage. Further, the experimental setup has been redesigned to include a hydrogen supply line shutoff, so that if the pressure reactor cell integrity is compromised, the hydrogen supply is shut off. The fume hood has been fitted with a hydrogen sensor. Lastly, researchers were reminded that hydrogen experimental setups should be verified by another person, and hydrogen-involving experiments should be carefully monitored.

Hydrogen development work is inherently hazardous, and greater precautions than the norm need to be taken to ensure a safe work environment.

Email (Primary):
Description
Characteristics: High Pressure (> 100 bar)
Contributing Factors: Design Flaw, Operation Induced Damage
Damage and Injuries: Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: No
Ignition Source:
Probable Cause: Equipment Failure
When Incident Discovered: During Operations
Lessons Learned:

High-pressure fueling hoses should be examined daily for signs of external damage, including corrosion, abrasion, cuts, and kinking. High-use fueling hoses should be replaced every six months.

Email (Primary):
Description
Characteristics: High Pressure (> 100 bar)
Contributing Factors: Lack of Protocol/SOP
Damage and Injuries: Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source:
Setting: Power Plant
When Incident Discovered: During Operations
Lessons Learned:
  1. Revise fire protection operations surveillance tests to include a maximum of 110% of the agent net weight to prevent over-filled cylinders from being placed in service.
  2. Identify cylinders currently in service that exceed the 110% maximum allowable limit, remove them from service, have them inspected and refilled to the new criteria, and then return them to service.
  3. Purchase new compressed gas pilot control cylinders.
  4. Revise the plant industrial safety program to enhance cylinder storage requirements by eliminating the use of ropes as restraints.
  5. Purge the gas storage shed of all non-related storage, secure all cylinders with chain, fabricate cylinder racks for small cylinders, and eliminate storage of items next to the hydrogen system emergency manifold.
  6. Conduct a probabilistic safety assessment of the possibility of missile hazards from compressed gas cylinders.
  7. Change from the current CO2 vendor to another qualified service vendor.
  8. Investigate whether a breakdown of the fire protection QA program occurred.
Email (Primary):
Description
Characteristics: High Pressure (> 100 bar)
Contributing Factors: Situational Awareness
Damage and Injuries: Minor Injury, Property Damage
Incident Date:
Severity: Incident
Leak: No
Ignition: No
Ignition Source:
When Incident Discovered: During Operations
Lessons Learned:
  • The fuel cell vehicle that was involved in the accident has been retired. The fuel cell power plant from that vehicle has been removed and is being used in another fuel cell vehicle.
  • The fuel cell vehicle accident reinforced the need for training of drivers, supervisors and emergency response personnel. As an action item, this project team will conduct refresher training courses for the drivers and local emergency response personnel. The project leads conducted training classes on hydrogen safety and incident response for local emergency response personnel; including the local fire department and the police prior to vehicle deployment and the station opening. A significant learning by this project team is that emergency response agencies are subject to frequent personnel changes. As such, training should be repeated periodically.
Email (Primary):
Description
Contributing Factors: Human Error, Operation Induced Damage
Damage and Injuries: Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: No
Ignition Source:
Equipment: Vent line
Probable Cause: Fault in original weld
When Incident Discovered: During Operations
Lessons Learned:

An investigation team was formed to determine the cause of this anomaly. The team performed a fault tree analysis which identified the following five main paths or gates: fault in the original weld, age, component not built per design intent, overstress, and design error. This led to 48 events on the fault tree, all of which were investigated. The investigation, analysis and testing determined that the first path, fault in the original weld, was the primary source of the failure.

The fault tree analysis pointed to three probable and two possible contributors of the failure. The probable contributors included incorrect filler rod selected by welder, insufficient fusion in the weld, and insufficient weld penetration. The possible contributors included incorrect power level setting and incorrect heat level applied during welding.

Materials laboratory work confirmed that improper filler metal (4043 Al) was used to weld the 5083 Al alloy vent pipe. The 5083/4043 combination is not recommended for Al welding per either current or past standards. In addition, the failed weld was a very poor weld, most likely a field weld. NDE data indicated a higher number and severity of defects in this weld compared to other welds in the vent line. According to the material lab's failure analysis report, the weld failure resulted from tensile overload originating at the bottom surface of the vent line during tanking operations and was likely the result of a single overload event.

Vent line inspection was performed to identify other suspect welds. Inspection methods included the following: visual inspection, conductivity tests on both sides of each weld to identify the pipe metal (5000 or 6000 series Al), helium leak check of welds, non-destructive evaluation (NDE) using a combination of x-rays and ultrasonic testing, and silicon etch tests to identify weld material. The silicon etch test procedure was successfully developed in the course of this effort to identify 4043 weld filler material in the field.

Instrumentation attached to vent lines during subsequent operations has provided data for the ongoing weld stress analysis and screening criteria to determine which welds required clamshell repairs. Weld defect growth/propagation can now be monitored at periodic intervals.

An unexpected temperature difference of ~200 deg F between the top and bottom of the piping was found to occur during tanking operations in the same area of the failed weld. This temperature difference was responsible for the high thermal stress seen there. Data indicates that the peak stress in that pipe section occurred at about the same time that the weld failed and was higher than predicted.

Email (Primary):
Description
Characteristics:
Contributing Factors: Design Flaw
Damage and Injuries: None
Incident Date:
Severity: Incident
Leak: No
Ignition: No
Ignition Source:
Probable Cause: Equipment Failure
When Incident Discovered: During Operations
Lessons Learned:

The combination of the cold water temperature (reducing the fatigue strength of the bolt), and the abnormally high number of cyclical stresses imposed by the imbalance from the hydraulic system check valve failure resulted in the failure of the fasteners.

Email (Primary):
Description
Characteristics:
Contributing Factors: Inadequate Inspection
Damage and Injuries: Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: No
Ignition Source:
Equipment: Valve
When Incident Discovered: During Maintenance
Lessons Learned:

Proper installation of check valves and other equipment should be visually inspected prior to pressurization.

Email (Primary):
Description
Characteristics:
Contributing Factors: Inadequate Inspection
Damage and Injuries: None
Incident Date:
Severity: Non-Event
Leak: Yes
Ignition: No
Ignition Source:
Setting: Laboratory
Equipment: Reactor
When Incident Discovered: During Operations
Lessons Learned:

Maintenance on the low-pressure venting system was not occurring at regular intervals. Ventilation integrity is now checked before starting an experiment.

Email (Primary):
Description
Contributing Factors: Weather
Damage and Injuries: None
Incident Date:
Severity: Incident
Leak: Yes
Ignition: No
Ignition Source:
Equipment: Valve
Probable Cause: Abnormal Operations
When Incident Discovered: During Operations
Lessons Learned:

Excessive venting of hydrogen from the tank due to lower facility consumption, in combination with extreme temperature conditions, placed thermal stress on the gland nut, causing a leak. The low consumption of hydrogen resulted from the shutdown of some production equipment and the delay of additional production equipment coming online. The tank size is too large for the facility's current hydrogen demand.

The hydrogen supplier will conduct annual training on handling all types of gases used by the facility and will include the local fire department in this training. The facility will continue daily rounds to look for visual evidence of leaks, and the hydrogen supplier will exchange the 9,000-gallon tank for a smaller 4,500-gallon tank to significantly reduce pressure build up from lower usage.

Email (Primary):
Description
Characteristics: High Pressure (> 100 bar)
Damage and Injuries: None
Incident Date:
Severity: Near-Miss
Leak: No
Ignition: No
Ignition Source:
When Incident Discovered: During Operations
Lessons Learned:

Disable the fueling process if the operator does not follow the proper sequence of steps in the fueling procedure. Improve the fueling procedure to make it inherently impossible for the sequence to be done improperly with the same result.

Email (Primary):
Description
Characteristics: High Pressure (> 100 bar)
Contributing Factors:
Damage and Injuries:
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source:
Probable Cause: Equipment Failure
When Incident Discovered: During Operations
Lessons Learned:

Shutoff valve location and/or redundant shutoff valves at storage vessels are important for containment to prevent escalation.
Consider design features within equipment skids to reduce the likelihood of cascading events.
Pressure-switch component replaced with better design.
Component listing standards are important to further development/deployment of this technology.
Consider improving leak/fire detection and shutdown systems.

Email (Primary):
Description
Characteristics: High Pressure (> 100 bar)
Contributing Factors: Failure to Follow Procedure
Damage and Injuries: None
Incident Date:
Severity: Near-Miss
Leak: Yes
Ignition: No
Ignition Source: None
When Incident Discovered: During Operations
Lessons Learned:

Follow CGA G-5.5 vent end system designs.

Email (Primary): thomaswitte@witteengineeredgases.com