Skip to main content

Lessons Learned

Lessons Learned

CHECK OUT OUR MOST RELEVANT INCIDENT LISTINGS!

Disclaimer: The Lessons Learned Database includes the incidents that were voluntarily submitted. The database is not a comprehensive source for all incidents that have occurred.

Description
Characteristics:
Contributing Factors: Unknown
Damage and Injuries: Property Damage
Incident Date:
Severity: Non-Event
Leak: No
Ignition: No
Ignition Source:
Equipment: Batteries
Probable Cause: Unknown
When Incident Discovered: During Inspection
Lessons Learned:

All installed and certified safety and emergency systems functioned as designed.

1. The fuel cell turned off immediately after fire detection.

2. The fire suppression system was immediately initiated thereafter.

3. The physical separation of the batteries, the fuel cell, and the hydrogen tanks prevented the fire from spreading. This separation was developed from the FMEA of the ship and hybrid system.

4. No hydrogen leaked (i.e., the physical separation worked). However, direct fire contact or overheating of the hydrogen tanks would lead to a controlled automatic discharge of hydrogen outside the vessel.

5. The CO2 fire-fighting system in the battery room was activated for fire suppression. However, the hatch was left open by the battery supplier for the test run, which reduced the effectiveness of the suppression.

Email (Primary):
Description
Characteristics:
Damage and Injuries: Minor Injury, Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source:
Setting: Laboratory
Probable Cause: Human Error
When Incident Discovered: During Operations
Lessons Learned:

Several procedural and design changes should be considered for the future:

  1. Replace the use of pure hydrogen with a 95:5 mixture of nitrogen and hydrogen to reduce the possibility of an explosive atmosphere occurring. Laboratory personnel should check each tank that is delivered to ensure that the gases are present in the proper ratio.
  2. Adhere to the manufacturer's recommendations for operation of the anaerobic chamber.
  3. Following the check of the lines to make sure all the connections are tight, all gas cylinders should be closed; then, only the desired gas cylinder should be opened for use.
  4. Use of "T" connections between gases should be eliminated. If there is continued use of a "T" connection, only connections with a toggle switch to limit the introduction of gas from a single cylinder should be used. No exceptions, even on a temporary basis.
  5. The laboratory should continue to investigate the availability of hydrogen and/or oxygen sensors with the hope of finding some that can withstand the corrosive atmospheric environment.
  6. All laboratory personnel should receive refresher training that includes standard safety precautions as well as a more detailed review of the hazards of working with hydrogen. Hydrogen use in anaerobic chambers is discussed in the Lessons Learned Corner on this website.
Email (Primary):
Description
Characteristics:
Contributing Factors: Operation Induced Damage
Damage and Injuries: Facility Closure, Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source: self-ignition due to gas friction; mechanical friction or electric current
Setting: Power Plant
When Incident Discovered: During Operations
Lessons Learned:

The turbine components that caused the vibrations were a retrofit design which had been in service for about two years and were under warranty from the vendor. The root cause analysis of the event determined that the damage was caused by a defect in the design or assembly of the turbines.

Email (Primary):
Description
Characteristics:
Damage and Injuries: Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source: burnstack
Equipment: Vessel, Piping, Valve
When Incident Discovered: During Operations
Lessons Learned:

Recommendations:

  1. The using organization should define necessary activities in order to place hydrogen systems in long-term periods of inactivity. The defined activities should address requirements for rendering inert, isolation (i.e., physical disconnect, double block and bleed, etc.) and periodic monitoring.
  2. The using organization should develop a process to periodically monitor hazardous systems for proper configuration (i.e., a daily/weekly/monthly check sheet to verify critical purges are active).
Email (Primary):
Description
Characteristics:
Damage and Injuries: Minor Injury, Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source:
When Incident Discovered: During Operations
Lessons Learned:

Although the preparation-for-transport procedures were done the same way they were done for previous outreach programs, this time it proved to be a different situation. It is not clear what caused the ignition of the first balloon, which then set off a chain reaction to the others. The incident shows that preparation for transport is a very important element in the overall process, and it should be evaluated for risk factors along with every other element of the process.

Email (Primary):
Description
Characteristics:
Contributing Factors: Situational Awareness
Damage and Injuries: Minor Injury, Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source:
Equipment: Balloons
When Incident Discovered: During Operations
Lessons Learned:

Safe storage and transportation of balloons filled with a hydrogen-oxygen mixture is a very risky undertaking. There are few scenarios that do not involve enclosed spaces (e.g., a car) and the potential for static discharge. Perhaps a mesh bag would work, as long as sufficient ventilation is ensured. Nonetheless, using lecture bottles and filling balloons on-site seems to be the safest method. Yet if the floor in the demonstration area were carpeted, enough static could be generated to ignite a balloon. The demonstrator's greatest fear is that a child might ask to participate in the demonstration, then reach out to touch the balloon and have it detonate in their face.

The demonstrator feels fortunate that his injuries were relatively minor (no respiratory damage). He urges that a full risk assessment be performed prior to balloon storage/transportation and setup/performance of this type of science demonstration. Guidance for undertaking a hazard analysis and risk assessment can be found in the Hydrogen Safety Best Practices Manual.

Email (Primary):
Description
Characteristics: High Pressure (> 100 bar)
Damage and Injuries: Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source: an electrical arc from a sump pump
Equipment: Piping, Valve
Probable Cause: Design Flaw
When Incident Discovered: During Maintenance
Lessons Learned:
  • Active GH2 sensors should be installed and continuously monitored in all enclosed buildings near GH2 sources. All buildings near areas where hydrogen is used should be designed to preclude GH2 entrapment (e.g., sloping roof with ventilation at the highest point).
  • Underground carbon steel lines beneath concrete pad areas should not be used for GH2 transmission. All GH2 lines are now stainless steel and above ground. - Any GH2 transmission lines buried underground should be proof-tested and leak-checked on a periodic basis.
  • Any below-grade piping installation should be in open trenches covered by grating.
  • Facilities should be protected from GH2, at a safe distance, by manual isolation valves. If remote-operated valves (ROVs) are required for operational isolation purposes, the ROVs should be in series with and downstream of the manual isolation valve.
  • The pressure between isolation valves and stand shut-off valves should be routinely monitored on a daily basis.
  • Field repair of mechanically severable valves in high-pressure systems should be eliminated.
  • Valves repaired in the field should be subjected to functional and leak checks, including actuator and valve seals at simulated operating conditions. A written procedure should be prepared and used.
  • Valves utilizing pneumatic actuators should have actuator piston and piston nut staked (or locked by other positive means) in the installed condition.
  • All high-pressure gas lines scheduled to be inactive for periods greater than 6 months should be physically isolated by blind flanges from active systems.
  • Supply system status of pressure vessels and lines (pressure and/or quantity) should be recorded at the start and completion of operations each day. All reservoirs should be isolated at close of business each day, and before weekends and holidays.
  • Corrosion protection systems for underground lines should be reviewed and tested to confirm the adequacy of the systems.
  • Operational and support buildings at hazardous sites should be isolated (i.e., interconnecting air conditioning systems should be avoided). Buildings connected to hazardous sites by tunnels and/or conduits should be physically isolated by seals. If physical isolation is not practical, then positive air flow should be maintained in tunnels and conduits.
  • Explosive gas detection meters should be included in the equipment carried by firefighters and emergency medical personnel.
  • Fire alarm transmitters should be located at all hazardous locations.
  • Emergency instructions for isolating GH2 and utilities for hazardous locations should be permanently posted with names and telephone numbers of key individuals to be contacted.
Email (Primary):
Description
Characteristics:
Contributing Factors: Human Error
Damage and Injuries: Property Damage
Incident Date:
Severity: Near-Miss
Leak: No
Ignition: No
Ignition Source:
Setting: Laboratory
Probable Cause: Design Flaw
When Incident Discovered: During Operations
Lessons Learned:

The incident resulted from an inadequate design for the storage location of the copper gas supply tubing (too close to an electrical outlet). The gas supply tubing was too long for its intended purpose and posed a hazard in its coiled state near the outlet. This near miss had the potential for more significant damage/impact to the facility and to the researcher because of a hydrogen gas supply line also in close proximity to the same outlet.

Laboratories should be inspected to ensure that gas supply lines are protected against electrical exposure in the following manner:

  1. Limit the amount of copper tubing to the length that is necessary to reach the intended equipment.
  2. Secure gas supply lines to the wall and/or counter top in a way that will prevent electrical exposure.
  3. Perform visual inspections for loose lines before removing electrical plugs from outlets.
  4. Ensure that there are no exposed energized parts of electrical circuits or equipment near your compressed gas systems.
Email (Primary):
Description
Characteristics:
Contributing Factors: Deficiency in Procedures
Damage and Injuries: Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: No
Ignition Source:
When Incident Discovered: During Inspection
Lessons Learned:

The direct cause of the over-pressurization of the two drums was the repackaging of the phosphoric acid into metal UN1A1 drums and the resultant hydrogen gas generation within the sealed drums. At the time of this incident (1997), 49 CFR and several MSDSs supported the selection of the UN1A1 drums. After the incident, laboratory studies conducted by facility staff indicated that the corrosion and subsequent hydrogen gas generation rates for the amount of phosphoric acid present would result in a pressure buildup and the drum failures observed. Facility staff then contacted DOT to request that changes be made to the packaging guidance listed for this material.

Email (Primary):
Description
Characteristics:
Contributing Factors: Situational Awareness
Damage and Injuries: Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source: spark from switching instrument on
Setting: Laboratory
When Incident Discovered: During Operations
Lessons Learned:

The lessons learned from this incident are:

  1. Verify the gas that you are using.
  2. Avoid using "quick-disconnect" fittings in this type of situation. If they are absolutely needed, there are sets available that ensure that every pair of matching fittings are different from all the others.
Email (Primary):
Description
Characteristics:
Contributing Factors:
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source: self ignition
Setting: Power Plant
Equipment: Valve, Pipe Wrench
Probable Cause: Human Error
When Incident Discovered: During Operations
Lessons Learned:
  1. Follow the rules (e.g., using a torque-amplifying device requires supervisor approval).
  2. Some valves are susceptible to disassembly, with potentially significant consequences, if excessive torque is applied to the handwheel.
Email (Primary):
Description
Characteristics:
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source: unknown
Equipment: Compressor, Gasket, Flange
When Incident Discovered: During Operations
Lessons Learned:

The investigation team concluded that hydrogen gas was released through a failed 19-inch diameter gasket and ignited under the roof of the compressor shed where it was partially confined. Some gas escaped from the shed prior to the explosion, but it was confined beneath the deck of an adjacent structure and overhead piping. The compressor shed was originally just a roof over the compressors, but over time, walls were added to aid winter operation and maintenance. These walls resulted in confinement of the hydrogen and contributed to the violent explosion.

Unauthorized modifications played a major role in this incident. The team discovered that the original design specifications called for a spiral-wound gasket, but for the previous 7 or 8 years, only compressed asbestos fiber (CAF) gaskets had been used. It appears that the risk of the gasket disintegrating or blowing out during a high-pressure leak had not been identified.

Actions taken as a result of the incident included the following:

  • Checklists for startup and shutdown procedures, and design and engineering safety were updated and became mandatory.
  • Pressure testing at operating pressure became mandatory prior to process startup.
  • Process hazard analysis was introduced. - Gasket material specifications were revised.
  • Separation standards were developed and implemented for all site buildings and facilities.
  • Shatter-resistant windows and doors were installed.
  • Process enclosures were minimized in new designs and existing enclosures were opened up and/or forced ventilation was upgraded.
  • A formal modification procedure was instituted (i.e., management of change).
  • Additional combustible gas detectors were installed, with frequent calibration and maintenance required. - Emergency shutoff valves were installed on vessels with critical hydrocarbon inventories.
  • Awareness of chemical processing hazards was increased among all employees through better communication and training.
Email (Primary):
Description
Characteristics:
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source: Either hot clinker or sparks from chisel
When Incident Discovered: During Inspection
Lessons Learned:

The investigation determined that hydrogen was formed by the reaction of hot aluminum and water, air was admitted via the inspection door, and the mixture was ignited by the hot clinker or sparks from the chisel. Aluminum should have been separated from the refuse prior to feeding it to the incinerator, and this incident could have been avoided. Specific lessons learned included:

  • Incineration of crushed material with a high aluminum content was stopped.
  • Nitrogen should be used for purging combustible gas generated when blockage is removed.
  • Sufficient cooling time and safety confirmation steps must be completed prior to removing any blockage.
  • Water injection is no longer allowed. - An industrial camera and thermometer were installed for early detection of blockage.
  • The capacity of the ash pusher was increased.
  • Standard operating procedures were prepared for employees to follow.
  • A warning was issued to the public emphasizing the need for complete separation of combustibles and non-combustibles from their refuse.
Email (Primary):
Description
Characteristics: High Pressure (> 100 bar)
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source: unknown
Setting: Laboratory
When Incident Discovered: During Operations
Lessons Learned:

The ignition of the fireball could have been caused by any of the following mechanisms:

  • The inverse Joule-Thompson effect of hydrogen (i.e., heating upon expansion)
  • Some of the oil and light ends were above their auto-ignition temperatures
  • Contact with hot surfaces of the autoclave
  • Static electricity
  • Electrostatic discharge of the mixture.

The possibility that the explosion may have been caused by the hydrogen discharged from the autoclave was thoroughly investigated. However, there were no signs of combustion in the upper part of the cell. Also, the explosion occurred approximately five minutes after the rupture disc release, long after the hydrogen source had been shut off and more than one air exchange had occurred in the cell.

The following were identified as lessons learned from the incident:

  • The reactor outlet line was plugged with viscous oil, which resulted in over-pressurization of the system. If the plug had been in the feed line, the rupture disc on the reactor would never have been over-pressurized. If the staff had added a second rupture disc near the pump outlet to relieve pressure before the reactor rupture disc, with a check valve in between the two, the release might have been avoided. This design change was incorporated into the system rebuild.
  • The fixed-catalyst-bed tubular reactor system should have been reviewed and approved by health and safety, facilities engineering, or building management staff.
  • The inadequate hazard analysis of the installation resulted in failure to anticipate the need for containment or remote discharge of the rupture disc effluent. A thorough hazard analysis should have been done before operating the system.
  • The operators were not adequately trained on the hazards of the system. - Combustible materials and gas lecture bottles should not have been stored in the autoclave cell (i.e., poor housekeeping).
  • The fire failed to activate the sprinkler system because the exhaust fan was turned on within seconds of the rupture disc release, so the air change volume minimized the temperature rise within the cell.
  • The roof pressure vent panel failed to adequately relieve the explosion pressure.
Email (Primary):
Description
Characteristics: High Pressure (> 100 bar)
Damage and Injuries: Human Life, Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source: contact with hot surfaces and/or spontaneous ignition
Setting: Refinery
Equipment: Compressor, Gasket
When Incident Discovered: During Operations
Lessons Learned:
  • Carefully revise the risk assessment process to evaluate any confined areas where hydrogen is handled.
  • Separate the fire and gas detection alarm system from the process information to provide easy and clear identification.
  • Refresh and re-enforce personnel training on fire and gas identification and interpretation.
  • Retrain personnel on emergency procedures and enhance their understanding and awareness of risky, flammable, explosive, and/or toxic substances. Avoid underestimation of the hazards and complacency.
  • Re-enforce training and review training courses for recently hired personnel.
Email (Primary):
Description
Characteristics:
Contributing Factors:
Damage and Injuries: None
Incident Date:
Severity: Non-Event
Leak: No
Ignition: No
Ignition Source:
Probable Cause: Human Error
When Incident Discovered: During Inspection
Lessons Learned:

Process changes have been implemented for development and review of safety basis documents that focus on a collaborative effort between the preparer and reviewers in order to provide a more in-depth review. This change is anticipated to provide new perspectives that may compensate for human error.

Email (Primary):
Description
Characteristics:
Contributing Factors: Failure to Follow Procedure
Incident Date:
Severity: Non-Event
Leak: No
Ignition: No
Ignition Source:
Setting: Laboratory
Probable Cause: Design Flaw
When Incident Discovered: During Operations
Lessons Learned:

1.  Combustible gas detectors calibrated for hydrogen can falsely report hydrogen alarms due the presence of other gases the detector may pick up, such as carbon monoxide from engine exhaust or other sources. Since this event occurred, two hydrogen-specific alarms have been installed at this facility to eliminate false hydrogen alarms.

2.  A building's ventilation system can be a source of gases that can trigger a hydrogen alarm, especially a combustible gas detector used for hydrogen detection. In this case, there were multiple sources of non-hydrogen gases that likely triggered the hydrogen alarm. A boiler needing maintenance that was operating near the building ventilation inlet was a possible source of non-hydrogen gas getting into the building, and it has subsequently undergone repairs to minimize the likelihood of it being a gas source. The loading dock that is partially inside of the building is used to start equipment like snow-blowers during cold weather and is also a possible gas source. Finally, when the fire department arrived with 15 fire vehicles operating near the building for 4 hours, some of the exhaust gases were likely sucked into the building ventilation system as the hydrogen alarms continued to alarm even though all the hydrogen bottles had been removed from the building by order of the fire department after the first alarm response.

3.  Hydrogen storage capacity must meet storage regulations as defined by various agencies, including OSHA. Subsequent investigation by OSHA after this event found a violation in the building construction related to the 3,000 cubic feet (CF) of hydrogen being stored in ten 300-CF bottles. One cubic foot less of hydrogen storage capacity would have complied with the OSHA hydrogen storage standard for this construction type (reference OSHA regulation 1910.103(b)(2)(ii)(c) Table H-2, that has three storage capacities: less than 3,000 CF, 3,000-15,000 CF, and in excess of 15,000 CF). In this event, the building did not meet the minimum distance in feet for 3,000 CF and greater hydrogen storage, so subsequently the storage capacity was reduced by the removal to two bottles to bring the hydrogen storage capacity under 3,000 CF.

4.  Personnel should follow procedures for reporting hydrogen alarms to minimize outside personnel being unnecessary activated. Procedures in place for reporting hydrogen alarms had the following three levels of action: 1) for up to 10% of the LFL, the system is to be shut down and the Safety Department (on 24-hour call) notified, 2) for above 10% to 20% of the LFL, the premises are to be evacuated and the Safety Department notified, and 3) above 20% of the LFL, the fire department is to be called. Note that above 25% of the LFL, the alarm system automatically calls the fire department. In this event, the alarm levels were below 10% of the LFL, but the fire department was notified unnecessarily by the operating personnel. The research facility and other involved entities incurred additional expenses for emergency response that could have been avoided if reporting procedures had been followed.

Email (Primary):
Description
Characteristics:
Damage and Injuries: Lost Time Injury, Property Damage
Incident Date:
Severity: Incident
Leak: Yes
Ignition: Yes
Ignition Source:
Setting: Refinery
Probable Cause: Equipment Failure
When Incident Discovered: During Operations
Lessons Learned:

Based on the results of the company investigation and analysis of an amateur video, the company determined that the incident could have been caused by the failure of one of the following plant components:
· pipes leading to the reactor pressure gauges
· the recycled quench gas pipe at the bottom of the reactor
· the diathermic oil pipe (hot oil) entering or exiting the heat exchanger
· the heat exchanger flanged joints and connection lines.

The company determined that a release from the hot oil circuit could not have triggered the fire, based on the evidence from the pressure data in the circuit, which showed that the failure occurred 30 minutes after the fire started. The video confirms the pipe rupture 30 minutes after the fire began. For the same reason, a release from the hydrogen pipes is not considered likely, as the records demonstrate that the hydrogen pipe failed seven minutes after the fire began. When the heat exchanger flanged joints were dismantled, it was seen that the joint gaskets were not damaged. Thus, the company considers the failure of a pipe from the reactor pressure measurement gauges to be the most likely cause of the accident (although there is no conclusive evidence to identify the specific failure that caused the pipe to rupture). This assumption is supported by the following facts:
· This pipe is located in the area corresponding to the epicenter of the fire.
· The area corresponds to the area visually identified by the witnesses.
· The product release (hydrogen and fuel oil) from one of these pipes can cause a 6-meter long jet flame, as occurred.
· The product supposedly released would have had a high enough temperature and pressure to self-ignite or ignite against a plant hot spot (e.g., the hot oil circuit).
· The damages recorded were caused by overheating (flame exposition) and were not caused by overpressure or explosion. The pressure measurement records confirm no significant pressure changes at the beginning of the event.

The company decided to rebuild the hydrotreatment plant, in compliance with regulations, and to introduce the following process design changes:
· complete separation of the light fuel oil section and the heavy fuel oil section to avoid the possibility of "domino effects"
· lowering the maximum height of the heat exchanger installations from 25 meters to 15 meters to facilitate fire-extinguishing operations
· redesign of the piping system to minimize adjacencies
· relocation of the valves on the hydrogen quench line to enable depressurization
· reduction of the number of measurement gauges
· insertion of valves in a safe area for depressurizing the hot oil circuit.

Email (Primary):
Description
Characteristics:
Contributing Factors: Unknown
Damage and Injuries: Property Damage
Incident Date:
Severity: Incident
Leak: No
Ignition: Yes
Ignition Source: unknown
Probable Cause: Unknown
When Incident Discovered: During Operations
Lessons Learned:

The root cause of the fire that burned the evaporator pad and distorted the plastic evaporator pad bracket remains unknown. The initial investigation did not reveal any obvious signs of an ignition source in the vicinity of the forklift operation. The on-board data acquisition system did not indicate any abnormalities in the operating parameters of the fuel cell system (e.g., temperature, pressure, voltage, current). The fuel cell was disassembled, but no evidence was found of any electrical shorts or other potential ignition sources. Thus it was concluded that the fuel cell unit itself was not the ignition source for this incident.

One theory presented the possibility of a spark (caused by static electricity) being the source of the ignition that caused the fire. Due to the proximity of the fuel cell unit to a shrink-wrap packaging machine at the time of the incident, this seemed to be a plausible hypothesis. However, sparking tests on evaporator pad materials failed to confirm this, and it seems highly unlikely that a wet evaporator pad would ignite from static electricity. The true ignition source for this incident remains unknown.

After the initial investigation, the company used a hydrogen meter to monitor hydrogen levels near the evaporator pad during fuel cell start-up (which they expected to be the highest, due to a system purge). They also wanted to investigate if hydrogen could become trapped near the vent covering the evaporator pad. The tests indicated hydrogen levels well below the lower flammability limit (0.022%). Similar readings were also detected from the exhaust on the other make/model fuel cells operating in the facility. They detected no sign that high levels of hydrogen were trapped near the vent of any fuel cell make/model.

Email (Primary):
Description
Characteristics:
Contributing Factors:
Damage and Injuries: Property Damage
Incident Date:
Severity: Incident
Leak: No
Ignition: Yes
Ignition Source:
Probable Cause: Human Error
When Incident Discovered: During Operations
Lessons Learned:

It appears that this was an isolated event caused by human error. The lessons learned are: (1) to caution workers to maintain their focus during fuel cell stack assembly, (2) to require verification that all tools and spare parts are accounted for prior to closing up the system, and (3) to review quality control procedures and assembly procedures with an eye toward improvement.

Email (Primary):