> Non redundant fuel pump that doesn't even restart on power failure
The crew weren't using the redundant fuel pumps. They were using the non-redundant fuel line flushing pump as a generator fuel pump, a task it was never designed for and which was not compliant.
That it doesn't restart on restoration of power is by design; you don't want to start flushing your fuel lines when the power returns because this could kill your generators and cause another blackout.
> Main engine shutting of (sic) when water pressure drops
Yeah, this is quite bad. There ought to be an override one can activate in an emergency in order to run the engines to the point of overheating, under the assumption that even destroying the engine will cause less catastrophic consequences than not having propulsion at the time.
> backup generator not even starting in time
There were 5 generators on board. Generators 1 through 4 are the main generators on the HV bus side, and the emergency backup generator is on the LV bus side.
When the incident occurred, the ship was being powered by generators 3 and 4, which were receiving their fuel via the non-redundant fuel line flushing pump. These generators powered the HV bus, which powered the LV bus via a transformer. The emergency backup generator was not running, so the LV bus was only receiving power from the HV bus via 1 transformer.
The incident tripped the circuit breaker for this transformer, disconnecting the HV bus from the LV bus, resulting in the first LV bus blackout. This resulted in main engine shutdown (coolant pump failure) and an automatic emergency backup generator startup.
There is an alternate (backup) set of circuit breakers and transformer that could have energised the LV bus, but the transformer switches were left in the manual position, so this failover did not happen automatically and immediately. There were no company procedures or regulations which required them to be left in the automatic position.
The LV bus also powered the fuel line flushing pump, so this pump failed. As a result, generators 3 and 4 started to fail (being supplied with fuel by a pump which was no longer operating). The electrical management system automatically commanded the start of generator 2 in response to the failing performance of generators 3 and 4.
Generator 1 and generator 2 were fed by the standard fuel pumps, which were available. One main generator is capable of powering the entire ship, so there was no need to start generator 1 as well; this would have just put more load on the HV bus (by having to run the fuel pump for generator 1 as well).
Instead of the automatic transformer failover (which was unavailable), the crew manually closed the same circuit breaker that had already tripped, 1 minute after the first LV bus blackout.
This restored power to the LV bus via the same transformer that was originally powering it, but did not restart the fuel line flushing pump supplying generators 3 and 4 (which were still running, but spinning down because they were being fed fuel via gravity only). This also restored full steering control, but this in itself was inadequate to control the vessel's course without the engine-driven propeller.
The main engine was still offline and takes upwards of half a minute to restart, assuming everyone is in place and ready to do so immediately, which was unlikely.
The emergency backup generator finally started 10 seconds later (25 seconds too late by requirements, 70 seconds after the first LV bus blackout).
Generator 2 had not yet gotten up to speed and connected to the HV bus before generators 3 and 4 disconnected (having exhausted the gravity-fed fuel in the line ahead of the inoperative fuel line flushing pump), resulting in an HV bus blackout and the second LV bus blackout. With only the emergency backup generator running on the LV side, only one-third of steering control was available, but again, this was inadequate without the engine.
3 seconds later, generator 2 connected to the HV bus. 26 seconds later, a crew member manually activated the alternate transformer, restoring power to the LV bus for the second time.
The collision was preventable:
- It is no longer a requirement that the engine automatically shuts down due to a loss of coolant pressure. It was at the time the vessel was constructed, but this was never re-evaluated. If it were, the system may have been tweaked to avoid losing the engine.
- If the transformer switches were left in the automatic position, the LV bus would have switched over to being powered by the second transformer automatically, and the engine coolant pumps and fuel line flushing pump would not have been lost.
- Leaving the emergency backup generator running (instead of in standby configuration) would have kept the LV bus energised after the first transformer tripped, and the engine coolant pumps and fuel line flushing pump would not have been lost.
- If the crew had opted to manually activate the second transformer within about half a minute (twice as fast as they reactivated the first one), and restarted the fuel line flushing pump, a second blackout would have been avoided, and the engine could have been restarted in time to steer away.
This shows the importance of leaving recovery systems armed and regularly training on power transfer procedures. It also illustrates why you shouldn't be running your main generators from a fuel pump which isn't designed for that task. This same pump setup was found on another ship they operated.
Ah so the crew modified the Generator to use the flush pumps instead? i really don't understand that. Why would using the flush pumps even be a viable alternative? were the normal pumps broken or was this just how the ship was built?
It saved them time on switching the fuel they were using. Within US waters they were required to either burn cleaner fuel or scrub the dirty (high-sulfur) diesel fuel they would use in open waters.
They didn't have a fuel scrubber and they didn't want to spend the time flushing the dirty fuel out of the fuel lines to switch to the clean fuel, so they bypassed the fuel lines and fuel pumps for generators 3 and 4 and used the fuel line flushing pump as a fuel pump to feed generators 3 and 4 with clean fuel (marine gas oil) instead.
They would then presumably start generator 1 and/or generator 2 once in open waters, feeding them with the regular, cheaper, dirtier diesel fuel, and shut down generators 3 and 4.
Bypassing the fuel lines and fuel pumps for generators 3 and 4 made them prone to the very failure they experienced.
The ship would not have been built this way; it wasn't up to code.
The crew weren't using the redundant fuel pumps. They were using the non-redundant fuel line flushing pump as a generator fuel pump, a task it was never designed for and which was not compliant.
That it doesn't restart on restoration of power is by design; you don't want to start flushing your fuel lines when the power returns because this could kill your generators and cause another blackout.
> Main engine shutting of (sic) when water pressure drops
Yeah, this is quite bad. There ought to be an override one can activate in an emergency in order to run the engines to the point of overheating, under the assumption that even destroying the engine will cause less catastrophic consequences than not having propulsion at the time.
> backup generator not even starting in time
There were 5 generators on board. Generators 1 through 4 are the main generators on the HV bus side, and the emergency backup generator is on the LV bus side.
When the incident occurred, the ship was being powered by generators 3 and 4, which were receiving their fuel via the non-redundant fuel line flushing pump. These generators powered the HV bus, which powered the LV bus via a transformer. The emergency backup generator was not running, so the LV bus was only receiving power from the HV bus via 1 transformer.
The incident tripped the circuit breaker for this transformer, disconnecting the HV bus from the LV bus, resulting in the first LV bus blackout. This resulted in main engine shutdown (coolant pump failure) and an automatic emergency backup generator startup.
There is an alternate (backup) set of circuit breakers and transformer that could have energised the LV bus, but the transformer switches were left in the manual position, so this failover did not happen automatically and immediately. There were no company procedures or regulations which required them to be left in the automatic position.
The LV bus also powered the fuel line flushing pump, so this pump failed. As a result, generators 3 and 4 started to fail (being supplied with fuel by a pump which was no longer operating). The electrical management system automatically commanded the start of generator 2 in response to the failing performance of generators 3 and 4.
Generator 1 and generator 2 were fed by the standard fuel pumps, which were available. One main generator is capable of powering the entire ship, so there was no need to start generator 1 as well; this would have just put more load on the HV bus (by having to run the fuel pump for generator 1 as well).
Instead of the automatic transformer failover (which was unavailable), the crew manually closed the same circuit breaker that had already tripped, 1 minute after the first LV bus blackout.
This restored power to the LV bus via the same transformer that was originally powering it, but did not restart the fuel line flushing pump supplying generators 3 and 4 (which were still running, but spinning down because they were being fed fuel via gravity only). This also restored full steering control, but this in itself was inadequate to control the vessel's course without the engine-driven propeller.
The main engine was still offline and takes upwards of half a minute to restart, assuming everyone is in place and ready to do so immediately, which was unlikely.
The emergency backup generator finally started 10 seconds later (25 seconds too late by requirements, 70 seconds after the first LV bus blackout).
Generator 2 had not yet gotten up to speed and connected to the HV bus before generators 3 and 4 disconnected (having exhausted the gravity-fed fuel in the line ahead of the inoperative fuel line flushing pump), resulting in an HV bus blackout and the second LV bus blackout. With only the emergency backup generator running on the LV side, only one-third of steering control was available, but again, this was inadequate without the engine.
3 seconds later, generator 2 connected to the HV bus. 26 seconds later, a crew member manually activated the alternate transformer, restoring power to the LV bus for the second time.
The collision was preventable:
- It is no longer a requirement that the engine automatically shuts down due to a loss of coolant pressure. It was at the time the vessel was constructed, but this was never re-evaluated. If it were, the system may have been tweaked to avoid losing the engine.
- If the transformer switches were left in the automatic position, the LV bus would have switched over to being powered by the second transformer automatically, and the engine coolant pumps and fuel line flushing pump would not have been lost.
- Leaving the emergency backup generator running (instead of in standby configuration) would have kept the LV bus energised after the first transformer tripped, and the engine coolant pumps and fuel line flushing pump would not have been lost.
- If the crew had opted to manually activate the second transformer within about half a minute (twice as fast as they reactivated the first one), and restarted the fuel line flushing pump, a second blackout would have been avoided, and the engine could have been restarted in time to steer away.
This shows the importance of leaving recovery systems armed and regularly training on power transfer procedures. It also illustrates why you shouldn't be running your main generators from a fuel pump which isn't designed for that task. This same pump setup was found on another ship they operated.