Why Thameslink's new trains can't cope with a power outage

Like many people in Mill Hill, I regularly use the the Thameslink service. Like many, I was massively impacted by the power supply problems on 9th August.  My wife even had to be evacuated from a train north of Kentish Town and walked along the railway lines. I am endebted to Robin Morel of Network Rail for sharing the report.  nationalgrideso.com/document/152351/download

Older trains were unaffected - Pic Copy Barneteye

The reason for the problem is that a 'Safety feature' of the new Thameslink 700 class trains means that if the power supply drops below 49Hz, the train will do a protective closedown. The drivers can do a manual reboot that takes ten minutes, but for 30 of the trains this didn't work, so technicians had to be sent out with a Laptop to restart the trains.

The section on planned mitigations is fascinating. It seems that Siemens, who manufacture the trains are now working on software to allow the train to restart itself automatically when the power returns to 49Hz. None of the older trains on the network were affected. One has to ask how they could not have considered that a power outage might occur and that needing an engineer with a laptop to access the train was not an acceptable or safe solution.

I am not an expert, but this is not only dangerous, but shows an almost criminal level of negligence in design by Siemens. Just suppose that my wife had been in a wheel chair or suffered from claustrophobia, as the power and air conditioning had shut down. Luckily her train was not too far from the station, but had it been in the middle of the mile long Belsize new tunnel, then that would have been terrifying.

Electric trains should be resilient and a driver should be trained to restart them. We need to know how long these changes will take to be implemented. We also need a robust recovery plan for such issues from GTR and Network Rail, in the event of such problems. Engineers need to live in the real world. It is all very well designing massively complicated trains, but if they are not resilient and predictable problems, that clearly can easily be resolved, cause the whole network to pack up, then it is clear that the whole design process is flawed. When the emergency cutoff was designed, someone should have said "what would happen if this train was packed and in a tunnel?".

Since the new class 700 trains were introduced, we've had air conditioning problems, ironing board seats and now we find out that if there is momentary power blip, half of them pack up completely. To me this is a sign of a very badly managed railway. Siemens should compensate commuters for the aggravation their bad design has caused. Maybe this would focus their minds when they come to designing their next generation of trains. The thing that irritates me beyond belief is that these issues always end up in a blame game. Obvious problems are excused as the contractors all start blaming each other, the government and the regulator. British Rail had its faults, but at least we all knew where the buck stopped.

Here is the full report.

Appendix F – Govia Thameslink Railway (GTR) technical report 
This Appendix contains the independent technical report provided to the ESO by GTR into the impact of the frequency disturbance on their trains on 09 August as part of the detailed investigation into the incident. Title Power Surge Disruption on 9th August 2019: Technical Review.

Date 3rd September 2019

Event Summary

1. From information supplied by the National Grid at 1653 on Friday the 9th of August the frequency of the OLE AC voltage supply dropped below 49Hz for 16 seconds. It dropped below 48.89Hz to a minimum of 48.8 Hz for milliseconds.
2. There was no identified OLE AC Voltage supply interruption.
3. All Desiro City class 700 and 717 units operating on AC Voltage suffered a Protective Shutdown where the converter, known as the 4QC (4 Quadrant Controller) shut down. None of the other AC trains in the GTR fleet suffered any power related issues from this event e.g. Class 387, Class 365, Class 313.
4. In Passenger Service this involved circa 60 units, consisting of Class 700 FLU/RLUs and Class 717, suffering a 4QC lock out.
5. The effect of the 4QC shutting down on the train is that the train switches to battery power which causes a loss of HVAC (fan only fresh air supply), stand by reduced lighting (emergency lighting is activated when battery voltage drops), no at seat power, no PIS displays (audio only from the cab), and no traction power. The GSM-R radio remains active.
6. Following failed attempts to repower the trains, the established first response for a train failing with these symptoms is for the Driver to perform a reboot of the train known as a Battery Reset. This takes approximately 10 minutes.
7. Fleet Control diagnosed the issue quickly and a global GSM-R call was broadcast to instruct drivers to carry out the Battery Reset process.
8. After this 27 of the affected units were recovered.
9. The remaining circa 30 units required the Protective Shutdown to be unlocked by the intervention of a Technician with a laptop attending each unit.
10. There were 17 Technicians available at the time of the event. These 17 were immediately sent to stranded units with laptops and a further 24 technicians were mobilized within the next hour. The trains affected were widely spread geographically and some were not easily accessible.
11.Therefore, this process took some time and resulted in 23 train evacuations and severe levels of service disruption.

Cause

1. The Desiro City from Siemens Mobility is the latest generation software enabled commuter train, so requiring protection against power supply frequency excursions for safety reasons and to protect low power electronics.
2. Siemens Technical Specification for the train states that the train will continue to operate with supply frequency drops down to 48.5Hz for short periods of time.
3. The NR Electrification System Compatibility document, NR/GN/ELP/27010(“Guidance for compatibility between electric trains and electrification systems”), a Manufacturing and Supply Agreement compliance requirement, identifies that the supply frequency can fall to 48.5Hz in extreme conditions.
4. A review of the Class 700 NoBo design submission (Siemens document A6Z00036309602 item 4.2.8.2.2) shows that compliance with both NR/GN/ELP 27010 and EN 50163 (“Railway Applications – Supply Voltages of Traction Systems”) would be demonstrated.
5. Following investigations, Siemens advised that the supply frequency response of the train was designed to comply with the EN 50163 Clause 4.2 Note 2. This note permits train drives to disconnected at 49Hz. Use of this supply frequency value in the train design led to the train protectively shutting down its drives when the supply frequency response fell below 49Hz.
6. Importantly Siemens have also clarified that there should not have been a Permanent Lockout on the train following a protective shutdown caused by a supply voltage frequency drop. All trains should have been recoverable via Battery Reset whereas 30 trains were not recoverable. This was not the intended behaviour of the train.
7. Therefore, the affected Class 700 and 717 sets did not react according to their design intent in these circumstances. The risk of this happening was not known prior to the power event on Friday 9 August.
8. Separately as a part of the new TCMS (Train Control Management System) software version 3.27, the ability for the driver to recover from a Permanent Lockout by using the Battery Reset process was removed.
9. On the 9th of August all the units which required a Technician to recover power were at software level 3.27 or above. The 28 units recovered by the driver performing a Battery Reset were at the previous TCMS software level of 3.25 or below.

Conclusion

1. All the Class 700 and Class 717 trains operating on AC suffered a Protective Shutdown of the 4QC controller because of a drop in the supply frequency below 49Hz for 16 seconds.
2. This was not how the train system had been specified to operate. This event should not have caused a Permanent Lockout fault on the trains.
3. The effects of this were exacerbated as the fleet was undergoing a software change, contained in this was a change in functionality removing the Battery Reset remedy for Permanent Lockout events. 4. This meant that the driver could not recover failed trains which were operating on the new software, instead a Technician was required to attend.

Planned Mitigation

1. Siemens are developing a software patch to allow units which protectively shutdown below 49Hz supply frequency to recover themselves without the need of a reboot or laptop when the frequency rises above 49.5Hz.
2. It is not proposed by Siemens or GTR to revert units to previous software versions as there are concerns this could severely impact unit availability. Based on discussions with the National Grid Head of Networks, the risk of frequency excursion dropping below 49Hz before the patch is fully introduced are considered extremely unlikely.
3. In addition to this Siemens will investigate how the train could be made to operate for a short time with supply frequency falling to 48.5Hz.