Update 24/09/22 13:00pm
Nasstar have provided the latest update, which seems very positive.
All services should now be restored. In the early hours of the morning [24/09/22] Nasstar engineers discovered a network problem. When isolating a circuit, all services tested clear, and the previously observed problems could not be reproduced.
This circuit has been forced out of service so that all services remain up. There should be no impact to service over the weekend, as removing this circuit relates only to capacity, for which requirements are low at the weekend.
Nasstar engineers have been mobilised to both end of the circuit in question and the 3rd part circuit provided is also on route to site to conduct a full diagnostic and troubleshooting on the circuit.
We will keep you updated on the progress via this channel.
Update 24/09/22 08:00am
Please be assured, Nasstar have lined up full technical and incident management resources to continue to work on this into the weekend if the issue persists.
Early indications from some users as of 07.30 this morning is that services do seem to be coming back up, however, there is no formal confirmation yet from Nasstar and they have indicated service me come and go throughout the day as they continue to work.
Nasstar are continuing to work on the fault currently and as such services may restore and drop again until we’ve confirmed the fix.
Impact: Users across multiple sites on the emPSN network are unable to use key internet services such as browsing and IP-based telephony. Users with explicit proxy are also affected but much more limited numbers.
Priority: P1
Background: Multiple sites across Nottinghamshire, Derbyshire, Lincolnshire, and Leicestershire reporting intermittent internet connection.
Current Status:
- Symptoms persist to affect users
- Troubleshooting continuing
- Engineers investigating circuit issues – circuit is up with reduced capacity
Activities completed include:
- IOS update to firewalls
- Reload of MPLS core routers
- Failover of supervisor cards on core switches to make passive card active
- Wireshark traces analysed
- JANET analysed trace routes and have checked with their security mitigation
- HSRP
- Port Channels disabled to adjust egress and ingress routes
- NAT rules investigated whilst deploying attempted workaround
- JANET tech Support and management are assisting the technical bridge with troubleshooting
- Internet traffic has been forced over a single 10GB link to improve service as a temporary workaround
- Supervisor card re-seated in Node4
- Supervisor card re-seated in Glaisdale
- The iOS upgrade of the Core switches completed
Official Statement from NASSTAR
23/09/2022 14:30pm
Since Tuesday lunchtime some services provided by Nasstar to emPSN have being experiencing loss of service. Detailed troubleshooting work is still on-going with the direct support of equipment manufacturers and other third parties that provide parts of the service, to ensure all avenues are explored. Intensive works are continuing with a high volume of highly skilled resources deployed around the clock, all focused on full restoration of services to emPSN and its customers.
We hope to be able to report a positive outcome soon. Please accept our sincere apologies for this disruption.
Update 23/09/22 13.00pm
Unfortunately, the P1 Incident still remains ongoing, with Nasstar technical teams and their external support partners working on a resolution.
Currently, we are still unable to offer a timeframe for resolution.
Impact: Users across multiple sites on the emPSN network are unable to use key internet services such as browsing and IP-based telephony. Users with explicit proxy are also affected but much more limited numbers.
Priority: P1
Background: Multiple sites across Nottinghamshire, Derbyshire and Lincolnshire reported intermittent internet connection.
Current Status:
- Symptoms persist to affect users
- Cisco TAC continue to work on this with our engineers Cisco TAC R&S have re-joined the investigation and are taking captures
Activities completed include:
- IOS update to firewalls
- Reload of MPLS core routers
- Failover of supervisor cards on core switches to make passive card active.
- Wireshark traces analysed
- JANET analysed trace routes and have checked with their security mitigation
- HSRP
- Port Channels disabled to adjust egress and ingress routes
- NAT rules investigated whilst deploying attempted workaround
- JANET tech Support and management are assisting the technical bridge with troubleshooting
- Internet traffic has been forced over a single 10GB link to improve service as a temporary workaround.
Further updates to follow.
Update 23/09/22 8:30am
Nasstar latest P1 update having implemented a workaround however at this time it appears that this has not resolved the issue. Further updates will follow throughout the day.
Impact: Users across multiple sites on the emPSN network are unable to use key internet services such as browsing and IP-based telephony. Users with explicit proxy are also affected but much more limited numbers.
Priority: P1
Background: Multiple sites across Nottinghamshire, Derbyshire and Lincolnshire reported intermittent internet connection.
Current Status:
- Symptoms persist to affect users
- Cisco TAC team are connected to devices and continues to work with Nasstar technical team.
Activities completed include:
- IOS update to firewalls
- Reload of MPLS core routers
- Failover of supervisor cards on core switches to make passive card active.
- Wireshark traces analysed
- JANET analysed trace routes and have checked with their security mitigation
- HSRP
- Port Channels disabled to adjust egress and ingress routes
- NAT rules investigated whilst deploying attempted workaround
- JANET tech Support and management are assisting the technical bridge with troubleshooting
- Internet traffic has been forced over a single 10GB link to improve service as a temporary workaround.
Currently, Nasstar are unable to provide an incident resolution timescale.
Update 22/09/22 16:00pm
Nasstar has provided their latest incident update and has indicated that they may have identified a failure – Further update to follow on this.
Impact: Users across multiple sites on the emPSN network are unable to use key internet services such as browsing and IP-based telephony. Users with explicit proxy access are unaffected
Priority: P1
Background: Multiple sites across Nottinghamshire, Derbyshire and Lincolnshire reported intermittent internet connection.
Current Status:
- Cisco TAC and all Nasstar technical parties including remote Data Centre Support continue to work through the issue.
- Nasstar remote engineers are still consoled in directly and testing from various network nodes.
- Nasstar are continuing to rule out different parts of the networks.
- Nasstar are making progress and may have identified a failure in their latest testing step.
Activities completed include:
- IOS update to firewalls
- Reload of MPLS core routers
- Failover of supervisor cards on core switches to make passive card active.
- Hardware traces analysed
- Janet analysed trace routes and have checked with their security mitigation
Currently, Nasstar are unable to provide an incident resolution timescale.
Update 22/09/22 11:55am
Nasstar continues to investigate the incident and has provided the following status, including tasks currently ongoing and completed.
At this time, Nasstar cannot offer a resolution timeframe.
Impact: Users across multiple sites on the emPSN network are unable to use key internet services such as browsing and IP-based telephony. Users with explicit proxy access are unaffected
Priority: P1
Background: Multiple sites across Nottinghamshire, Derbyshire and Lincolnshire reported intermittent internet connection.
Current Status:
- Support Partner Cisco TAC and all Nasstar technical parties including remote Data Centre Support continue to work through the issue.
- Network Traces are being analysed by TAC teams.
- Remote Nasstar engineers are testing from various network nodes.
Activities completed include:
- IOS update to Nasstar Core firewalls
- Reload of Nasstar Core routers
- Failover of Cards on Nasstar Core switches to make passive card active.
More updates to follow.
Update 22/09/22 11:00am
Nasstar investigations are ongoing
Support Partner’s Cisco TAC, Nasstar engineers and all Technical TAC teams continue to work on the incident.
Currently sat at the highest escalation within our service partners.
More updates to follow.
Update 22/09/22 08:00am
The incident remains ongoing with technical teams still fully engaged.
Focus remains on the Centralised Core Firewalls within the emPSN Datacentres.
Current Status:
- Our support partner Cisco has verified the firewalls are now stable and require testing from an end-user perspective to determine the current user experience.
- The resolving team are still actively investigating the network to ensure correct behaviour.
Next Status:
- Continuing to monitor firewall performance.
- Nasstar TAC engineers are reviewing the core network and routing in parallel
Updates to follow as the incident progresses.
Update 18:10pm
A Technical bridge remains in place for Nasstar TAC teams, with Nasstar engineers in our DC’s supporting.
Nasstar has loaded new IOS onto both our Core resilient Cisco FW’s and completed a hard reboot as part of a Cisco recommendation.
Nasstar continues to investigate, with school IT staff on-site supporting testing.
Update 14:45pm
Technical teams continue to investigate the ongoing incident.
Unfortunately at this time we are still unable to provide a timeframe for resolution.
Nasstar will continue to work throughout the evening to provide a fix for this issue.
Updates to follow as the incident progresses.
Update 11:40am
The issue is unfortunately still ongoing, with Nasstar technical teams continuing to try and resolve the incident as a matter of urgency.
Updates to follow as the incident progresses.
emPSN currently has a Major incident ongoing, impacting sites within the emPSN network.
Engineers continue to investigate.
Currently, there is no restoration timescale, with the next update due at 11.00am.
emPSN apologise for the issues this is causing customers.
