We've observed some of our monitoring SIMs go offline.
This constituted a major incident affecting our mobile carrier and their telemetry/messaging partner. Though the incident response is now over, and services have restored, we await a root cause analysis and "reason for outage" document from them.
The carrier has updated the closed incident with the following notes:
Event start date and time: 2025-01-10 14:30 CET (12:30 UTC) Event end date and time: 2025-01-10 22:19 CET (20:19 UTC)
Our monitoring and signaling has returned to the expected values, network tests pass and subscribers can fully use all services.
The cause of the issue is currently under investigation.
The carrier has closed the incident with the following notes:
We are observing full service restoration and error clearing since 22:19 CET (20:19 UTC). Our monitoring tools indicate signaling levels have returned back to expected values. Customers can use all available services in full once again.
Our team will continue to monitor the situation carefully and with the highest priority.
Latest carrier update:
We are observing full service restoration and error clearing since 20:19 UTC. Our monitoring tools indicate signaling levels have returned back to expected values. Customers can use all available services in full once again.
Our team will continue to monitor the situation carefully.
Latest carrier update:
Our signaling vendor has completed the unblocking of all Swedish SIM ranges and is currently working on restoring full LTE service availability. From initial observations in our monitoring tools the recovery is running successfully and LTE traffic is gradually coming back.
The resolution is solely dependent on the device behaviour, therefore an ETA is not available.
Latest carrier update:
Our signaling vendor is nearing the completion of Swedish SIM range unblocking and will afterwards switch focus to recovering the full LTE service availability. We are currently observing success rate for 2g/3G network registration at 100%, 98% success rate on 2g/3g/4g data session requests and 98% success rate for regular voice.
Due to the complex nature of this process an ETA for the completion of it is still unavailable.
Latest carrier update:
An additional batch of Swedish SIM ranges have been unblocked by our signaling vendor for traffic and further unblocking is ongoing. We are observing link stability and gradual improvements in our monitoring graphs. LTE services is recovering slower due to the complexity of the recovery mechanism, therefore it will take additional time until they are fully operational.
An ETA of for the completion of the unblocking is still undetermined.
Latest carrier update:
Our signaling vendor has unblocked traffic for five of the Swedish SIM ranges and continuing with further unblocking.
Our monitoring tools indicate link stability, safe for further range unblocking.
Since the speed of the process depends on device behavior after the unblocking, an ETA for the completion of this task cant be determined at this time.
Latest carrier update:
Our signaling vendor has completed unblocking the Estonian SIM range traffic, and we are observing device recovery.
The unblocking of Swedish SIM range traffic is currently in progress.
An ETA for the completion of the unblocking process remains under analysis, as the speed of the process depends on device behavior.
Latest carrier update:
The Core Team has implemented the changes.
The vendor and the responsible internal team have stabilized the signaling links and are now unblocking the traffic. Together, they are gradually unblocking the affected ranges.
An ETA for the completion of the unblocking process is still being analyzed, as the speed of unblock is dependent on device behavior.
All of our "canaries" are alive again. SIM session connection rates appear back to normal.
But we think there is probably a "thundering herd" of SIMs trying to reestablish their sessions, and this might be log-jamming at the carrier.
Latest carrier update:
Internal Core Team is making changes to the indentified cause. Meanwhile representatives from our vendor and responsible internal team are stabilizing the signaling links to be ready to accept the currently still blocked traffic. These checks are necessary to avoid overloading both vendor and carrier network infrastructure. An ETA for the unblocking is currently unavailable.
We're seeing some of our "canaries" come back online, and are observing more SIM session attempts successfully complete now.
Latest carrier update:
The possible root cause of the issue has been identified and we are currently monitoring the signaling links to stabilize enough before enabling the traffic back. Once links are deemed stable enough, traffic will be unblocked back.
Latest carrier update:
Our signaling carrier has blocked part of the traffic that is sent to stabilize the signaling links.
Involved teams are currently monitoring the link stability for them to be stable enough for the traffic to be enabled back.
The exact root cause is still under investigation.
Latest carrier update:
The cause of general service degradation is still being investigated together with our vendors and signaling providers.
We are observing network authentication issues, as well as SMS and Voice service unavailability, however, once devices are registered 2G/3G/4G data sessions are not affected.
The root cause of the issue is still undetermined.
Latest carrier update:
We are continuing to investigate the cause of general service degradation. In our monitoring, we observe a huge increase in signaling.
2/3G is affected partially and can take time to attach to the network.
All responsible teams are involved in investigating root causes and working on solutions.
Initially we believed these were related to a problem with the Vodafone UK network (which is having some roaming issues), but we've just had a wider alert come through from the carrier:
Monitoring tools and customer reports are indicating general service degradation. The information is still being analyzed to understand the root cause and exact impact on customer devices.