Rackspace hosted Exchange suffered a disastrous interruption starting December 2, 2022 and is still continuous since 12:37 AM December fourth. Initially referred to as connectivity and login concerns, the guidance was eventually upgraded to reveal that they were dealing with a security event.
Rackspace Hosted Exchange Issues
The Rackspace system decreased in the early morning hours of December 2, 2022. At first there was no word from Rackspace about what the problem was, much less an ETA of when it would be fixed.
Customers on Buy Twitter Verification reported that Rackspace was not reacting to support emails.
This has been rather the day with #Rackspace. Every hosted exchange customer has actually been down for 14 hours approximately. Assistance isn’t reading/responding to tickets. Updates are unhelpful.
I am worried now that they fell victim to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace consumer privately messaged me over social media on Friday to relate their experience:
“All hosted Exchange clients down over the previous 16 hours.
Uncertain the number of business that is, however it’s substantial.
They’re serving a 554 long delay bounce so people emailing in aren’t familiar with the bounce for several hours.”
The main Rackspace status page provided a running update of the failure but the initial posts had no information besides there was a failure and it was being examined.
The very first official update was on December 2nd at 2:49 AM:
“We are investigating a concern that is affecting our Hosted Exchange environments. More details will be posted as they appear.”
Thirteen minutes later on Rackspace started calling it a “connection issue.”
“We are examining reports of connectivity problems to our Exchange environments.
Users might experience a mistake upon accessing the Outlook Web App (Webmail) and syncing their email customer(s).”
By 6:36 AM the Rackspace updates explained the continuous problem as “connection and login issues” then later on that afternoon at 1:54 PM Rackspace announced they were still in the “examination phase” of the interruption, still trying to find out what failed.
And they were still calling it “connection and login issues” in their Cloud Office environments at 4:51 PM that afternoon.
Rackspace Recommends Moving to Microsoft 365
4 hours later on Rackspace referred to the situation as a “considerable failure”and started providing their consumers complimentary Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround until they understood the problem and might bring the system back online.
The official assistance mentioned:
“We experienced a significant failure in our Hosted Exchange environment. We proactively closed down the environment to prevent any additional problems while we continue work to restore service. As we continue to overcome the origin of the problem, we have an alternate service that will re-activate your capability to send and receive emails.
At no cost to you, we will be providing you access to Microsoft Exchange Strategy 1 licenses on Microsoft 365 up until additional notification.”
Rackspace Hosted Exchange Security Occurrence
It was not until almost 24 hours later at 1:57 AM on December 3rd that Rackspace formally announced that their hosted Exchange service was experiencing a security event.
The statement further revealed that the Rackspace professionals had powered down and disconnected the Exchange environment.
“After additional analysis, we have identified that this is a security incident.
The recognized effect is separated to a part of our Hosted Exchange platform. We are taking required actions to examine and safeguard our environments.”
Twelve hours later that afternoon they updated the status page with more details that their security group and outside professionals were still dealing with fixing the failure.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has actually not launched details of the security occasion.
A security event normally includes a vulnerability and there are two serious vulnerabilities presently in the wile that were patched in November 2022.
These are the 2 most existing vulnerabilities:
Microsoft Exchange Server Server-Side Request Forgery (SSRF) Vulnerability
A Server Side Request Forgery (SSRF) attack enables a hacker to check out and change data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an enemy is able to run destructive code on a server.
An advisory released in October 2022 explained the effect of the vulnerabilities:
“A validated remote attacker can perform SSRF attacks to escalate privileges and carry out arbtirary PowerShell code on vulnerable Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mailbox server, the enemy can possibly access to other resources by means of lateral movement into Exchange and Active Directory site environments.”
The Rackspace failure updates have actually not indicated what the specific problem was, only that it was a security incident.
The most current status update as of December fourth specified that the service is still down and consumers are motivated to move to the Microsoft 365 service.
Rackspace posted the following on December 4, 2022 at 12:37 AM:
“We continue to make development in dealing with the incident. The schedule of your service and security of your data is of high value.
We have actually devoted extensive internal resources and engaged first-rate external competence in our efforts to decrease unfavorable impacts to customers.”
It’s possible that the above noted vulnerabilities relate to the security incident affecting the Rackspace Hosted Exchange service.
There has been no statement of whether client details has been compromised. This occasion is still ongoing.
Included image by Best SMM Panel/Orn Rin