January 15th, 2018 The DataBank teams have been hard at work this past week testing and addressing the Meltdown and Spectre vulnerabilities. If you missed our previous communication that described the situation and status, please click here for some valuable background information sent on Tuesday, January 9th at 11:48am. This communication is intended to update you on our plans for remediation and mitigation of this set of vulnerabilities.
As of January 15, there are still no known vulnerabilities identified by government or private security experts. There has been talk in some media about a java script related proof-of-concept (POC) exploit and some anti-virus vendors have created signatures to detect this item. The media reports have done a good job reporting that this exists, but they have not clearly identified that this POC was developed by the academic teams that identified the vulnerability and has been closely guarded. Security experts do not consider this POC to be a credible threat and neither do we at this time. Security vendors and government agencies continue to classify the overall situation to be a moderate to low-moderate threat. This is because 1) there is no known or available exploit and 2) exploitation would require extensive, localized access to a server or system. At DataBank / Edge Hosting, a DataBank company, our security systems and functions are designed towards a multi-layered security approach, which means that an attacker would have to break through multiple barriers to be successful in an attack. This multi-layered security model further mitigates the possibility that Meltdown or Spectre will impact your systems.
This past week (January 5-12) DataBank Network, System, and Security Engineers have accomplished and continue to work towards evaluating, testing and implementing patches and countermeasures deployments. We have:
- Updated the Intrusion Prevention System (IPS) digital vaccines to detect the POC exploit and other indicators of an attempted attack.
- Updated anti-virus signatures to detect and thwart potential attackers.
- Determined that firewalls (Cisco ASA and ASAv) used by our managed services customers are not impacted by the vulnerability. If you are a customer that manages your own firewall, you should review your equipment with the supplier/vendor. You can also submit a ticket for assistance in determining your exposure.
- Developed a plan of action for patching underlying virtual machines, routers, switches and other networking devices that are impacted.
- We have begun the deployment of patches to some underlying virtual hypervisors.
This will now allow for the successful patching of operating systems. The coming week will see continued patching of underlying hypervisors. Based upon vendor recommendations and our own testing, some patches must be deployed in sequence in order for later patches to be effective. We have determined that a five-tier approach is necessary to ensure that patches occur in sequence. Individual communications will be going out to customers when a pending tier will impact them. Communications will come via the ticketing system in the same manner as other patch notifications. In order to minimize downtime and ensure that the right patches are applied in the sequence specified by the vendor, patching is going to take some time to accomplish. We will continue to update you each week with the best and most accurate information we have. If you have specific questions, please utilize out ticketing system to submit them. We will address any and all questions you have.
January 9, 2018 This past week you may have heard of two major vulnerabilities, Meltdown and Spectre, that impact the large majority of computer processors including Intel, AMD, and ARM. These vulnerabilities manipulate widely adopted performance enhancing features built into most modern CPUs. This could allow an attacker that already has compromised a system to access contents of memory that would normally be hidden.
As of this date, no known exploits have been identified by any security experts and a successful attack requires privileged access to the system with rights to run applications. Even with these mitigating factors the DataBank team is proactively taking steps to address this issue.
Vendors are working to, or have already delivered, patches for operating systems and hypervisors. We are currently evaluating the delivered patches in our test lab environments while concurrently identifying potentially affected systems.
For Colocation Customers: We suggest you evaluate the patch in your test lab environment. There are reports that application of the patch could impact CPU performance 5-30%. The impact on performance has been reported on database and application servers during peak load.
For IaaS Customers: We suggest you evaluate the patch in your test lab environment. You should apply patches to your servers after validation and testing. We will be applying patches to the underlying routers, switches, and VMware hypervisors. You will be notified if your services may be impacted in any way.
For PaaS and Managed Hosting Customers: We will initially focus on patching routers, switches, and hypervisors and lastly surrounding services systems such as load balancers and firewalls. We will schedule maintenance windows where applicable. Each maintenance window will be communicated in advance and include potential impact expectations and durations. Due to the lack of an exploit in combination with reports that installing patches has the potential to slow performance from 5-30%, we have opted to not push the patches out-of-band and ahead of schedule. Customers who utilize managed patching services for their operating system and application patches should see no changes to their existing patch window(s) for the compute layer of the environment. Operating system patches will be applied as per your current schedule. If you desire to advance the patching schedule, you may do so through a request ticket to our support teams.
DataBank runs mission-critical functions for our customers, and our team will continue to work diligently to maintain the highest level of security and uptime. Our defense-in-depth approach involves multiple levels of security that will protect your data even if one layer were to be breached via a vulnerability. For more information on Meltdown and Spectre, we recommend reading the following link posted by 451 Research on January 5th. We will continue to update our customers on steps that we are taking to address this situation. We encourage our customers to review details about this vulnerability from the U.S. Computer Emergency Response Team (US-CERT) at https://www.us-cert.gov/. Further reading and academic papers on the subject can be found at https://meltdownattack.com/. We will continue to communicate with impacted customers as needed and provide the consultation and guidance necessary to see us all through this challenging effort.
Dallas, TX – January 5, 2018 – Databank / Edge Hosting a DataBank company is aware of the ‘Meltdown’ and ‘Spectre’ vulnerabilities that have been impacting virtually every computer processor that is currently available from Intel, AMD and ARM. We have been closely monitoring the rapidly developing situation, communicating with our vendors and developing plans to address the situation. We take this vulnerability seriously and will continue to monitor for changes.
At this time the U.S. Computer Emergency Response Team (US-CERT) has stated that they are “not aware of any active exploitation at this time.” (Jan 5, 2018 15:49). Further, the National Cyber Security Centre (UK) has identified that there is “no evidence of any malicious exploitation and patches are being produced for the major platforms.” (Jan 5, 2018 15:52) If the vulnerability were to be exploited, an attacker would need to have access to and run code on the impacted device. Customers should ensure that their passwords are secure and users pay attention for phishing attacks that may be used to gain access to systems.
Databank / Edge Hosting, a DataBank company, will continue to update our customers on steps that we are taking to address this situation. We encourage our customers to review details about this vulnerability from the U.S. Computer Emergency Response Team (US CERT) at https://www.us-cert.gov/. Further reading and academic papers on the subject can be found at https://meltdownattack.com/.
Databank is focused on delivering the best solutions and support for our customers. This situation is unprecedented in the breadth and depth of impact that it has across the technology landscape. Solving the problem takes ingenuity and perseverance at all levels. The DataBank team has worked with vendors, customers and partners to engineer, develop and implement a plan to overcome this challenge. We will continue to communicate with impacted customers as needed and provide the consultation and guidance necessary to see us all through this challenging effort.
About DataBank
DataBank is a leading provider of secure enterprise-class data center, cloud, and interconnection services, offering customers 100% uptime availability of data, applications and infrastructure. Databank’s managed data center services are anchored in world-class facilities. Our customized technology solutions help customers effectively manage risk, improve their technology performance and allow them to focus on their core business objectives. DataBank is headquartered in the historic former Federal Reserve Bank Building, in downtown Dallas, Texas. For additional information on DataBank locations and services, please visit http://www.databank.com or call 1(800) 840-7533.
About Digital Bridge Holdings
Founded in 2013 by Marc C. Ganzi and Ben Jenkins, Digital Bridge is focused on the ownership, investment and active management of companies in the mobile and internet infrastructure sector. Since inception, Digital Bridge has raised over $6.5 billion of debt and equity capital used to acquire and invest in the development of communications infrastructure businesses, including DataBank, Vantage Data Centers, ExteNet, Vertical Bridge, Andean Tower Partners, and Mexico Tower Partners. ###