Recent Facebook & Data Exposure Teaches Data Responsibility Lesson
You might’ve heard that in early April, it was announced that over 500 million Facebook records were found on two exposed AWS servers earlier this year. The user data was discovered by Upguard, a cybersecurity firm dedicated to identifying data leaks (ZDNet). The data, purchased from Facebook by two third-party app development companies, was stored on Amazon servers such that it was possible for the public to download it.
This scenario is nothing new for Facebook, but here’s what interesting: after the exposure was announced, stories began circulating throughout the media, publicly identifying AWS as responsible for the exposed data. No one can say for sure what misinformation spurred the slew of commentary, but one thing’s for certain—both third-party companies who purchased the data from Facebook were responsible for securing the data and ultimately Facebook is responsible for how its data is maintained; AWS likely held little or no responsibility for data storage. The narrative was odd to begin with, and after its inception, AWS didn’t make any public statements to refute the claims. It’s possible that this was because the stories were simply broadly circulated misinformation, thus there was no real need to make a statement. Whatever the reason, the story blew over, quickly superseded by other information security news events that arose shortly after.
But there’s a lesson to be learned: when a company places their data in a public cloud environment, it’s incumbent upon the data owner or data steward to ensure the operating system is hardened, the database, storage devices, and any transmission mechanism has encryption, and access controls are in place. The truth is, both Facebook and their business customers failed to secure their environments. Though the media reports laid blame on AWS, each of the app development companies placed the data on the AWS servers, and each are responsible for securing it.
“What a lot of people don’t understand about cloud service providers, especially at AWS, is that they’ll give you the platform or operating system to place your data and applications on, but the company that’s contracted for that service is responsible for the implementation of most of the security controls. At minimum, a cloud customer should conduct due diligence to ensure that stated security controls are in fact in place. Cloud customers must understand that, as Data Owners, they are ultimately 100% responsible for the data and that can never be transferred to a cloud provider .”
-Mark A. Houpt, Chief Information Security Officer, DataBank
What Are Data Owners Obligations to Exposed Records?
Curiously, AWS was accused of failing to do their job even though they’re likely not ultimately responsible for the data that’s placed onto their servers. We’ve established that it was the responsibility of the Cloud Service Provider customers, but where does the Data Owner come in?
For starters, Data Owners do legitimately share data with third-party customers. It’s a standard practice, and legally allowed in the United States, for Data Owners. In Europe and other locations this is not always permitted. This data sharing practice is a large contributor to a Data Owners revenue stream. However, it would behoove a Data Owner to acknowledge their ownership and to ensure that third-party, sharing customers like app developers and marketing agents are maintaining secure environments where the data is processes. In many cases, large enterprises with large resources will conduct third-party vendor security programs that include testing and vendor risk analysis to help customers secure the data. Smaller and medium sized businesses have less resources but should still conduct a security program over vendors. Data owners have a legal and ethical obligation to continue to ensure shared data is secured through its entire lifecycle, regardless of transfer or location, after it’s provided to customers.
“It is best practice in the Information Security industry to be doing a trust but verify type of situation, such that they either insist on third-party reports, hire third parties to conduct security testing and analysis, or use an internal security team to conduct testing on customers to make sure that data is being secured.”
-Mark Houpt, Chief Information Security Officer, DataBank
A Message Made Clear: Cloud Providers Never Take Responsibility for Data.
It could be argued that much of the problem is that customers using public cloud resources have a notorious tendency to assume that the risk and ownership of data is transferred to the Cloud Provider and education needs to take place to dispel these notions. False assumptions on this topic continue to purvey among enterprises across all industries, and there aren’t enough information security experts stepping forward to correct the narrative.
Unfortunately, it’s a common occurrence for any public cloud service subscriber to assume that just because they’ve put their data in an environment, the cloud provider takes responsibility for anything that occurs within their environment, and this is simply not the case.
Upon taking a closer look at the fine print, it is clear that the data owner is always the entity from which the data originated. Data processors, such as hosting providers, do not bear ownership, and never assume ownership responsibility or transfer risk or liability of data.
Dialing Down Security Fears with Transparent Cloud Hosting Providers
Let’s talk about how we handle the proverbial line in the sand when it comes to data here at DataBank.
One of the first things we do with our customers is clearly communicate that DataBank never assumes ownership for customer data. The ownership always remains with the customer. It’s our principle to be transparent in our methodology for securing our environment and systems, as well as what customers are responsible for within their own environments. We provide an implementation summary, as well as a detailed customer responsibility matrix to ensure that responsibilities, as well as the format in which data is secured, is clear.
Public cloud providers are useful for particular use cases, but as customers, it’s important to recognize that most large-scale CSPs only do so much in the way of security. Comparatively, at DataBank we go to the extent of employing NIST 800-53R4 controls to ensure strong levels of protection. Within a Managed Services PaaS environment where a customer has selected all security tools and features offered by DataBank, there are 325 different security controls DataBank helps maintain. Eighty-five percent (85%) are of the IaaS and PaaS controls are managed by DataBank on behalf of the customer, who is responsible for the remaining 15%. We also clearly identify and provide consultative guidance on how to apply controls to your application(s) and data.
This makes DataBank’s CloudPlus an arguably more secure environment for enterprises. Here’s what seems to trip people up: although security-focused cloud providers, such as DataBank, will manage a certain amount of controls, the customer remains responsible for conducting a security analysis to verify that the CSP is in fact delivering on their commitment. To assist with this, DataBank provides reports so that customers can trust and verify that the other 85% of controls are in fact in place, accurate, and functional.
It’s critical to note: this level of security is achieved through PaaS managed security. In selecting the IaaS managed security model, for example, DataBank would manage significantly less security controls as a result of the customer choosing to take on responsibility for a greater portion of the environment. Colocation infrastructure comes with its own responsibility delegation, as well. The environment you choose affects your level of responsibility.
If this publicized data exposure triangle teaches us nothing else about information security, remember the following:
Any customer who places data in a public cloud environment remains the Data Owner is responsible for that data.
Trust, but verify all of your vendors and business partners. In security terms, this essentially means continuous monitoring. DataBank, for example, provides customers a report that confirms a particular security control is not only in effect, but operational. Additionally, the DataBank portal allows customers to log in and visualize their intrusion protection system, its data, where attack items are being blocked, and more. DataBank is committed to our security responsibilities, but transparency is offered to customers to provide the wherewithal not only to trust, but to verify.
There are options with respect to which cloud provider you choose for security support. Some do more, some do less. Choose wisely.
Logix Fiber Networks, the largest independent fiber network provider in Texas, has now extended its fiber network to DataBank’s DFW3 facility, bringing Dallas’s premier data center one step closer to thousands of Texas enterprises.
LOGIX has more than 6,200 route miles of network spanning 225,000 fiber miles, connecting more than 2000 office buildings, 100+ data centers, and 12,000 potential enterprise-class customers.
With the build into DFW3, Logix will now have a presence at all three of Databank’s Dallas-area data centers, giving customers on-net access to all of DataBank’s colocation, cloud and managed services capabilities. In DFW1 and DFW3, Logix is in the MeetMe room. In DFW2, Logix can a fiber drop on the campus.
Hyperscale and enterprises will now have access to low-latency, layer 2 or wave connectivity between their Dallas offices and all of DataBank’s facilities in the area, making it an ideal place to build, scale and collocate the most demanding applications or establish robust cloud connectivity strategies.
If you visit the DFW3 location, be sure to pick up some literature on Logix and learn more.
DataBank and Georgia Tech Progress with Purpose-Built Data Center in Tech Square
The DataBank ATL1 data center is not your average data center.
When Georgia Tech set out to create a high-performance computing center for their institution, they turned to DataBank to build a data center environment capable ofmeetingthe performance needs of the center, and overcoming the inherent challengesthat often accompany HPC initiatives.
DataBank’sATL1 facility will be located in the new CODA building of Georgia Tech’s Technology Square, smack dab in the middle of the Atlanta’s tech hub, where academia meets innovation, research, and fortune enterprises. ATL1will not only be different from any other DataBank facility, it will be will be one of the most advanced data centers in the country.
Right out of the gate, ATL1 will possesstwo things unique to any data center in the region:
1. Southern Crossroads (SoX)
SoXserves as the Southeast connector to National Lambda Rail (NLR), Internet2 and other major U.S as well as International research networks. It connects southern schools, including Mississippi, Alabama, Georgia, Florida, and South & North Carolina. SoX is a special network fabric that privately interconnects many different schools and federal institutions.
2. The Georgia Tech Supercomputer
Georgia Tech was awarded $3.7 million from the National Science Foundation to cover 70% of the costs of a new, state-of-the-art high–performance computing resource for the CODA building’s data center, ATL1.The new HPC systemwill support data-driven research in astrophysics, computational biology, health sciences, computational chemistry, materials and manufacturing, and more. It will also be used to research energy efficiency and performance of HPC systems themselves.
“ATL1 is an ecosystem. A social compute family where we’re bringing in any company that wants to be a part of it. Anyone who wants to witness a supercomputerand real innovation is welcome.”
–Brandon Peccoralo, General Manager at DataBank
The Green Initiative and a New Generation of Data Center Cooling
But, there’s something else that makes ATL1 quite different: its cooling technique.
For the past thirty years, the data center industry has largely relied upon the same cooling techniques and architectures and, with minor exceptions, cooling has not been a subject of great innovation. In ATL1, DataBank, pushes the envelope with its new ColdLogikDx Rear Door cooling solution from QCooling.
Data centers focus tremendous amounts of time seeking the most efficient use of power. The biggest factor in power consumption is one that data centers have little control over: a customers’ hardware. But the second biggest factor in power consumption isPUE –an industry-standard measurement of how efficiently a data center cools the waste energy coming off computers.This is an area data centers have lots of room to innovate and find improvements.
Traditionally, this has meant taking energy from the utility, and running AC units within the data center as efficiently as possible to cool the overall environment. When you stop and think about it, this approach is comically inefficient. You are basically taking the heat that is generated from consumption of energy (in the form of computers) and consuming even more energy to cool it (in the form of AC).
QCooling’s solution for ATL1 adds a unique approach. Waste energythat comes out of a server in a cabinet is exchanged by heating naturally cold water that runs through a closed-loop system in each cabinet rear door. The water is heated, and then rejected out of the building as new naturally cold water is cycled back in to replace it keeping the cabinet continuously cool.
That system alone would represent a major innovation in data center cooling But, in our mission to continuously evolve the data center experience, DataBank took the implementation one step further.
In the typical data center environment, any heat removed with CRAC units is usually sent back to the central plant, exchanged for condenser water, and receded off the roof via typical heat rejection. Instead of wasting the heat, the ATL1 facility is actually sending it over to the CODA building’s high-rise boilers and allowing tenantsto reuse and repurpose it to heat their offices in colder weather, thus further offsetting energy use.
Using QCooling’sColdLogik rear door coolers, DataBank and Georgia Tech are cooling 50 kW per enclosure, per rack, using 73-degree warm water. And this rear door cooling system can use that same capacity to cool up to 100 kW per rack with just minor changes in infrastructure.
Cooling 50 kW with 73 degrees, with the potential to double
90% less energy consumption
80% more real estate than traditional cooling
Cool, right?
“Being able to cool 50 kW with 73 degree, and able to go 100 kW in the same footprint? That’s unheard of.”
On June 11, 2019, Adobe released patches for ColdFusion 11, ColdFusion 2016, and ColdFusion 2018 that addresses a critical vulnerability in these versions that allow for arbitrary code execution. More information can be can found at the following link: https://helpx.adobe.com/security/products/coldfusion/apsb19-27.html
Adobe has labeled this patch as a priority 2 patch.
Is my server affected?
If your server is running ColdFusion 2018, ColdFusion 2016, or ColdFusion 11, your server may be affected. To check, log in to your ColdFusion admin for each instance and navigate to Server Update > Updates. From there, click Check for Updatesand check your version against the affected version table below:
Product
Affected Versions
Platform
ColdFusion 2018
Update 3 and earlier versions
All
ColdFusion 2016
Update 10 and earlier versions
All
ColdFusion 11
Update 18 and earlier versions
All
My server is affected. What needs to be done to mitigate the vulnerability?
First, it is recommended to read the release notes as ColdFusion will need to be running a specific version of Java for the patch to work properly. If Java is not updated and the patch listed above is run, your server is NOT protected. Java patching is a prerequisite to the patch. The security updates require JDK 8u121 or higher (for ColdFusion 2016) and JDK 7u131 or JDK 8u121 (for ColdFusion 11).
Once Java has been updated, navigate into each ColdFusion instance and log in. Click Server Update > Updates. From there, click Check for Updates and apply the update. This will require a ColdFusion restart. Once done, confirm that the version of ColdFusion is updated to the version listed in the below table:
DataBank has partnered with CF Webtools, an Adobe partner to help with these ColdFusion and Java updates. CF Webtools can be reached by phone at 402.932.3386 or by sending an email to databanksupport@cfwebtools.com.
For Databank customers that have subscribed to 24/7 ColdFusion Support, DataBank will honor these obligations through the CF Webtools partnership. You should expect a ticket from DataBank in the coming days requesting permission to proceed with the update, including timing. You may opt out of the update when the ticket arrives.
For those customers that have not subscribed to 24/7 ColdFusion Support from DataBank, you may contact CF Webtools directly for assistance with updates. CFWebtools will guide you through their process, including making arrangements for payment directly with CF Webtools.
A World Gone Hybrid Calls for Better Management
It’s rare to find a business that’s not running, or considering, a hybrid IT strategy. Many enterprises choose to operate a combination of deployment methods to create a flexible model that will meet a diverse range of needs. A hybrid strategy, combined with services across the spectrum, is helping businesses on the journey to digital transformation.
51% of enterprises have adopted at least 1 hybrid workload
32% of enterprises’ cloud spend is allocated to hybrid, more than any other cloud delivery model (TBR, Dell)
Hybrid has become an opportunity to reach the next step in the evolution of business, but it’s also introducing additional complexity. As enterprises grow and opt for hybrid strategies,IT teams must content with an ever more complex, variable and sprawling infrastructure.
Partnering with a seasoned managed services provider is a strategic tool for enabling this hybrid IT revolution. It’s a difficult journey to take alone. Enterprises who aren’t able to innovate are falling behind, and it calls for the help of a partner who can take the time to understand your transformational vision, and provide the geographic reach, technical acumen and range of services that will help you get there.
How Managed Services Give Your Business a Leg Up on Transformation Goals
OEMs and data center providers alike are recognizing the need to offer full solution suites rather than single components of IT infrastructure. Many are starting to offer managed services, but not all have a truly comprehensive selection.
It’s common for IT service providers to label their business as an MSP, but many are limited in their offerings.When seeking to create a full solution set to optimize your hybrid IT strategy, service providers who can cover every technology practice area will be able to accelerate your digital transformation most effectively.
AtDataBank, we manage across the full platform of IT. Our in-depth approach applies to monitoring a customer’s solution portfolio, including the platform, applications, hardware, security, recovery, and tech support 24/7/365. Managed services cover everything from the network layer to IT infrastructure, patching, and tickets. It’s a package deal.
“We become an extension of their IT team. That’s why it’s really critical to have a managed service provider that can deliver the right level of competence across the board. We want our customers to be able to focus on their core responsibilities and let us handle their platform and service.”
Security and compliance also create added pressures that can be offloaded to an MSP. Staying on top of the many moving parts of compliance requirements is difficult to achieve internally, and governance will only continue to evolve.
Using resources on IT operations when it isn’t a core competency doesn’t make much sense in today’s business environment. MSPs are a great option for redirecting efforts and identifying the right strategy for excelling at digital transformation.
Making IT Easy with DataBank Managed Services
Knowing that managed services are a way to quicken the path to a hybrid IT strategy, we’ve put a great deal of effort into extending our managed services offerings and fortifying them with compliance support that can meet even the most stringent requirements—FedRAMP included.
Our managed service offerings include:
Monitoring
Recovery
Tech support
Patching
Backups
Security
“Through Databank’s robust managed services, we enable our customers to transition to a true hybrid cloud model. Instead of a one size fits all philosophy, we help align the platform, the technology stack and customer’s technology goals. Our customers buy across the spectrum: colocation, public cloud, dedicated private cloud, our virtual private cloud, and we enhance those offerings with visibility, connectivity, 24/7 operations, security and compliance. It’s all about enabling our customers to innovate”.
-Vlad Friedman, CTO, DataBank
One Year Passes Post-GDPR
The EU’s General Data Protection Regulation(GDPR) took effect on May 25, 2018. Announced in 2016, the EU gave organizations two years to shore up privacy processes in preparation of the deadline, upon which some of the most robust personal privacyprotection laws created would be put into effect.
What ensued was a mad rush to achieve compliance, often without due diligence or professional consult. Today, compliance professionals across industries are reflecting on what’s changed since GDPR went into effect.
Four Ways GDPR Has Changed How Companies Do Business in the U.S.
There are four primary ways GPPR has changed, or not changed, the way companies do business in the U.S., as well as its impact upon consumers.
1. GDPR has spurred a renewed privacy debate, leading to a new outlook on the subject by the general populace.
The back-and-forth on privacy is nothing new in the media. After all, the U.S. has had privacy laws since the 1960s. However, GDPR, coupled with the major mishaps of social media data sharing in 2016, have significantly influenced how the American populace views and thinks about what businesses are doing to protect consumers.
GDPR has given way to a new mindset, whereby consumers and end users have a strong sense of ownership over their personal data. People are very aware that companies use and retain their personal data in order to profit from what some would consider personal or intellectual property.
2. New laws are being written.
Most notably are the privacy laws in California. There are now increasing efforts to enact a person-centric, U.S. national privacy law, much like GDPR, although this is stumbling in the halls of Congress and meeting great opposition by business lobbyists in Washington.
3. Companies rushed to comply with GDPR. Enterprises across industries have put forth more budget, as well as new processes and procedures, to ensure compliance.
Have you noticed the uptick in websites requesting permission, or informing visitors of their use of cookies in popup ads? This is a prime example of the changes being made within businesses to ensure compliance with GDPR. Many of the companies impacted by GDPR were in a mad rushto comply, but without a complete understanding as to which elements of business required compliance, or why—and many still don’t. Decision makers commonly read media reports and lean on compliance professionals who say GDPR is the law, but it seems that few in the U.S. stopped to determine whether they’re under the jurisdiction of the EU.
4. An area unchanged: some companies have done their research, effectively determining they’re not required to comply.
For the companies who did take the time to conduct adequate research and consult legal counsel, ultimately determining that compliance isn’t required, not much has changed. At present, the question of whether EU law has jurisdiction inside the U.S. for a solely U.S.-based company is still in deliberation, and will be for years until a U.S. court determines the answer officially.
Bringing it Back to Basics: Privacy Laws Aren’t New
“For now, GDPR has not changed one thing we do. Companies rushed to comply as a result of media attention in late 2017 and early 2018. Since June of 2018, I have not had one compliance questionnaire or entity come to me to validate my GDPR compliance. This may change as GDPR matures and court cases determine jurisdiction and even practical implementation.”
-Mark Houpt, CISO
GDPR as a catalyst for privacy laws and compliance changes is often confounded with the social media data sharing revelations that took place in 2017. While it’s true that GDPR serves as a model for California law, and may for U.S. national privacy law as well, it didn’t initiate the real push. If anything, GDPR has demonstrated the consequences to American enterprises of failing to lead on the issue of privacy. We’ve all witnessed the damage data mishandling, leaks, and breaches inflicts upon marketing and sales efforts, as well as business overall. American businesses are used to owning the data they spend billions of dollars annually to collect. GDPR, and similar laws, have turned the tables.
On May 15, 2019, the President issued an Executive Order (EO) directing that communications technology or services be more tightly controlled within the United States, both federal and commercial spaces. The specific order can be read here.
As part of an ongoing effort to keep our customers and partners appraised of significant security developments, we wanted to share our impressions of the EO and its likely impact on our, and possibly youroperations.
The intent of the EO is to prevent “sabotage”, “subversion”, or other “catastrophic effects” of infrastructure and technology or technology services such as code, designs, manufacturing, or general supply. The EO was in response to and vaguely directed at Huawei and accusations that they are an arm of the Chinese government,. However, the EO was broad enough to encompass any similarly accused providers including those of future, actual, or perceived adversaries. It should be noted that China, and Huawei deny these accusations.
While there is no immediate action to be taken, we do anticipate the following impacts and changes to operations in months ahead:
This EO will impact how DataBank acquires systems and our overall acquisition process.
Our CISO, Mark Houpt, will begin working with our internal acquisitions teams when more information is provided.Specifically, we’ll be looking to the U.S. Federal Governmentfor a list of defined “adversaries” or a program similar to the ITAR/EAR programs operated through the State Department and Department of Commerce. This list will likely include country, company, and individual person sources that we will have further evaluate. We will then need to develop policies and processes to ensure that we acquire technology and services in accordance with any new standardsand we will need to follow those strictly. It should be noted that we already acquire technology and services in accordance with U.S. Federal Standards as defined by Supply Chain controls for FedRAMP. We anticipate these new policies or standards to be more strict.
We should be prepared for more stringent regulation and testing – aka auditing – of our acquisition process and supply chain as it relates to SSAE18, HIPAA, FedRAMP and FISMA.
Our network, server and storage devices, and even our environmental systems such as UPS, generators and transfer switches, will have to be verified and certified in some way as not being impacted by “adversaries” as defined by the U.S. Federal Government. Our CISO expects that in the coming weeks and months additional guidance regarding this will be coming from the FedRAMP PMO or other applicable agencies and it will be provided as soon as it is available.
We’ll continue to closely monitor developments and appraise our customers and partners of operational changes as they become clear.
CVE-2019-0708 and Remote Desktop Services
On May 14, 2019, Microsoft released a patch for Windows 2003, Windows 2008, and Windows 2008 R2 servers. The specific patch mitigates the possibility that an attack could happen via Remote Desktop Protocol (RDP). It is important to note that RDP is not by itself vulnerable. This exploit is pre-authentication and does not require user interaction. If exploited, the vulnerability could spread to other vulnerable servers in a worm-like fashion. To exploit this vulnerability, an attacker would need to send a specially crafted request to the server via RDP. The exploit could then run code to take over control of a system, delete files, and/or install programs.
Is there an exploit available for this yet?
At the time of this writing, there is not a public exploit available. However, given the severity of the patch (Critical), it is likely only a matter of time before an exploit becomes available.
Which server versions are affected?
Windows 2003, Windows 2008, and Windows 2008 R2. While Windows 2003 is no longer supported by Microsoft, this is an out of band patch that is being released. Customers should consider migrating away from Windows 2003 to a supported OS.
Our servers are protected by VPN though. Does that help mitigate the risk?
While DataBank always recommends locking down ports on the firewall and using secure methods of connecting to a server, such as via VPN, this does not prevent an exploit from occurring. While a VPN does lower the surface area of an attack as the communication is encrypted and the RDP service is not opened externally, the risk still exists until the patch is deployed. The risk can also be partially mitigated with Network Level Authentication (NLA) in that the exploit would not spread to other vulnerable systems with NLA enabled. DataBank recommends deploying the patch as soon as possible.
Mission Critical IT Demands an Always Available Support Team
Today, technology no longer merely supports the business; technology is the business. Every enterprise out there, regardless of vertical, is heavily reliant upon technology, with mission critical apps carrying out the bulk of day-to-day business tasks whether in ecommerce, a hospital environment, or a financial organization. It’s not the support environment of yesteryear. When internal IT resources are limited, their daily tasks quickly become dominated by flashing lights and alarms, while the core mission falls to the wayside.
Meanwhile, digital transformation has made hybrid strategies commonplace, with businessesincreasingly choosing mixed infrastructures. Cloud usage is growing, as is complexity.
79% of enterprises say that lack of internal resources and expertise is their primary challenge with the cloud (RightScale).
84% of businesses who choose to outsource IT have initiated discussions, conducted pilots, or implemented solutions around transformation. When executed properly, disruptive outsourcing can actually bring competitive advantage (Deloitte).
If you choose to enlist the help of third-party support in conjunction with outsourced IT, your chosen partner should prioritize functioning as an extension on the IT staff and bringing proactivity to operations. At DataBank, for example, forklifting everyday distractions is a key objective. Complexity varies from one customer to another, but businesses rely on our support team for everything from managing mission critical applications and monitoring alarms to patching and upgrades. And we’re incredibly proactive in our approach. In fact, DataBank responds to over 19,000alarms per month, with technicians logging in to identify the source 98% of the time, and most being resolved within 15 minutes, often before customers are even aware there was an issue.Customers can easily reach out if they notice an application is running slowly to request an upgrade. Support can be as comprehensive as the customer requires.
“We’re an extension of their IT staff, and they don’t have to hire technical IT people because we’re managing their applications and we go all the way down to the application stack. Making sure their assets and sites are up and running and they can concentrate on their core business objectives”
-Michael Deamer, Vice President of Client Services, DataBank
Increasing the Positive Business Impacts of Managed Services
When you rely on managed services to streamline some of the operational tasks that weigh down your own IT team, experienced support engineers should be part of the package. In order to maximize efficiency impacts, support engineers need to have competency that spans infrastructure to operating systems.In evaluating support services with your chosen managed service provider, they should cover the following practice areas:
General tech support
Monitoring
Recovery
Patching
Backups
Security
How DataBank Makes Life Easier for Customers Through Comprehensive Support
The DataBank Support experience actually starts with implementation and provisioning. Depending on the project at hand, the implementation team will configure services and assist with the migration, ensuring apps are up and running. Once implementation is complete, Supportleverages a deep bench of analysts, development, and database administrators to keep systems up and running continuously.
95-96% of every ticket submitted by a customer is handled by the same person who answers the phone.
98% of alarms are fixed within 15 minutes.
DataBank opens 75% of tickets overall. Proactive monitoring ensures that potential issues are identified before they impact customers.
Quality, depth and breadth of support is critical when selecting an IT infrastructure and managed services partner. Having the right technical staff who know multiple operating systems and can contribute proven expertise across the spectrum of support activities is the key to success.
“We have the right staff. They’re technical, and whether there’s an alarm or an open ticket, they’re proactive and they identify issues before they ever become a problem for the customer. That’s what support is really about.”
-Michael Deamer, Vice President of Client Services, DataBank
This article first appeared in Data Economy, written by DataBank CEO, Raul K. Martynek.
It’s difficult to read any industry publication today and not see an article about “The Edge”. Most articles seem to assume that it is a foregone conclusion that there will be thousands or tens of thousands of edge data centers in the not so distant future. Most articles attribute the evolution of edge data centers with the introduction of 5G and concepts like Mobile Edge Computing. And, almost all articles mention latency as the primary driver of edge data centers and the need to reduce latency to support a range of applications from autonomous vehicles to artificial intelligence to robotic surgery.
In this article, I would like to examine each of these assumptions and provide some observations regarding latency, the evolution of 5G and what DataBank is seeing in the marketplace today. I will argue the The Edge is here already today, but happening in an overlooked space – inside “traditional” data centers in secondary markets. I will also argue that the incremental benefits of micro edge data centers, for example those proposed at the base of cell towers, would provide negligible incremental benefits to latency and in fact introduce significant operational and technical hurdles to deploying infrastructure over large geographies.
The Impact of 5G
Let’s begin with looking at 5G and how it will impact latency and the decision on where to deploy infrastructure. There is no doubt that the widespread adoption of mobile 5G will have important consequences for the evolution of the modern Internet. Broadly speaking, the two most important benefits of 5G are increased throughput and reduction in latency. Everyone understands increased throughput. VZN’s limited mobile 5G rollout in Chicago and Minneapolis has delivered download speeds of up to 600Mbps. More bandwidth is great and I’m sure we’ll see new higher definition video codecs and other technologies emerge to consume that bandwidth.
Improved latency is the other big benefit of 5G. What is less understood is exactly how 5G improves latency. Below is a chart (Figure 1) that illustrates how data is transferred by a smartphone and indicates the approximate latency introduced by the various segments of a mobile network. You should note that the majority of the latency in 4G networks is created by the “air interface”, that is, the communication between your smartphone/mobile device and the Radio Frequency antennas mounted on cellular towers. On average over 50% of the latency on 4G networks is caused by the air interface. The latency in the transport and core network is much less significant compared to the impact caused by the air interface.
Figure 1
Below in Figure 2 is the same chart but for 5G. As you can see, 5G dramatically reduces the latency of the air interface, dropping round trip times from an average of 50-60ms (or greater) to sub 10 ms. The implications are clear: 5G – even without any change to where data centers are located, will dramatically accelerate application performance even from distant “cloud data centers”. Everyone will get an incremental latency benefit from 5G since the problem with the air interface will be largely addressed and terrestrial network will continue to move at the speed of light. By implication, applications that today (or tomorrow) could not function with the existing latency of 4G networks will perform adequately under 5G. Obviously, it is hard to predict what applications and in what instances, but it seems reasonable to assume that some group of applications will perform sufficiently well under 5G networks that no change in data center topology will be required to support them.
Now 5G does include Multi-access Edge Computing, also known as Mobile Edge Computing, which enables the incorporation of cloud computing capabilities within a cellular network and will be leveraged for applications like IoT. And for some set of applications, MEC will be useful. Our friends at Packet who are supporting the Sprint Curiosity IoT network with their bare-metal cloud is one such use-case. But the latency benefits of 5G will accrue to every mobile internet user regardless of where the application they are using is physically located.
The impact of 5G on future applications reminds me of the evolution of the early internet and development of VoIP. In the bubble era (for those that remember) many people tried to introduce VoIP as an alternative to TDM voice. Nothing worked in the 1995-2000 era as the internet was fundamentally not optimized for latency. As pipes got bigger and peering and caching evolved, VoIP over the open internet became possible. This is the opportunity that Vonage exploited and came to market with an OTT voice application in the early 2000s, something that could not have been commercially viable a few years earlier. I believe the same thing will happen with the introduction of 5G.
Figure 2
But let’s dig a little deeper into the latency argument for edge data centers and examine the incremental benefit of deploying compute and storage infrastructure closer to end users in a given metro market.
Before we do, it’s helpful to recall how most large-scale internet applications are configured today from a data center footprint perspective. Recently, we bid on a 2MW requirement for an ad-tech company who was looking to establish a footprint in Dallas. Ad-tech is one of the most latency sensitive internet use-cases. For their business model to function, they need to be able to identify the characteristics of an internet user on one of their web properties, determine their potential buying habits, conduct an auction to a group of ad buyers and insert the ad into the web page in under 100ms. Wow. The ad-tech firm had conducted significant network testing from various markets and determined that to be able to achieve national sub 100ms response, they would need to deploy in three US locations: Ashburn, Dallas and Santa Clara. This East, Central and West configuration is how most web scale applications are deployed. This is precisely why the largest wholesale data center markets are in Ashburn, Dallas and Santa Clara.
So this is what works (in broadly the same form) for most of the existing internet application use cases today. The argument for micro edge data centers is that new applications will require near real time latency and for that an East, Central and West configuration will not work. Instead, you will need to deploy at thousands of locations. Fair enough, so let’s examine the improvements in latency if we deploy infrastructure locally as opposed to solely in an East/Central/West configuration.
Figure 3 below shows a map of the US and expected latency from East/Central/West configuration to Minneapolis, MN. For the purpose of this exercise we used the actual DataBank locations in SLC, DFW and BLT. I would imagine the latency to Santa Clara and Ashburn would add another 5-10ms. I picked Minneapolis because it’s geographically most distant from the most common wholesale data center locations that web scale companies would likely deploy (exclude Chicago for the moment) so the benefit of reduced latency would be most pronounced. In addition, DataBank has two data centers in Minneapolis so we have firsthand knowledge of the available data center options and IP network dynamics in the market.
Figure 3
As you can see, the latency to Minneapolis introduced by terrestrial fiber transport is less than 45ms over the open internet from East, Central and West locations. Actual latency on a private MPLS/backbone network would be about 5ms-10ms lower. Recall that with 5G, the latency of IP packets will be greatly reduced even without the introduction of micro edge data centers since there will be significant improvement in the air-interface. This means that with 5G, most cloud data centers will be 25ms-50ms away from end-users, a significant improvement over 4G.
Nonetheless, let’s say you are a provider of an application that demands very low latency and looking to serve users in Minneapolis. What would you do? You would have two broad options: either deploy in a “traditional” data center within Minneapolis, for example, at either the DataBank MSP1 or MSP2 data center, or split your workload and deploy in a larger number of smaller micro data centers within Minneapolis. Let’s look at the incremental benefit of deploying in 5 micro edge data centers compared to deploying in a single traditional data center.
Figure 4 below estimates what your expected latency would be to various locations within the Minneapolis market if you deployed in a traditional data center like MSP1 or MSP2. If networks were perfectly optimized whereby an end user packets could go directly to the data center “as the crow flies” instead of having to hairpin through a peering point or carrier hotel, the latency would range from .16ms (for 10 miles) to .63ms. In reality, one would have to add the incremental latency to get to the carrier hotel or peering point in the market- but that will affect all users and all examples equally. Our ping tests indicate that from either MSP1 or MSP2, applications would be able to reach most destinations in the greater Minneapolis metro area within 3ms-5ms or less.
Figure 4
Figure 5 shows what would happen if you chose to deploy in 5 micro data centers, with MSP1 and MPS2 as two of the locations. In this scenario, your theoretical end user would be almost always within a 10 mile radius of a micro data center and be able to achieve ~.16ms round trip latency (Again, assuming networks are perfectly optimal…which they are not). But the incremental improvement from going from one data center location to 5 micro data center locations only improves your round trip latency by less than 1-2ms. Compared to an East/Central/West configuration, the vast majority of the latency benefit happens by simply choosing to deploy 1 node within a metro market. Deploying in tens-hundreds-thousands of micro-data centers would only improves latency by 1ms or less, and in some cases introduce latency depending on where the peering occurs.
Figure 5
When you consider the complexity, cost and operational support needs of deploying infrastructure in the field coupled with the benefits of scale that comes with aggregating infrastructure in a single location, the single data center deployment to serve a metropolitan market is superior. The conclusion is that once an application is deployed in a single location in a given market the actual latency to reach any eyeball in that market is dramatically reduced compared to the common East/Central/West configuration of web scale data centers and the incremental benefits of micro data centers evaporate.
We do business with many of the top 20 largest cloud and content players and we do see those customers leveraging our traditional data centers geographies in secondary markets to expand their capabilities. It seems logical to us that before the large cloud and content players deploy at 10,000 cell tower locations, they will first deploy a single cluster in a traditional data center in the top metro markets that they are looking to service and be able to reach any eyeball in those geographies with very low latency. In large geographic markets, for example LA metro, the provider can easily choose 2-3 traditional data centers already in the market.
Spend enough time in the telecom and technology industries and it becomes clear that the hype of many new technologies usually precedes the reality by 5-10 years. We believe that is the case with micro edge data centers. While it’s possible that at some point in the future new applications will emerge that we can only dream of today that will require an even more distributed data center geography, our view is the smart money will focus on this “medium edge” of second tier markets where cloud and content deployments are happening today.