DoorDash, a $4 Billion company which enables small businesses to provide its customers with local delivery services, announced on Sep 26 a breach of financial and other private data of almost five million customers. According to DoorDash, the customers suffering stolen data were limited to those who created accounts prior to April 5, 2018, though not every consumer prior to April 5 was affected. Those who were, had their name, email, delivery addresses, order history, and phone numbers stolen. Further, over 100,000 delivery workers had their driver’s license information stolen, and both delivery workers and merchants had the last four numbers of their bank accounts stolen.
This article’s purpose is not to take shots at DoorDash. Many organizations today struggle to keep sensitive data secure despite rigorous attention and extensive effort. DoorDash is merely a high profile company that recently announced a breach, and there are a number of items reported in the TechCrunch article about the breach that should grab your attention if your organization handles sensitive data and depends on services and products provided by vendors.
Surprises and Mistakes
There are three things that might surprise you about the incident, even though they really shouldn’t. Other details surface after analyzing the incident that point to six common mistakes made in managing vendor risk.
Surprise #1 – DoorDash unaware for five months
DoorDash claims this most recent breach occurred on May 5, 2019, which means they were unaware for nearly five months. If that sounds like a long time, you will be surprised to learn that five months is actually less than the average. A study by the Ponemon Institute on behalf of IBM found the average time required to identify a data breach is currently 197 days — almost six and a half months!
It gets worse. The study also found that the average time required to contain a data breach after it’s discovered is another sixty-nine days. That’s 266 days — almost nine months — from the occurrence of the breach to its containment. Serious damage can be done in nine days — or even nine hours. It’s obvious that the industry average of nearly nine months for detection and containment needs to shrink dramatically!
In this enormous time gap, hackers and thieves are able to take their data loot to the black market and make their deals with no one the wiser. That data is disseminated and combined with other sources, and then used against the ultimate victims – the individual customers, patients, or employees. Of these individual victims, some begin to subsequently notice, and each begins to sound their alarm bell. Too often it is only when enough small alarm bells join to form a persistent and loud enough noise to alert the original breached company, that enough dots are connected to draw a line to the source. Too many customers, patients, and employees will continue to suffer from these breaches until organizations are able to dramatically improve the average time to detect an incident or breach. Since organizations exist because of these customers, patients, and employees, you might think that most organizations are focusing their resources on incident detection. But, as you will see in the common mistakes section in this article, you will be surprised!
Surprise #2 – Vendor responsible for the breach
DoorDash is pinning the blame on one of its vendors. DoorDash spokesperson Mattie Magdovitz blamed the breach on “a third-party service provider,” but the third-party was not named. According to Crunchbase, DoorDash employs eighty-nine other technology products and services in its company technology stack, and forty-nine active technologies on its website. This is not an unusual number, as Grubhub – a competitor for DoorDash – uses sixty-three in its company technology stack and eighty-eight technologies on its website.
Due to increasing specialization in the information technology industry, and the efficiency and effectiveness it brings technology solutions, it shouldn’t be a surprise that vendors are also increasing as the reason for a breach. According to Yahoo! Finance, in a recent survey by Ponemon of over 1000 information security professionals in the United States and United Kingdom, fifty-nine percent of respondents confirm that their organizations experienced a data breach caused by a third-party or vendor. And while vendor dependence is growing rapidly, confidence among IT security professionals is decidedly not. According to the same survey, only 16 percent say their organizations are highly effective in mitigating third-party risks.
Surprise #3 – Another DoorDash incident twelve months prior
It’s ironic that DoorDash experienced a security incident a year previous almost to the day. Below is more information about the earlier incident.
A number of DoorDash customers complained that their accounts were hacked on September 25, 2018. At the time, TechCrunch interviewed a dozen or so affected customers from that incident. Some of their responses conflicted with DoorDash’s official explanation that the incident was wholly due to hackers getting access to some users’ commonly used passwords by hacking another site, and then in turn using them on the DoorDash site (credential stuffing). This explanation dealing with the September 2018 incident, according to DoorDash, was verified by a third-party forensic firm.
The TechCrunch interviews, however, seem to rule out credential stuffing as the sole explanation, at least for some of the users interviewed. The TechCrunch reporter from the earlier incident in September 2018 states: “But six people we spoke to said that their password was unique to DoorDash, and three confirmed they used a complicated password generated by a password manager…when asked, DoorDash could not explain how six accounts with unique passwords were breached.”
The information TechCrunch elicited from a few of their customers should have been viewed as a favor that might have helped more fully understand the incident and possible risk. It’s unclear from the September 2018 article if it caused DoorDash to do any further investigation, or if they dismissed the idea that the incident was caused by something other than their official explanation. DoorDash probably has not yet finished its investigation into the more recent breach. The two may never prove to be related. But if we learn that they were related, will you be surprised?
Six Common Mistakes
DoorDash is probably still analyzing and responding to the breach they just announced in late September. We don’t know what mistakes were made in either incident. A good dose of humility is in order, because incidents and breaches can happen even to companies that have very aggressive and comprehensive risk management programs and tools in place. Nevertheless, certain aspects about what we do know about these incidents highlight six common vendor risk management mistakes organizations often make, and a few more surprises.
Mistake #1 – Not maintaining a comprehensive list of your vendors
This may strike you as so obvious that it seems silly to note at all, much less cite as mistake number one. However, you will be surprised to learn that “nearly two-thirds of IT security professionals surveyed stated that their organizations do not maintain a comprehensive list of third-party vendors and dependencies.” (Yahoo! Finance, November 21, 2018)
If you estimate that your organization only depends on a few vendors, you may not believe it is too difficult to establish and maintain an accurate and comprehensive list. Until you do a thorough inventory, though, you may be setting yourself up for an unwelcome surprise at just how many vendor dependencies you really have. In fact, some organizations discover that they depend on many hundreds of vendors, with new ones being added and others dropping off routinely.
When your vendor dependencies reach any kind of scale, merely maintaining a current, accurate, and comprehensive list of which vendors handle what sensitive data, or introduce technology which might compromise certain sensitive data, and which do not, represents a herculean task. It’s especially daunting when you consider that if your organization is like most and does not have a comprehensive list of vendor dependencies, just getting one established seems very labor and time intensive when you estimate you may be starting from an existing report from accounts payable of fifty, 500, or even 5,000 current active vendors to comb through and classify.
Mistake #2 – Not evaluating risks from vulnerabilities introduced by your vendors
It’s natural when evaluating a vendor to focus on the capabilities and resulting improved performance or bottom-line impact its product or service brings to your organization. On average, relatively less energy is expended in evaluating risks from vulnerabilities introduced by vendors. This is critical, because a vendor during the sales process will rarely bring up vulnerabilities he introduces to your organization — that’s your responsibility.
Yet the majority of organizations are not systematically shouldering that responsibility. According to an August 2017 Ponemon study sponsored by F5, only twenty-seven percent of IT executives who play the senior information security role for their company “have established a direct communication channel between the organizations’ security program and management responsible for contracts and procurement.” Security, whether intentional or not, is effectively an afterthought. This results in a huge business process gap — and a fundamental mistake — that is very common: seventy-three percent of the time.
In the context of information technology, software developers and IT professionals naturally focus on finding a tool, a piece of equipment, an application, or code library that solves or helps simplify adding a new feature or solving a problem. Your organization must also adopt the discipline that the discovery of the solution is merely the first step in determining whether to incorporate the new dependency. Your organization must maintain a culture where a new vendor dependency is thoroughly researched, learned, and discussed as to what vulnerabilities it may bring once incorporated. Preferably prior to the contract or the service adoption so those vulnerabilities can be more easily mitigated and baked-in to any agreement. But even post adoption, vulnerabilities need to be systematically searched for and discovered, and persistently tracked and tested.
Mistake #3 – Not marshaling more resources for incident detection and response
Managing risk consists of incident prevention, detection, and response. For years the industry has focused much more on prevention than it has on detection and response. That’s reflected in spending budgets, where industry estimates suggest that up to seventy-five percent of their security budget is consumed on prevention technologies, leaving only twenty-five percent to split between detection and response. According to Gartner, that has started to change some in recent years, as organizations come to grips with the inevitability of incidents.
As mentioned earlier, the average time to detect a breach is six-and-a-half months, and to contain a breach is almost two-and-a-half months more. When you acknowledge that so much damage can occur in mere hours or days, the conclusion you must draw is that you need to spend more of your resource budget on detection and response. If you could immediately detect a breach as it occurs, or within hours of it happening, and have automated and semi-automated response processes kicked-off and working to respond to mitigate and contain the breach, chances are the damage can be limited. But in most organizations, resources are stretched and budgets remain tight.
Here is where creative strategic thinking needs to be employed. It’s infinitely better to marshal a good number of resources to each do a little now than to wait for your budget “ship to come in.” Start by shifting more of your focus. Call a meeting or two to brainstorm with key people on your internal IT staff about simple and inexpensive ways to improve detection and response to incidents (btw, pizza typically increases participation and improves creativity). Make one of the first brainstorming sessions about how to identify and involve as many key vendor resources as possible to help leverage your team’s efforts. As the English writer John Heywood said, “Many hands make light work.” Remember, your organization is the customer, and most smart vendors will want to help in any reasonable way. As your focus shifts, your understanding of where to direct more of your future spending to provide for your most important priorities which emerge from your strategic discussions.
One area where any organizations are asking vendors who have technology solutions to cooperate in detection and response is by integrating with the organization’s Security Information and Event Management (SIEM) software system. The promise is that by correlating data from disparate systems, incidents will be easier to track, and possible causes will be more readily discovered.
Communicating better is the foundation for fixing most of life’s problems. If the upshot of your first meeting or two is merely getting a better list of key vendor resources and discussing how to maintain a contact list and improve tracking of issues — that’s a terrific start! Of course, as you consider involving vendors in incident detection discussions, you will need to be disciplined about how to share the right things freely yet still keep sensitive information contained to the proper people in your organization. In general, as you communicate more efficiently and start a dialogue with your vendors about how to better integrate to detect and respond to incidents more efficiently, each organization individually will reap enormous benefits. Better yet, the whole will prove to be better than sum of its parts.
Mistake #4 – Not conducting a thorough root cause analysis
If you don’t determine the true root cause, you can’t be confident that your data won’t be compromised again due to the same or similar cause. That’s self-evident. Nevertheless, there are a number of reasons offered as to why organizations still don’t conduct a thorough root cause analysis. In fact, sixty percent of respondents in a Ponemon report sponsored by AccessData stated that “they didn’t have the tools, personnel, and funding to conduct one. Thirty-eight percent of respondents say it could take a year to know the root cause of a cyber attack and forty-one percent of respondents say their organizations will never know with certainty the root cause.”
Remember the following mistaken assumptions in conducting a root cause analysis: (a) sometimes “the” problem may be actually two or more problems, and (b) sometimes a single problem may prove to have multiple causes.
The details of the earlier DoorDash incident from September 2018, combined with the TechCrunch user interviews, suggest that DoorDash may not have done a thorough enough job in root cause analysis in their initial incident investigation in that they didn’t appear to respond to users who told TechCrunch that they used unique and complex passwords for their DoorDash account. Is it possible that the single problem may have had multiple causes? Obviously, we can’t know unless DoorDash reports that they considered but eliminated this assumption – or confirmed it. According to their announcement, after the September 2018 incident they did employ some additional and significant monitoring tools to actively try to detect suspicious activity sooner, which is to their credit.
Even if you’re operating on a lean budget, your team needs to develop the skills and habits required to conduct a root cause analysis. Start by either updating an existing plan, or composing a simple plan with a sound methodology as to how you will go about it. An incomplete but written plan is better than none at all. This is especially true once you begin to iterate new written versions of the plan with your team. Involve as many people as is reasonable to review the plan and suggest edits and ideas. You don’t need — and probably cannot afford — your whole team to spend weeks at a time on it. Start short and simple and have a few people iterate on it for an hour or two once per month or once per quarter. Create one or two hour development tickets to research to become familiar with best practices and develop some expertise.
Again, expand your team’s resources and knowledge through creative thinking about how to involve your vendors, where reasonable and appropriate, as part of your incident root cause planning and analysis team. It’s perceived to be a tough “ask” to get another organization to devote some of their resources to help your organization think creatively about a problem at little or no cost. However, it’s hard for any vendor to resist answering questions a customer (you) might pose. It’s not extraordinarily difficult to get a vendor to schedule twenty minutes on a telephone call to field questions that help you brainstorm possible solutions, or eliminate impossible solutions, to a difficult problem. It’s amazing the time you can save, and the answers and other help you can get when you muster the courage or creativity to ask good ones!
Mistake #5 – Not routinely and periodically re-evaluating your vendors’ risk profile
Too often we don’t remember or apply the advice of the old Russian proverb “Trust, yet verify.” Companies, markets, products, and solutions change rapidly. How often have you been surprised to find how much a product, service, or its underlying technology has morphed since the last time you interacted with it? Even products or services that didn’t handle sensitive data or employ technology when you originally started using it, may incorporate new sensitive data, or new technology in a seemingly innocuous revision. That revision gets automatically installed, attaches to your network, or installs something to a device on it, and creates an unknown vulnerability. It’s imperative to re-evaluate your vendors and their products and services on a schedule so that you can discover new vulnerabilities or changes in risk profile. This provides control to either not allow the new vulnerabilities and risks to be introduced, or not allow them to remain hidden and unmitigated.
This sounds like a great idea in theory, but to someone who has experience on the front lines it may seem so daunting as to be “a bridge too far.” It probably is a very resource intensive process for your organization merely to accomplish initial evaluations of new vendors. You simply have too many existing vendors and not enough resources to review all of them once a year, or even once every two years. According to a recent Ponemon study (sponsored by Censinet) where 554 healthcare IT and security professionals involved in vendor risk management at their organization were surveyed: “Two out of three respondents believe that current manual risk management processes cannot keep pace with cyber threats and vulnerabilities…and only 27% assess all vendors annually.” Nevertheless, this remains a business process gap that is destined to plague your organization and your customers at some point with unwelcome surprises.
To solve this problem and bridge the gap, you will need to make sure you don’t make mistake #6 below, and instead look for smart automation tools for a solution.
Mistake #6 – Not employing an integrated and automated vendor risk management program
The following list represents the proactive flip-side of the mistakes we covered. Put these items on your to-do list for your organization’s vendor risk management program:
- Systematically establish and maintain a current list of vendors, and identify key resources and contact information for each vendor.
- Evaluate new vendor solutions for vulnerabilities and risks.
- Periodically re-evaluate vendor solutions to assess emerging risks.
- Routinely engage in a dialogue with vendors to improve the risk profile of products and services.
- Game plan together on how to detect and respond efficiently and effectively should an incident occur.
- Get your team up-to-speed and together develop a plan on how to conduct a thorough root cause analysis.
As anyone familiar with home improvement projects knows, having just the right tool for a task can make all the difference in how challenging the project will be and how long it will take. In the case of vendor management, you need to a tool specifically designed to help you evaluate your vendors routinely, on a schedule, and one which makes you very efficient in managing any follow-up an evaluation may require.
To avoid the mistakes we covered (and the ones we didn’t), you will need a robust vendor risk management system like ConfidentVMS that integrates well into your organization’s vendor management workflow to help you discover and eliminate gaps, and that employs smart automation and assessment tools to make the items above achievable, efficient, and affordable.
Written by: Curtis Jones (LinkedIn)
With over 25 years of experience leading healthcare information technology companies, Curtis helps leaders at healthcare organizations and their vendors manage their shared risk effectively, efficiently, and cooperatively.
About us: ConfidentVMS helps healthcare organizations reduce the resources required to lower vendor security risk, increase compliance, and achieve and maintain vendor HIPAA OCR audit readiness. Contact us for more information or to request a demo.