How do we ensure device cybersecurity performance?
Device cybersecurity evaluations (i.e., security tests) are a critical, widely understood, and practiced Design Verification activity for connected medical devices. Many medical device manufacturers (MDMs) rely heavily on, and sometimes solely on, security testing. It must be said, however, that ‘testing’ should be the last line of defense to demonstrate the robustness of the security architecture. In previous blog posts, we discussed Secure Product Development Framework (SPDF) [Blog Post #1], Threat Modelling [Blog Post #2], and Cybersecurity Risk Management [Blog Post #3] practices. These three topics, and others, build upon each other to develop confidence with device security performance. Developing a robust security architecture with direct links to device risks, clear software integration and software system verification plans, code development standards, and code reviews (e.g., Static, Dynamic) should always be performed before actual device testing. Our goal is to ensure that a successful but usually one-off security ‘test’ does not provide a false sense of comfort but allows the security test to confirm all the underlying design practices, demonstrating a robust and secure design. A positive outcome for SPDF concludes with a completed security test but also incorporates identification of device security requirements, which, when properly crafted, influence the security architecture definition, designs, and implementation plans. Think of the individual SPDF practices as ‘layers’ to your device's security. FDA cybersecurity evaluations
Why should you test your product? FDA cybersecurity evaluations
Security evaluations (i.e., testing) ensure that device requirements are implemented correctly and align with the necessity and functionality of the device. As we stated in our SPDF blog, cybersecurity requirements should be aimed at protecting the device system assets against a list of threats carried out by potential attackers.
Various testing activities assist with vulnerability identification, device design, and implementation of the security requirements to mitigate these vulnerabilities and demonstrate the performance of the device system security architecture. Early identification of security-related vulnerabilities helps guard against potential security incidents that may have severe impacts on device users, user data, user reputation, and user operational costs.
Why are there so many types of security testing?
Device testing is performed using several different methodologies. These methodologies are complementary, as each focuses on different aspects, potentially identifying of various vulnerabilities. The difference between the security test types is predicated on the effort required to carry out the activity. Some tests may be automated and executed with low effort, whereas others require more effort, such as manual testing. The effort impacts the frequency of these tests executed during the development lifecycle. To articulate these security evaluation ‘layers’, we highlight the following:
Code review
Code review is the process of analyzing and inspecting the source code of an application to identify flaws that may lead to security vulnerabilities, as well as application logic and implementation vulnerabilities. Code reviews may be executed manually by a human reviewer, ideally independent from the original developer, or automatically using dedicated tools. Consider this security evaluation layer # 1.
Fuzz testing
Fuzz testing is the practice of sending large amounts of malformed or unexpected messages to the device to assess the device system's response. Generation of these malformed messages is usually automated by a ‘Fuzzer,’ which may create random messages or apply domain-specific knowledge to stress the device application boundaries. The purpose here is to explore the input space of the application as much as possible to identify implementation errors in message handling. Consider this security evaluation layer # 2.
Vulnerability testing
Vulnerability testing evaluates device components to identify the presence of known vulnerabilities. This is a particularly useful case for third-party components (e.g., Off the Shelf Software - OTSS, Software of Unknown Provenance – SOUP) such as open-source libraries. Information about known vulnerabilities is generally available in dedicated databases or vendor advisories as discussed in our Cybersecurity Risk Management Deep Dive # 2. Identifying these known vulnerabilities allows device developers to evaluate the strongest possible Risk Control Measures (RCM) for implementation. Consider this security evaluation layer # 3.
Penetration testing
Penetration testing is a manual process where the tester executes an in-depth analysis and study of the device to identify previously unknown vulnerabilities in the software logic, code design, or implementation. A penetration testing campaign may involve various techniques, ranging from simple network traffic sniffing to sophisticated fault and glitch injections. It is strongly advised that a Penetration Test be executed before device market release to identify vulnerabilities that eluded other evaluations performed during development. Consider this security evaluation layer # 4.
Why is Penetration Testing so powerful?
Penetration testing may be executed in three ways based on the amount of information available to the tester:
If the tester has no access to device information, we consider this ‘Black box’ testing
If the tester has access to the full device design documentation or software code, we consider this ‘White box’ testing
If the tester has partial access to the device design documentation, we consider this ‘Grey box’ testing
This testing mode is evaluated case by case; generally, ‘White box’ or ‘Grey box’ testing requires less effort as it allows the tester to skip most reverse engineering, a time-consuming element of ‘Black box’ testing.
Penetration testing involves two main steps:
Identification: Step one involves an extensive study and analysis of the device to find possible ‘weak points’ that may be exploited to compromise an asset. Among the most common techniques we recognize is ‘sniffing’ of the network communication, analysis of binary code, and analysis of the available interfaces.
Intrusion: In step two, the elements identified in the previous step are further studied to identify how they may be exploited to compromise a system asset successfully. A wide range of techniques may be used to obtain such results, starting from the most common and documented exploitation methods, such as a buffer overflow, to create ways the tester may misuse a system feature.
As the last layer of the security evaluation, Penetration Testing analyses the device design, creating deep knowledge of the device to attempt to break it down from a device system level.
What does FDA require?
From FDA guidance (2023):
“FDA recommends…the following types of testing…:
Security requirements…evidence that each design input requirement was implemented successfully…evidence of [the] boundary analysis and rationale for…boundary assumptions.
Threat mitigation…details and evidence of testing that demonstrates effective [RCM] according to the threat models…ensure the adequacy of each cybersecurity risk control.
Vulnerability Testing…provide details and evidence of the following…:
Abuse or misuse cases, malformed and unexpected inputs (Robustness, Fuzz testing, Attack surface analysis, Vulnerability chaining, Closed box testing…, Software composition analysis…and static and dynamic code analysis)
Penetration testing…identify and characterize security-related issues via tests that focus on discovering and exploiting security vulnerabilities in the product.”
Furthermore, FDA guidance reveals a few more details of Penetration Testing:
“Penetration test reports should be provided and include the following elements:
Independence and technical expertise of testers
Scope of testing
Duration of testing
Testing methods employed
Test results, findings, and observations”
It is important to remember that anomalies (i.e., Bugs) and vulnerabilities discovered during security evaluations must be assessed for potential security impacts as part of the Cybersecurity Risk Management process (see Deep Dive #2).
Remember, the most secure device systems are developed intentionally and in ‘layers’. A simple set of requirements defined during the start of development, culminating with a Penetration Test at the end of development, does not depict sufficient ‘layers’ to make us feel confident in the security architecture.
How Security Pattern can help
We offer a variety of Penetration Tests to assess the security of a system or device. They are intended to circumvent device security policies and include software, firmware, and hardware techniques, both non-invasive and invasive. Using the equipment in our lab, such as our EMFI or Side-Channel setup, our skilled Penetration Testers mimic the behaviour of attackers. Examples of techniques adopted during our Penetration Tests are fault injection, side channel attacks, and protocols/API fuzzing.
Discover our Penetration Testing Service
Discover our Penetration Testing Training
About the Author
Dr. John Salvato, President, Design Quality Services LLC
Design Quality Services (DQS) was established to support large and small innovators' product development endeavours, resolve quality system challenges, and provide technical training. The pathway to commercialization is complex, regulatory expectations can be ambiguous, and quality management systems (QMS) may become unstable. DQS brings a pragmatic approach to enable these pathways, fostering business growth and market success. We are here to help you solve your toughest challenges. John has an impressive 30-year career spanning quality assurance, manufacturing, and product development in the medical device and automotive industries. As the president of DQS, he brings exceptional engineering abilities and unparalleled global management experience. His leadership has led to successful development projects globally. John leverages his many years of professional experience with his academic work.
Comments