In today's fast-paced digital landscape, security and compliance are paramount for organizations, especially when evaluating potential SaaS partners. SOC2 reports have become critical for assessing these service organizations' reliability and security.
Whether you're evaluating Prismatic or any other SaaS business, understanding how to read and interpret a SOC2 report is crucial to making informed decisions.
This post aims to demystify SOC2 reports, providing you with the insights needed to discern the quality and transparency of these documents. As an insider with nearly three decades of IT and cybersecurity experience, I'm sharing my expertise to help you navigate the complexities of SOC2 reports effectively.
I'll cover the key elements of a SOC2 report, highlight potential red flags, and offer practical advice on what to look for. You'll be better equipped to assess the security posture of any SaaS provider, ensuring that you choose partners who prioritize security and compliance.
If you already understand SOC2 reports and Trust Services Criteria well, you can skip to the sections on Evaluating the applicable Trust Services Criteria and related controls and Common gotchas and red flags for the details on how SOC2 reports vary and how that might affect your decision-making process.
Overview of SOC2 reports
SOC2 reports are essential tools for evaluating the security and compliance of service organizations, particularly in the SaaS industry. These reports, issued by independent auditors, assess how well a company protects data and adheres to industry standards, helping businesses make informed decisions when choosing a partner.
SOC2 reports are either Type I or Type II:
- Type I reports focus on the suitability of the design of controls at a specific point in time. This report answers the question: Are the necessary controls in place and appropriately designed to achieve the desired outcomes?
- Type II reports, on the other hand, go a step further by evaluating the operating effectiveness of those controls over a specified period, typically ranging from six months to a year. This report answers the question: Do the controls function as intended over time?
The distinction between Type I and Type II is crucial. While a Type I report can give an initial level of assurance that the proper controls are in place, a Type II report proves that those controls are not only present but consistently effective.
Key sections of a SOC2 report and Trust Services Criteria
Let's consider each section and subsection of a SOC2 report in detail, including what you should expect to find in them.
- Independent Service Auditor's Report: The auditor's professional opinion regarding the effectiveness of the controls. Look for a clear and unqualified opinion indicating that the controls function as intended.
- Management's Assertion: Management's understanding of the system and the controls in place. Look for consistency with the auditor's findings and completeness in describing the system.
- Description of the System: An overview of the services, commitments, and requirements that define the system. Look for a comprehensive description that aligns with your understanding of the service.
- Services Provided: The specific services or products the organization offers. Look for clarity regarding what is being delivered and its relevance to your needs.
- Principal Service Commitments: Promises about security and other criteria, often in SLAs. Look for specific commitments that match your expectations and requirements.
- System Requirements: Technical and operational needs to deliver the services. Look for assurance that the system requirements support the stated services and commitments.
- Applicable Trust Services Criteria & Related Controls: The relevant criteria and controls ensuring security and compliance. Look for detailed controls that effectively address each relevant Trust Services Criterion.
- Security: Ensuring that system resources are protected. Look for robust controls that prevent unauthorized access.
- Availability: Ensuring system accessibility. Look for controls that guarantee uptime and reliability per SLAs.
- Processing Integrity: Ensuring accuracy and reliability of system processing. Look for mechanisms that ensure data processing is accurate and timely.
- Confidentiality: Safeguarding sensitive information. Look for encryption and access controls that protect confidential data.
- Privacy: Managing personal information according to privacy policies. Look for controls that ensure compliance with applicable privacy regulations.
- Tests of Controls and Results: The auditor's evaluation of control effectiveness. Look for strong test results with minimal exceptions or deviations.
- Types of Tests: Methods used by auditors to assess controls. Look for comprehensive testing methods that adequately evaluate the controls.
- Results Analysis: Interpretation of the test results to determine control effectiveness. Look for a clear and thorough analysis indicating effective control performance.
- Scope and Boundaries: Detailed declaration of what the audit includes and excludes. Look for a well-defined scope that covers all critical areas relevant to your needs.
- Other Information: Additional details provided by the service organization. Look for additional insights or clarifications to enhance your understanding of the report.
Evaluating the applicable Trust Services Criteria and related controls
The Applicable Trust Services Criteria and Related Controls section is the most critical part of a SOC2 report. It provides a detailed view of how a service organization meets specific security, availability, processing integrity, confidentiality, and privacy criteria. This section is key to understanding the control environment and ensuring that the service organization aligns with your organization's security and compliance needs.
Operational context is essential
Before diving into the details, it's important to consider the context in which the service organization operates. Different industries have varying levels of risk and regulatory requirements, so understanding the business context is necessary for evaluating whether the controls are sufficient.
For instance, we have different expectations for an established financial services company than we do for a tech startup, even if both are assessed against the same Trust Services Criteria.
Steps to review the Trust Services Criteria and controls
Step 1: Identify the applicable Trust Services Criteria
Start by identifying which Trust Services Criteria are covered in the report. Typically, the criteria include Security, Availability, Processing Integrity, Confidentiality, and Privacy. However, not all criteria apply to every service organization, so it's crucial to note which ones should apply compared to the ones that are being evaluated.
The Privacy Trust Criterion is typically used by businesses engaged in business-to-consumer (B2C) transactions and is not always relevant for business-to-business (B2B) transactions. For example, at Prismatic, we focus exclusively on B2B transactions, so we do not use the audit of the Privacy Trust Criterion. Understanding which criteria apply to your service organization is necessary for a proper evaluation. Many companies focus on only the Security Trust Criterion.
Step 2: Review the controls related to each criterion
Once you've identified the applicable criteria, review the controls in place for each one. These controls are the mechanisms, policies, and procedures the service organization has implemented to meet each criterion.
Under the Security Criterion, you might find controls related to encryption, access control, and incident response. Controls can be easily searched and obtained on the Internet for each Trust Service Criterion.
Step 3: Understand the testing performed and the results obtained
The auditor's report includes a description of the tests performed to assess the effectiveness of these controls. These tests might involve inquiry, observation, inspection, or re-performance of specific activities. Pay close attention to the results of these tests, as they are evidence of how well the controls are functioning.
The image in the next step provides an example of this with the "Test Applied by the Service Auditor" column.
Step 4: Pay attention to any deviations or exceptions noted
Finally, carefully examine any deviations or exceptions the auditor calls out. These could indicate weaknesses in the controls or areas where the service organization did not fully meet the criteria. Understanding the significance of these exceptions is critical, as they could impact your organization's decision to trust the service organization with sensitive data or critical processes.
The following image shows the Trust ID control, the COSO Principle tested, the SOC2 trust Control Description, the Test Applied by the Service auditor, and the Test Results for a couple sample controls. How the auditor performs the test is important, and the test result should generally be "No exceptions noted." If exceptions are noted, the auditor should list the exceptions and explain why they exist.
Larger companies like Alphabet and Microsoft will always have some exceptions. This is true because security becomes exponentially harder to manage at scale. And when a team of auditors studies a security program for months at a time, it will find exceptions.
It's critical that you understand each exception, get clarification from the service organization, and determine if they are deal-breakers.
Importance of a thorough evaluation
Evaluating the Applicable Trust Services Criteria and Related Controls section is more than a box-ticking exercise. It's about deeply understanding the service organization's control environment and assessing whether it meets your business's security, availability, processing integrity, confidentiality, and privacy needs. Following the steps outlined above, you can make a well-informed decision about whether a service organization is a suitable partner for your organization.
Common gotchas and red flags
When reviewing a SOC2 report, look beyond the surface to identify potential issues or gaps in the service organization's control environment. The following are common gotchas and red flags that could indicate problems with the report or the controls it describes.
Not all controls are reviewed
One of the most significant red flags in a SOC2 report is when not all controls are reviewed. This could be due to scope limitations imposed by the service organization or the auditor.
Let's look at what this means.
Scope and Boundaries section
Review the report's scope and boundaries section, which outlines what the audit included or excluded.
If the scope is limited, it may mean that certain controls were not reviewed. When a company limits its scope of controls it could be that it's trying to hide controls it doesn't score well on or controls that it has yet to implement.
Both of these situations could be negative. In some cases it's due to the company security team thinking the control does not apply to its specific situation when it should be the auditor making that decision. However, a company not having a full scope for a specific Trust Criteria is normally a red flag.
Independent Service Auditor's Report section
The auditor's opinion might contain disclaimers or statements indicating that certain areas were not included in the audit.
Phrases like "the audit was conducted on a sample basis" or "certain areas were excluded from the audit" are key indicators that the audit is not comprehensive. While "the audit was conducted on a sample basis" isn't bad (since that might speak to testing a reasonable subset of a large volume of things against the control), "certain areas were excluded from the audit" is normally a red flag.
Management's Assertion section
This section of the report may mention controls excluded due to scope limitations or other reasons.
You'll want to ensure that these exclusions make sense for your needs and expectations. Statements like, "We didn't include controls on BCP because our managed service organization takes care of that for us," intentionally hide something when a managed service organization's services should be part of what the audit covers.
Comparison to industry standards
If you're familiar with the typical controls for similar services, compare the report's findings with what you would expect. If you are unfamiliar, it's very helpful to use your favorite search engine or AI to search for the Trust Criteria controls pertaining to the report you are viewing.
If expected findings are missing, this could indicate that not all controls were reviewed. This could be a red flag. In rare cases, it may be a negligent oversight by the auditor.
Direct inquiries to auditor or service organization
If the report does not explicitly state that controls were excluded, yet some appear to be missing, it's reasonable to contact the service organization or the auditing firm to clarify whether all relevant controls were reviewed.
In the section on Evaluating the applicable Trust Services Criteria and Related Controls, when evaluating the Applicable Trust Services Criteria and Related Controls subsection, you'll note which controls are being assessed. If you notice gaps or areas that aren't covered, this could signal that not all controls were reviewed. It's important to correlate the findings in that section with the report's scope to identify any omissions.
The bottom line is that when key controls are left out of the review, there may be significant gaps in the assessment, making it harder for you to trust the report's conclusions. Understanding the potential impact of these omissions on the service organization's overall security and compliance posture is crucial
Exceptions and deviations
SOC2 reports often include exceptions or deviations where the service organization did not fully meet the assessed criteria. These exceptions could range from minor issues to significant control failures. You'll need to carefully analyze the significance of any exceptions the auditor documents.
Consider how these deviations might affect the service you're evaluating. For example, if a control related to data encryption has an exception, this could pose a severe risk, particularly if your organization wants to entrust the service organization with sensitive information.
Here's an example from a popular service organization's SOC2 Type 2 report. Under identity management there was an exception. The auditor stated: "For the selected quarter one and quarter three reviews, noted that 2 of the 35 users marked for modification or removal were not modified or removed in a timely manner during the quarter three user access review."
This means that this service organization knew that these two employees were no longer at the company but did not remove their access to the systems in a timely manner, even after noticing the issue in a quarterly access review. In this situation, the service organization had knowledge of the issue and did not act on it promptly. This is legally defined as willful negligence and is one of the worst outcomes possible in a security review.
Management's responses
When exceptions or deviations are noted, the report may include management's responses, outlining their remediation plans. While these responses are important, they should be scrutinized closely. Make sure there are detailed, actionable plans demonstrating a commitment to addressing the issues.
Vague or non-committal responses can be red flags, indicating that the service organization may not be fully committed to resolving the identified problems.
In the above example that dealt with not removing employees after their access was no longer needed, management responded with the following: "The quarterly access reviews supporting the applications in question did not include all relevant access groups related to its supporting services environments." Management's response in this case strongly indicates there is a problem with the auditing firm or a problem with the service organization. Either way, the integrity of that information is in question. This is a red flag.
Out-of-scope controls
Another common issue is the presence of out-of-scope controls or services. These areas are intentionally excluded from the scope of the SOC2 report, often due to business or contractual reasons. Identifying any out-of-scope controls and understanding their relevance to your organization is crucial.
If a critical service or system is out of scope, this could mean that significant risks are not being addressed in the report. Ensure the excluded areas do not compromise your company's security and compliance needs. For example, if you plan to send sensitive data through an embedded iPaaS and the service organization's SOC2 states that its encryption controls within the embedded iPaaS infrastructure are out of scope, this should raise a red flag.
Reliance on subservice organizations
Many service organizations rely on third-party vendors or subservice organizations to provide part of their services. When this is the case, the SOC2 report should address how these subservice organizations are managed and evaluated.
However, these third parties are sometimes not fully covered in the report. This can be a significant red flag, as it leaves gaps in assessing the overall control environment. Ensure that the report clearly outlines how subservice organizations are integrated into the primary service organization's control framework and whether they have their own SOC2 reports.
Complementary User Entity Controls (CUECs)
Complementary User Entity Controls (CUECs) are controls the service organization expects its users to implement as part of the overall security environment. These controls are crucial for the effectiveness of the service organization's controls.
A common gotcha occurs when these CUECs are not communicated or are overly burdensome for the user to implement. It's important to understand what CUECs are expected from your organization and assess whether they are realistic and achievable. Here's an example of an unreasonable CUEC:
"The user entity must implement multi-factor authentication (MFA) for all systems interacting with the service organization's platform, regardless of system sensitivity or access level."
Why it's unreasonable
- Misaligned with scope: This CUEC requires MFA for all systems, even if some systems don't directly interface with sensitive data or critical processes. For example, enforcing MFA for minor systems like simple logging servers or internal tools that don't interact with the platform may be overkill and costly.
- Costly or burdensome for the client: Implementing MFA across all systems, regardless of relevance, may be too burdensome for small or mid-sized user organizations. This could involve significant time, infrastructure changes, and additional licensing fees that outweigh the security benefit for systems that don't require stringent access controls.
- Misaligned with risk: The CUEC doesn't account for the risk-based nature of access controls. Not all systems warrant the same level of security as those that handle sensitive or regulated data. This one-size-fits-all control doesn't consider the user organization's specific business and security needs.
In this case, a more reasonable CUEC would be: "The user entity must implement multi-factor authentication (MFA) for systems that interact directly with the service organization's platform or contain sensitive data." This is more aligned with risk-based security practices and places the burden where it should.
Disclaimers and limitations
Finally, be aware of any disclaimers or limitations in the auditor's opinion. These can signal areas of concern where the auditor may have been unable to gather sufficient evidence or where the scope of the audit was limited. Such disclaimers can significantly impact the reliability of the report. Here's an example of a red flag disclaimer:
"The audit was conducted on a sample basis, and not all controls were tested across every location where services are provided. Additionally, cloud infrastructure assessment was excluded from the scope of this audit, as the service organization relies on third-party cloud providers who were not evaluated as part of this SOC2 report."
Why it's concerning
- Sample basis: When controls are tested on a sample basis, the auditor only reviews a subset of the total processes or locations. This means there could be potential gaps in controls that should have been reviewed. For example, if the organization has multiple data centers or offices, and only one location is reviewed, issues in other locations may be missed.
- Cloud infrastructure exclusion: The exclusion of cloud infrastructure from the audit scope is a significant limitation, especially for a service organization that relies on cloud services for its operations. If critical parts of the system (e.g., storage, compute power, or security) are managed by a third-party cloud provider that isn't audited, there is a gap in the assessment of the overall security and reliability of the service. This could pose a serious risk, as the cloud provider might not meet the necessary security standards.
In this case, such disclaimers can undermine the reliability of the SOC2 report, as they indicate that some critical areas were either not fully examined or were intentionally left out of the audit's scope. When reviewing SOC2 reports, you'll need to recognize these limitations and assess how they might affect the overall trustworthiness of the report and the service organization's control environment.
When I performed security audits for many companies, I had a customer with 45 physical locations. My team audited every one of those locations without exception.
Carefully consider the implications of any disclaimers or limitations and whether they affect your confidence in the service organization's control environment.
Conclusion
Understanding and evaluating a SOC2 report is essential for any organization that relies on third-party services, particularly in the SaaS industry. You gain critical insights into the service organization's control environment by thoroughly reviewing the report,
When assessing the Applicable Trust Services Criteria and Related Controls section, it's vital to consider your organization's business context and specific needs. Pay close attention to how well the controls align with your security, availability, processing integrity, confidentiality, and privacy requirements. This section is often the most revealing, as it provides a detailed look at the effectiveness of the service organization's controls.
But don't stop at simply reading the report. Be vigilant for common gotchas and red flags, such as controls that were not reviewed, exceptions and deviations, out-of-scope controls, and reliance on subservice organizations. These issues can significantly impact the report's reliability and must be carefully considered in your decision-making process.
Regularly reviewing SOC2 reports is crucial, especially when there are changes in the services provided or shifts in regulatory requirements. A proactive approach ensures that your organization continues to meet its security and compliance objectives. Sometimes what the SOC2 report doesn't say is more important than what it does.
Finally, if you need our SOC2 report as part of the process for evaluating Prismatic's embedded iPaaS, we are glad to help. Our platform is designed with security and compliance at its core, and we're here to answer any questions you may have as you navigate the complexities of SOC2 reports. Ensuring the security and integrity of your data is our top priority, and we're committed to supporting you every step of the way.
Recommended reference
AICPA (American Institute of Certified Public Accountants) - SOC2® - SOC for Service Organizations: Trust Services Criteria
- Description: The official source for SOC2 standards and Trust Services Criteria, providing foundational information on the structure and purpose of SOC2 reports.
- URL: https://www.aicpa-cima.com/topic/audit-assurance/audit-and-assurance-greater-than-soc-2
