1. How your CA first became aware of the problem

During internal conversation about how to handle the root certificate modification to bring key usage in the associated root certificates into conformance, now covered by 1652581, we identified the issue relating to SHA-1 usage.

This led to us speaking to our auditors, and the various contacts at root programs about the decision to re-issue only updating the key usage field and continuing the use of SHA-1 over compatibility concerns. These conversations led us to believe that using SHA-1 was conformant with audit requirements and acceptable to the root programs.

We then created 1652581 and provided the updated certificates. During the review of that bug a member of the Mozilla forum expressed their concerns about how one of the replacement Root CA certificates did not replace the use of the SHA-1 with SHA-256 or SHA-384.

2. A timeline of the actions your CA took in response.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident.

N/A. This issue is not related to a sub ca or end entity certificate. Instead this incident addresses a new version of a trust anchor that is already present in the Mozilla root program. This modification ceremony occured on 2020-08-13 in response to incident 1652581 with the goal of making minimal changes to address a specific compliance issue so as not to impact compatibility with existing user agents.

This decision to make minimal changes resulted in the replacement, like its predecessor, to utilize SHA-1. No more replacement Root CA certificates are expected to be created.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.)

N/A. This issue is not related to a sub ca or end entity certificate. Instead this incident addresses a new version of a trust anchor that is already present in the Mozilla root program. This new version was created on 2020-08-13 in response to incident 1652581 with the goal of making minimal changes to address a specific compliance issue so as not to impact compatibility with existing user agents.

This decision to make minimal changes resulted in the replacement, like its predecessor, to utilize SHA-1. No more replacement Root CA certificates are expected to be created.

5. In a case involving certificates, the complete certificate data for the problematic certificates.

While six root certificates were updated as part of incident 1652581 only one of them utilized SHA-1.

In the response to this incident the decision was made to make minimal changes necessary to resolve identified conformity issues. As such, since the original version of this particular root certificate was created with SHA-1, which was allowed at the time, the new version also made the use of SHA-1.

The relevant crt.sh link is:
https://crt.sh/?id=3448659678

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

SHA-1 is a cryptographic hash algorithm that is used in a number of digital signature schemes, including sha1WithRSAEncryption. Due to several collision attacks that were found with this algorithm [1], the CA/B Forum and Mozilla decided to sunset its use[2][3].

With ballot 118, the CA/B Forum banned the use of the SHA-1 algorithm for signing end-entity and SubCA certificates after the 1st of January 2016. The ban did not apply to Root CA certificates.

It is our understanding that the exclusion of trust anchors from this language was intended to give root programs flexibility to make the decision of how to handle this issue. This is because, at least in well designed software, the signature in a root certificate is not trusted. Instead that trust is derived from how the user agent distributes and secures the policy that states that those root certificates are trusted.

The Mozilla Root Store Policy version 2.4[4] was subsequently updated to include additional requirements on the use of SHA-1 in signatures on digital certificates.

This update limited the use of SHA-1 by CAs included in the Mozilla Root Store Program only for signing intermediate certificates which chain up to roots in Mozilla's program if the certificate to be signed is a duplicate of an existing SHA-1 intermediate certificate where the new certificate limited the changes to the following:

• a new key (of the same size);
• a new serial number (of the same length);
• the addition of an EKU and/or a pathlen constraint to meet the requirements outlined above.

It is our understanding this accommodation exists to preserve the interoperability with existing CA certificates in the rare circumstances those certificates must be modified.

During the mitigation of incident 1652581, Google Trust Services needed to reissue six root CA certificates to fix a certificate profile conformity issue of those root CA certificates.

Though, as noted, well behaved clients should not rely on the signature in a root certificate, the breadth of the behaviors seen in clients validating X.509 certificates vary enough that there was concern that the algorithm change would introduce potential interoperability issues.

While researching the policy requirements relating to this modification we looked at the Mozilla language which did not include any requirements on the included Root CA certificates and reached out verbally to Mozilla, Microsoft and Apple and discussed our plan to only make the targeted changes necessary to address the conformity issues. In those conversations no concerns were raised so we continued with our plans.

Based on the Policy Authority’s interpretation of the intent of the Mozilla Root Store policy language, and the associated verbal discussions with the largest root programs we moved forward with the impression that doing so was compliant with the intent of the Mozilla policy.

This decision was based on the understanding
that the concerned certificate is a new version of an existing root certificate that already used SHA-1,
that the only change in the associated profile was to the key usage value,
that SHA-1 is an algorithm allowed by both the BRs and the Mozilla Root Store Policy for root certificates,
that the signature is not relied upon by Mozilla or other well behaved user agents.

However, the use of SHA-1 to sign a new Root CA certificate does not fulfill the requirements for using a key that also belongs to a previously included Root CA, which is also a possible interpretation of the Mozilla policy for this scenario.

The Policy Authority did not apply this interpretation when making the decision.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

The certificate in question expires on Dec 15 2021 GMT - a half-year from now. Given that:

• the signature on a trust anchor is not relied upon by Mozilla or other well behaved user agents,
• that modifying the root certificate with SHA-256 or SHA-384 represents a potential interoperability issue for non-well behaved user agents,
• attempts were made to coordinate with Mozilla prior to creation of this modified certificate.

It is our hope that Mozilla accepts the new certificate as is, or begins distribution of the 5 other certificates from 1652581 and holds off distrusting the SHA-1 signed root certificate until after it expires later this year.

Should those options not be possible, our intention would be to, upon approval, modify the root certificate again, this time signing with SHA-256 or SHA-384 and submit the revised certificate as a replacement for the aforementioned SHA-1 variant.

With that said, to limit the chance of issues occurring like this again we intend to bring future profile changes in root CA certificates to the public list for public consideration before moving forward.

Questions for GTS:

1. What was your process for researching the policy requirements associated with this scenario?

2. Did your research flag section 5.1.3 of the Mozilla Root Store Policy as potentially applying to this scenario?

3. Were you aware of the previous discussion of this policy at https://www.mail-archive.com/dev-security-policy@lists.mozilla.org/msg11762.html and if so, how did it factor in to your decision that this use of SHA-1 was compliant with the intent of Mozilla policy?

4. When reaching out to Mozilla, did you disclose that your plan would violate section 5.1.3 of MRSP?

5. At the time that the Policy Authority made its decision, was it aware of the interpretation that this use of SHA-1 was disallowed?

What was your process for researching the policy requirements associated with this scenario?

Most changes we deploy require some level of policy assessment. To support this we have a process that involves independent and collaborative reviews from the Policy Authority, Compliance and Engineering. This process starts with the creation of a shared document that details the issues, outlines potential solutions, and references any applicable requirements and precedents that have been identified by research that may inform the decision.

From this document both offline and interactive reviews of the topic take place during which the document is expanded with different perspectives and additional insights. From this a recommendation is made, sometimes, as was the case in this situation, additional insights are sought from auditors and root programs.

In this particular case, we felt third-party insights were needed so we reached out to both our auditors and members of root programs. Our interpretation of their responses fed into the ultimate decision made by the Policy Authority which led to the decision that was made in this case. This artifact and the final decisions are captured in an internal case management system.

Did your research flag section 5.1.3 of the Mozilla Root Store Policy as potentially applying to this scenario?

Yes, 5.1.3 and its predecessors were flagged as applicable and drove our decision to seek additional opinions. More details below.

Were you aware of the previous discussion of this policy at https://www.mail-archive.com/dev-security-policy@lists.mozilla.org/msg11762.html and if so, how did it factor in to your decision that this use of SHA-1 was compliant with the intent of Mozilla policy?

Yes this and other similar discussions were considered. Our conclusion from this was that clarity was needed on requirements for root modification so we reached out to both our auditors and members of root programs.

When reaching out to Mozilla, did you disclose that your plan would violate section 5.1.3 of MRSP?

We presented our interpretation without a specific determination as we were seeking additional feedback.

If we had arrived at a conclusive determination that our plan would lead to a policy violation, we would not have reached out, as there would be no reason to propose a non-compliant solution. Additionally, had any of the people we contacted expressed they felt there was a violation of section 5.1.3 or any other policy requirement, we would have pursued an alternative course.

At the time that the Policy Authority made its decision, was it aware of the interpretation that this use of SHA-1 was disallowed?

Based on our research on this topic we saw several ways to interpret the requirements relating to root modification. The policy authority made its decision based on the inputs from the policy authority itself, compliance, engineering, auditors and root programs.

During our review we researched section 7.1.3 of the Baseline Requirements and section 5.1 of the Mozilla Root Store Policy. Both requirements allowed the signing of SHA-1 hashes over root certificates, so we believed based on these sections modification of a certificate with the RSASSA-PKCS1-v1_5 with SHA-1 signing algorithm was not prohibited.
However, it was also noted that the Mozilla Root Store Policy had specific provisions on using SHA-1 to sign hashes with the key of an already included Root CA (section 5.1.3).

Since there was not an explicit coverage of how Mozilla would handle this case we sought guidance on which of these interpretations to follow, the one that applies to new Roots or the one that applies to previously included Roots.

Based on these conversations we determined that we could move forward with the understanding the modification would follow the process that applied to new Root CAs which had no prohibition on the use of SHA-1 based signatures.

During our review we researched section 7.1.3 of the Baseline Requirements and section 5.1 of the Mozilla Root Store Policy. Both requirements allowed the signing of SHA-1 hashes over root certificates, so we believed based on these sections modification of a certificate with the RSASSA-PKCS1-v1_5 with SHA-1 signing algorithm was not prohibited.

Could you explain why you believe Section 5.1 of the Mozilla Root Store Policy allows the signing of SHA-1 hashes over root certificates?

At the time that the Policy Authority made its decision, was it aware of the interpretation that this use of SHA-1 was disallowed?

Based on our research on this topic we saw several ways to interpret the requirements relating to root modification. The policy authority made its decision based on the inputs from the policy authority itself, compliance, engineering, auditors and root programs.

Could you link the inputs you based your decisions on? I cannot find related comments on bugzilla that would come even close to implying that it is fine to issue new SHA-1 certificates, and the offline communications referenced in Bug 1652581 Comment 14 have not yet been shared.

During our review we researched section 7.1.3 of the Baseline Requirements and section 5.1 of the Mozilla Root Store Policy. Both requirements allowed the signing of SHA-1 hashes over root certificates, so we believed based on these sections modification of a certificate with the RSASSA-PKCS1-v1_5 with SHA-1 signing algorithm was not prohibited.
However, it was also noted that the Mozilla Root Store Policy had specific provisions on using SHA-1 to sign hashes with the key of an already included Root CA (section 5.1.3).

Since there was not an explicit coverage of how Mozilla would handle this case we sought guidance on which of these interpretations to follow, the one that applies to new Roots or the one that applies to previously included Roots.

Is this something you can share with us? I couldn't find such communications linked in this issue, nor in Bug 1652581, and any record or paraphrasing of this conversation would be appreciated.

Based on these conversations we determined that we could move forward with the understanding the modification would follow the process that applied to new Root CAs which had no prohibition on the use of SHA-1 based signatures.

At least in Bug 1652581, no mention was made that you would re-sign your keys using SHA-1.

The CA certificate with Subject "OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign" with an RSA key whose SPKI SHA-256 hash is 8a27b5557b4bec7cc0305fbf3d53d1f71cd3f34910c5d65e27ecddb82077ba3d is included in the Mozilla root program.

Section 1.1 of the Mozilla Root Store Policy (MRSP) version 2.7 states that the MRSP applies to "CA certificates included in, or under consideration for inclusion in, the Mozilla root program" as well as the "the CAs which control or issue them".

Section 5.1.1 states "When a root or intermediate certificate's RSA key is used to produce a signature, only the following algorithms may be used". Under the algorithm "RSASSA-PKCS1-v1_5 with SHA-1" it states "see section 5.1.3 for further restrictions on the use of SHA-1".

Section 5.1.3 defines a list of permissible uses of SHA-1, none of which includes signing a root certificate with SHA-1, and ends with "CAs MUST NOT sign SHA-1 hashes over other data".

In a previous discussion about this policy, Wayne Thayer from Mozilla stated "What it does not permit is the issuance of new SHA-1 CA certificates".

Google Trust Services (GTS) argues that that the Baseline Requirements permit the use of SHA-1 for roots. However, MRSP Section 2.3 states: "In the event of inconsistency between Mozilla's Root Store Policy requirements and the Baseline Requirements, Mozilla's Root Store Policy takes precedence". It even explicitly calls out that "Mozilla may restrict permitted algorithms to a subset of those allowed by the Baseline Requirements". Therefore, the Baseline Requirements' position on this matter is irrelevant.

Given the above, it's quite clear that using the RSA key with SPKI SHA-256 hash 8a27b5557b4bec7cc0305fbf3d53d1f71cd3f34910c5d65e27ecddb82077ba3d to produce a SHA-1 signature over a root certificate is forbidden by the Mozilla Root Store Policy. It's hard to see how GTS came to the conclusion that there was not "explicit coverage of how Mozilla would handle this case" or why "clarity was needed". This incident report doesn't help to understand how this happened. Instead of acknowledging an issue with GTS' interpretation, it doubles down on it, calling the correct interpretation of MRSP, which is supported by both the text and prior discussions of the policy, as merely "also a possible interpretation".

Furthermore, this incident greatly resembles Bug 1612389, in which GTS somehow interpreted Mozilla's policy as permitting P-384 with SHA-256, even though the MRSP at the time listed P-256 with SHA-256 and P-384 with SHA-384 as the only permissible curve-hash combinations for ECDSA keys. Similar to this incident, GTS claimed that the policy was "ambiguous" without elaborating why. Bug 1612389 Comment 9 suggested that GTS raise on mozilla.dev.security.policy any requirements that they find confusing. GTS did not follow this advice. Had they followed it, this incident could have been avoided.

I think that GTS needs to review https://groups.google.com/g/mozilla.dev.security.policy/c/oP8XuNXrANw/m/oIYt70IiAAAJ and redo this incident report. It should acknowledge the problem, and it should take a more holistic approach to find the root cause, including an examination of why the changes made in response to Bug 1612389 failed to prevent this incident.

Comment 0 states (emphasis added):

This led to us speaking to our auditors, and the various contacts at root programs about the decision to re-issue only updating the key usage field and continuing the use of SHA-1 over compatibility concerns

and:

This modification ceremony occured on 2020-08-13 in response to incident 1652581 with the goal of making minimal changes to address a specific compliance issue so as not to impact compatibility with existing user agents.

(the phrase "minimal changes" is used 5 times in total)

and:

reached out verbally to Mozilla, Microsoft and Apple and discussed our plan to only make the targeted changes necessary to address the conformity issues

and:

that the only change in the associated profile was to the key usage value,

However, examining the original and reissued certificates reveals the following additional changes:

• Changing the serial number from 04:00:00:00:00:01:0f:86:26:e6:0d (88 bytes long) to 02:03:e4:f4:61:ec:99:d9:d5:79:66:ca:7a (104 bytes long)
• Reversing the order of the AKI and CRLDP extensions

Bug 1667844 Comment 11 confirms that these additional changes were made knowingly and intentionally by GTS. To see GTS describe the changes as "minimal" and "only ... the key usage field" here and in discussions with root programs calls into question the truthfulness of GTS' incident report and their candor when seeking input on their planned course of action.

It also underscores the arbitrariness of GTS' concern that switching to SHA-256 would cause "interoperability issues". GTS assumes this without evidence, yet deems a serial number length change and extension re-ordering as "non-material".

Finally, while it still would have been a clear-cut violation of MRSP 5.1.3 had the serial number length and extension order remained the same, these additional differences make this issuance an even greater deviance from the limited exceptional cases which MRSP 5.1.3 allows.

(In reply to ryan_hurst from comment #0)

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident.

N/A. This issue is not related to a sub ca or end entity certificate.

All certificates of a CA included in the Mozilla Root Store are subject to the requirements, unless explicitly excluded. And as this is a CA for Server certificates, I see no reason why you would classify this as not applicable.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.)

N/A. This issue is not related to a sub ca or end entity certificate.

The kind of certificate is not important here. You are mistakenly scoping the 'report problematic certificates' requirement to certificates issued under the root, but all certificates in the hierarchy are included, including the root certificate itself. Thus, this request for a report is applicable.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

It is our hope that Mozilla accepts the new certificate as is, or begins distribution of the 5 other certificates from 1652581 and holds off distrusting the SHA-1 signed root certificate until after it expires later this year.

It is my belief and understanding that this certificate has been misissued, and as such, not fit for consumption.

Other than that, I also find it important to note that the Google Trust Services audit report [0] does not mention the problematic issuance of SHA-1 certificates. I should note that the audit report does mention Bug 1652581 where the issue was noticed, but its short description only addresses the problem as detailed in the first few comments, not those mentioned in and after Bug 1652581 Comment 22.

Of further note is that this audit report does not include Bug 1581183 [†], Bug 1612389, Bug 1630079 [‡] and Bug 1634795, as issues disclosed to the auditor (which was EY again, ref Bug 1525446).

[0] https://www.cpacanada.ca/generichandlers/CPACHandler.ashx?attachmentid=246228

[†] mitigations were not completed until the audit period of that audit report had started
[‡] This details an issue from 2018, but was never documented

Could you explain why you believe Section 5.1 of the Mozilla Root Store Policy allows the signing of SHA-1 hashes over root certificates?
First let me clarify that we fully understand that you do not share our belief that MRSP needs clarification on this topic. We would also like to stress that we are not defending an interpretation that MRSP allows SHA-1, instead we are trying to explain why we thought that it was unclear on the use of SHA-1 for modifying CAs that are already in the root store, why we thought confirmation from Mozilla and our auditors was needed, and how we used those conversations to influence how we proceeded.

To this end, I would like to explain why we felt the language was inconclusive so that we can work together to determine how best to move forward.

• The text in Section 5.1.3 appeared incomplete as it addresses the conditions for modifying intermediate certificates but does not mention how to handle modifications of root certificates or if they are allowed at all.
• At the same time, Section 5.1 included text accommodations for the usage of SHA-1 based signatures for both root and intermediate certificates in some cases.
• In our conversations with auditors, their interpretation was that the reissuance of root certificates was not covered because there are still many SHA-1 roots in operation and it was their understanding that neither the CA/B Forum nor the browsers are currently intending to prohibit their usage before the end of their regular life cycle.
• In fact, despite the desire to limit SHA-1 based signatures, a total of 48 root certificates that were signed using this algorithm remain in the root program as of today.
• In other cases where the CA/B Forum or browsers wanted to discontinue a practice such as a particular validation method or the issuance of certificates with long validity periods, they have taken very clear policy action and expressly codified their sunset in the text.
• The fact that this had not been done for a number of SHA-1 signing operations from root and intermediate CAs, suggested to us that existing SHA-1 roots may continue to operate until they expire. As such we assumed that this would include fixing profile conformity issues in the same manner in which they would be fixed in other operational CAs.

Given these points, we believed that there was ambiguity as to the objective of the policy. To be clear, it was not our intention to ignore the prohibitions in Section 5.1.3. That is why we reached out to Mozilla to get confirmation as to what their interpretation was.

It would have been better to have taken this conversation to m.d.s.p.. The feedback of the larger community would most likely have helped to point out the SHA-1 implications of the profile change and it would have helped ensure that there is a broader consensus on how to treat them.

If similar questions on important issues come up in the future, we will be sure to start a community discussion on the topic.

Were you aware of the previous discussion of this policy at https://www.mail-archive.com/dev-security-policy@lists.mozilla.org/msg11762.html and if so, how did it factor in to your decision that this use of SHA-1 was compliant with the intent of Mozilla policy?

Yes, we did consider this discussion. Mozilla initiated it in response to a private inquiry regarding a similar policy question.

When we reviewed the discussion, we found it did not cover our specific scenario, that different opinions were expressed and that it did not end in a conclusion. Instead this thread appeared to result in an action item to draft an amendment to the MRSP so that the Mozilla community can decide what scope the SHA-1 prohibition should have.

In hindsight, we believe that we should not have proceeded while this decision was still pending. It would have been better to bring our question to the list with a similar post to this one to give everyone an opportunity to share their view on what the correct interpretation is for this specific scenario.

You state that you intended to make minimal changes and that key usage was the only change you made to the certificate profile however I have noticed two other changes

Thank you for bringing this up. I should have been more careful in my earlier wording.

Our goal was to minimize the changes in the certificate profile and not necessarily the content or matching the strict sequence of encoding used. This was because our goal was to minimize potential compatibility issues, while following both the requirements and processes that apply to this scenario.

I have tried to answer your specific examples of these other differences below:

• The Baseline Requirements mandate that root certificates utilize a random value generated with a CSPRNG as their serial number. Since we made modifications to the certificate, we generated a new serial number so that the different certificate versions can be distinguished.
• The serial number of the initial certificate was shorter than required by the BR in force today, so our ceremony tool correctly generated a longer compliant one.
• We are also not aware of any requirements relating to the ordering of the extensions nor have we seen interoperability issues in the past that were caused by the order in which extensions are represented. As such we did not feel modification of the ceremony tooling to preserve the ordering was warranted.

All certificates of a CA included in the Mozilla Root Store are subject to the requirements, unless explicitly excluded. And as this is a CA for Server certificates, I see no reason why you would classify this as not applicable.

As this is not an included root the answer of N/A seemed appropriate. We are happy to provide more detail but for us to do so, we need more information to understand what information is requested.

The kind of certificate is not important here. You are mistakenly scoping the 'report problematic certificates' requirement to certificates issued under the root, but all certificates in the hierarchy are included, including the root certificate itself. Thus, this request for a report is applicable.

Given that this incident relates to a root certificate that has not been deployed to clients, it seems that in this case the appropriate answer was N/A.

If this were about end sub ca or end-entity certificates, we would have provided the list of end-entity or sub-ca certificates as the answer to this question. We did provide a link to the root certificate. For reference that link is:

https://crt.sh/?id=3448659678

We are happy to provide more detail but for us to do so we need more information to understand what your question is.

I also find it important to note that the Google Trust Services audit report [0] does not mention the problematic issuance of SHA-1 certificates.
WebTrust audits are assessments against the WebTrust Principles and Criteria for Publicly Trusted Certificate Authorities which in turn are based on CA/Browser Forum requirements.

While we discuss policy related questions with our auditors, WebTrust audit reports do not cover root program policies and CA/Browser Forum requirements do not prohibit the use of SHA-1 in root certificates.

With that said, I want to stress that, at the time we did the root modification, and given the information we had, it was our understanding that the modification was not in conflict with the MRSP. If we had been aware of a conflict, we would have taken a different course of action.

Of further note is that this audit report does not include Bug 1581183 [†], Bug 1612389, Bug 1630079 [‡] and Bug 1634795, as issues disclosed to the auditor (which was EY again ref Bug 1525446).

The MRSP requirement to publish such information in audit reports became effective May 1, 2021. This means that the expectation is that our 2020/2021 report includes this information. Our current audit period ends in September and the corresponding report will be published before the end of the year and we will be requesting our auditors to include a table of all in scope incidents in that report.

We believe we have answered the questions presented so far. Please let us know if more information is needed.

The MRSP requirement to publish such information in audit reports became effective May 1, 2021

You are correct that this requirement became effective May 1, 2021.

Because this requirement was only effective as of May 2021, could you explain why only 4 of the 8 incidents in that period were included?

Non-compliance to RFCs due to malformed/incorrect/out-of-date OCSPs and CRLs sound like BR-related issues, incorrect registrations at CCADB and a potential delay for audits not so much.

All certificates of a CA included in the Mozilla Root Store are subject to the requirements, unless explicitly excluded. And as this is a CA for Server certificates, I see no reason why you would classify this as not applicable.

As this is not an included root the answer of N/A seemed appropriate.

Please do note that the MRSP specifically include (1.1 (1)) CA certificates that are to be included in the root store, (1.1 (2)) CA certificates that chain up to such root as specified in 1.1 (1), and (1.1 header text) the CAs that control or issue these.

First, this certificate was, to the best of my knowledge, explicitly asked to be included in the Mozilla Root Store, and thus qualifies under MRSP 1.1 (1).
Second, the certificate was signed with a key signature, which corresponds to the key of a root certificate that currently exists in the root store, thus also qualifying under 1.1 (2).
Finally, if somehow we'd discover that there exists e.g. an email signature signed by any included root CA key material, should we not ask these questions because it isn't included in the hierarchy of only the included root CAs? I believe not, it is a signature over data

To me it seems like Google Trust Services does not understand the scope of the root store, and fails to unserstand critical portions of both the MRSP, the BR and the RFCs related to certificate validation, resulting in worrying statements regarding the applicability of expectations.

I also find it important to note that the Google Trust Services audit report [0] does not mention the problematic issuance of SHA-1 certificates.
WebTrust audits are assessments against the WebTrust Principles and Criteria for Publicly Trusted Certificate Authorities which in turn are based on CA/Browser Forum requirements.

They also include validations that the CP/CPS of a CA correctly reflect the actual operations of the CA (see "CA Business Practices Disclosure" in [0], and "2.0: CA Business Practices Management" in the same document), and correct application of the relevant RFCs should thus by extension be included.

We believe we have answered the questions presented so far. Please let us know if more information is needed.

You have yet to reply to my questions in Comment 4 ("Could you link...", "Is this something ..."). I can understand that the requested information can be considered sensitive, but you're referring to these communications as being the reason you've gone through with signing new signatures over SHA-1, and as such I'd like to understand what happened in these communications, as they seem critical to understanding the issue.

[0] https://www.cpacanada.ca/en/business-and-accounting-resources/audit-and-assurance/overview-of-webtrust-services/principles-and-criteria, v2.2 and v2.2.1

I believe the answer to this question is in comment #9 but perhaps we did not state it as clearly as we intended.

As the subject of the WebTrust audit we have the obligation to disclose all known compliance issues to our auditors, moreover this is a responsibility we take seriously and believe have always lived up to. In addition, as a matter of practice, as we discover compliance issues we publicly disclose them to the Mozilla community.

Though WebTrust audits do not assess conformance with MRSP requirements as of May 1st 2021 these reports should include all Bugzilla bugs including those which are related to MRSP violations. This requirement presumes that CAs such as GTS will disclose any identified MSRP issues to their auditors so that the final report can reference them.

It has always been our practice to disclose such issues to our auditors and as such we expect the upcoming audit report for GTS covering Oct 1, 2020 to Sept 30, 2021 to also include all relevant facts as per section 2.4 of the Mozilla Policy.

I hope this helps.

(In reply to ryan_hurst from comment #13)

I believe the answer to this question is in comment #9 but perhaps we did not state it as clearly as we intended.

[snip]
It has always been our practice to disclose such issues to our auditors and as such we expect the upcoming audit report for GTS covering Oct 1, 2020 to Sept 30, 2021 to also include all relevant facts as per section 2.4 of the Mozilla Policy.

I hope this helps.

Thank you for the reply, it does clarify the question I should ask. Given your previous statement, do I understand correctly that you did disclose the mentioned "undisclosed" issues to the auditor, but were left out from the report by the auditor?

Flags: needinfo?(ryan_hurst@hotmail.com)

I believe you've overlooked my Comment 12 while resetting needinfo. Could you please provide your input on it's contents, and tell if you're planning to answer the questions of Comment 4?

1. How your CA first became aware of the problem

During internal conversation about how to handle the root certificate modification to bring key usage in the associated root certificates into conformance, now covered by 1652581, we identified the issue relating to SHA-1 usage.

This led to us speaking to our auditors, and the various contacts at root programs about the decision to re-issue only updating the key usage field and continuing the use of SHA-1 over compatibility concerns. These conversations led us to believe that using SHA-1 was conformant with audit requirements and acceptable to the root programs.

We then created 1652581 and provided the modified certificates. During the review of that bug a member of the Mozilla forum expressed their concerns about how one of the replacement Root CA certificates did not replace the use of the SHA-1 with SHA-256 or SHA-384.

2. A timeline of the actions your CA took in response.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident.

This is now being treated as a new trust anchor under consideration for inclusion in the Mozilla Root Store Program. The prior version of this trust anchor, which was included in Mozilla before the prohibition on SHA-1 and also included a signature over a SHA-1 hash, is still in the root program and is still in use.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.)

While six root certificates were modified as part of incident 1652581 only one of them utilized a signature over a SHA-1 hash. The relevant CA has the subject "OU=GlobalSign Root CA - R2, O=GlobalSign, CN=GlobalSign", was issued on 2006-12-15 and subsequently reissued on 2020-08-13.

5. In a case involving certificates, the complete certificate data for the problematic certificates.

The relevant crt.sh link is:
https://crt.sh/?id=3448659678

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

SHA-1 is a cryptographic hash algorithm that is used in a number of digital signature schemes, including sha1WithRSAEncryption. Due to several collision attacks that were found with this algorithm [1], the CA/B Forum and Mozilla decided to sunset its use[2][3].

With ballot 118, the CA/B Forum banned the use of the SHA-1 algorithm for signing end-entity and SubCA certificates after the 1st of January 2016. The ban did not apply to Root CA certificates.

It is our understanding that the exclusion of trust anchors from this language was intended to give root programs flexibility to make the decision of how to handle this issue. This is because, at least in well designed software, the signature in a root certificate is not relied upon. Instead that trust is derived from how the user agent distributes and secures the policy that states that those root certificates are trusted.

This is because, according to the path validation algorithm in Section 6.1 of RFC5280, the signature of a trust anchor is not validated, and a trust anchor in the form of a self-signed certificate is used only for initializing inputs to the certification path validation algorithm.

The Mozilla Root Store Policy version 2.4[4] was subsequently updated to include additional requirements on the use of SHA-1 in signatures on digital certificates.

This update limited the use of SHA-1 by CAs included in the Mozilla Root Store Program only for signing intermediate certificates which chain up to roots in Mozilla's program if the certificate to be signed is a duplicate of an existing SHA-1 intermediate certificate where the new certificate limited the changes to the following:

• a new key (of the same size);
• a new serial number (of the same length);
• the addition of an EKU and/or a pathlen constraint to meet the requirements outlined above.

It is our understanding this accommodation exists to preserve the interoperability with existing CA certificates in the rare circumstances those certificates must be modified.

During the mitigation of the now resolved incident 1652581, Google Trust Services needed to reissue six root CA certificates to fix a certificate profile conformity issue of those root CA certificates.

Though, as noted, well behaved clients should not rely on the signature in a root certificate, the breadth of the behaviors seen in clients validating X.509 certificates vary enough that there was concern that the algorithm change would introduce potential interoperability issues.

While researching the policy requirements relating to this modification we felt the language was inconclusive for the following reasons:

• The text in Section 5.1.3 appeared incomplete as it addresses the conditions for modifying intermediate certificates but does not mention how to handle modifications of root certificates or if they are allowed at all.
• At the same time, Section 5.1 included text accommodations for the usage of SHA-1 based signatures for both root and intermediate certificates in some cases.
• In our conversations with auditors, their interpretation was that the reissuance of root certificates was not covered because there are still many SHA-1 roots in operation and it was their understanding that neither the CA/B Forum nor the browsers are currently intending to prohibit their usage before the end of their regular life cycle.
• In fact, despite the desire to limit SHA-1 based signatures, a total of 48 root certificates that were signed using this algorithm remain in the root program as of today.
• In other cases where the CA/B Forum or browsers wanted to discontinue a practice such as a particular validation method or the issuance of certificates with long validity periods, they have taken very clear policy action and expressly codified their sunset in the text.
• The fact that this had not been done for a number of SHA-1 signing operations from root and intermediate CAs, suggested to us that existing SHA-1 roots may continue to operate until they expire. As such we assumed that this would include fixing profile conformity issues in the same manner in which they would be fixed in other operational CAs.

In addition, there were no previous instances of a Root CA certificate modification in the Mozilla Root Program that we could find. We felt that the discussion that can be found at https://www.mail-archive.com/dev-security-policy@lists.mozilla.org/msg11762.html did not cover our specific scenario since it concerned the signing of a SHA1 hash over a Subordinate Certificate.

As a result, we reached out verbally to Mozilla, Microsoft, Apple and our auditors and discussed our plan to only make the targeted changes necessary to address the conformity issues. These conversations specifically discussed the decision to use SHA-1, how we felt there was ambiguity referencing the same points outlined in this IR, and sought insight as to how the update would be processed under the policy.

Based on the Policy Authority’s interpretation of the intent of the Mozilla Root Store policy language, and the associated verbal discussions with the largest root programs we moved forward with the impression that doing so was compliant with the intent of the Mozilla policy.

This decision was based on the understanding:

• that the concerned certificate is a new version of an existing root certificate that already used SHA-1,
• that the primary change in the associated profile was to the key usage value,
• that SHA-1 is an algorithm allowed by both the BRs and the Mozilla Root Store Policy for root certificates,
• that the signature is not relied upon by Mozilla or other well behaved user agents.

However, the use of SHA-1 to sign a new Root CA certificate does not fulfill the requirements for using a key that also belongs to a previously included Root CA, which is also a possible interpretation of the Mozilla policy for this scenario.

The Policy Authority did not apply this interpretation when making the decision to move forward based on the above plan.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

The certificate in question expires on Dec 15 2021 GMT - a half-year from now. Given that:

• the signature on a trust anchor is not relied upon by Mozilla or other well behaved user agents,
• that modifying the root certificate with SHA-256 or SHA-384 represents a potential interoperability issue for non-well behaved user agents,
• attempts were made to coordinate with Mozilla prior to creation of this modified certificate.

It is our hope that Mozilla accepts the new certificate as is, or begins distribution of the 5 other certificates from 1652581 and holds off distrusting the SHA-1 signed root certificate until after it expires later this year.

Should those options not be possible, our intention would be to, upon approval, modify the root certificate again, this time signing with SHA-256 or SHA-384 and submit the revised certificate as a replacement for the aforementioned SHA-1 variant.

With that said, to limit the chance of issues occurring like this again, we intend to create a playbook for changes that affect our Root CA certificates. This will include getting the opinion of our auditors, the root programs, and the public list before moving forward.

In your previous statement, do I understand correctly that you did disclose the mentioned issues to the auditor?
Yes. We are required to disclose all known compliance issues to our auditors and did so in this case. We also reported the identified issues to the Mozilla community which our auditors are also a member of.

I believe you've overlooked my Comment 12 while resetting needinfo. Could you please provide your input on it's contents
I have re-read comment 12 and it is unclear to me which of the elements covered here you feel are not addressed. We have also just published an updated incident response which may also provide further clarification. If you feel a question in Comment 12 is adequately addressed it would help if you could clarify which questions you still have I will try to address them.

Tell if you're planning to answer the questions of Comment 4?
I believe we have answered all the questions in Comment 4 other than the question requesting copies of the communications with others about this issue. We do not have any audio recordings or transcripts of those conversations so unfortunately, we can not provide them. In the updated incident report we tried to clarify the nature of those conversations and how they influenced our decision making and hopefully, this will help address this question.

First, thank you for your comments, and sorry for bothering you. I was indeed mistaken. I failed to recognise overlap between the first part of my Comment 12 and the comments made by Andrew, more specifically Comment 5. You mentioned the concerns voiced by Andrew, and by extension my concerns, regarding applicability of the MRSP. I shall note that my concerns regarding your (lack of) perception of the MRSP and BR remain, and would like to urge you to implement changes similar to those suggested in the final paragraph of Bug 1708516 Comment 19.

Regarding communications with the root programs: I am disappointed that you have no unambiguous documentation regarding what questions were asked to the root programs, and what responses were given. You did mention the names of the root programs and what topics were discussed, but failed to document or mention the actual responses of the root programs; instead you mentioned that your interpretation of those responses was that you could continue, while the responses of the root programs are left as an exercise to the reader.

Gathering, documenting and then archiving evidence that is used for signing certificates is one of the natural requiremens of the WebPKI, and especially when a CA is unclear on requirements I would expect that CA to document their communications with the root programs so that no confusion can exist about the contents of the discussed items.

I urge you, and any CA really, to carefully document any clarifications received from root programs (and auditors), especially when you are one that requested those clarifications.

Other than that, it would be greatly appreciated it if Mozilla were able to clarify what communications with Google Trust Services regarding the re-issuance of a SHA-1 root prior to its issuance. Although ultimately it is up to Google to provide evidence, they clearly can't provide it. Additionally, the related mail thread on MDSP doesn't really go into any more detail than that the Google root program didn't receive any questions by mail, and the original issue for which the re-issue was initiated didn't have much details about responses to requests for clarification either.

15 days ago in Comment 5, I said:

I think that GTS needs to review https://groups.google.com/g/mozilla.dev.security.policy/c/oP8XuNXrANw/m/oIYt70IiAAAJ and redo this incident report. It should acknowledge the problem, and it should take a more holistic approach to find the root cause, including an examination of why the changes made in response to Bug 1612389 failed to prevent this incident.

GTS has not done this in Comment 15. Comment 15 is almost word-for-word a rehash of Comment 0 with Comment 8 added in.

Although GTS has provided several reasons why they arrived at their belief that the Mozilla Root Store Policy was unclear about signing a root certificate with SHA-1, none of the reasons have anything to do with the actual text of the MRSP. They have cited no specific text that would support the interpretation that signing a root with SHA-1 is allowed, or even any text that is allegedly unclear. Contrast to Comment 5 which cites specific text which clearly shows this use of SHA-1 to be forbidden.

While there is always some degree of subjectivity when interpreting words, there has to be some objective basis and textual evidence to support a claim of ambiguity. Consider a CA issuing a certificate based on a pinky promise that the applicant controlled the domain, and saying in their incident report that the root cause was that the Baseline Requirements weren't clear that pinky promises are forbidden. This would obviously be unreasonable and unacceptable.

Or, to use a less fanciful example, consider a CA which issues a certificate validated using a forbidden domain authorization document. In their incident report, they say that the certificate is a replacement for an existing certificate validated using a domain authorization document, and the Baseline Requirements are unclear because, despite the "MUST NOT", the BRs don't explicitly address the case of replacing an existing certificate, and because there are still extant certificates that were validated using domain authorization documents. These are essentially the same arguments that GTS is making here, and they do not support a claim of ambiguity in the policy. They explain why the CA would like the policy to say something, but just because a CA wants the policy to say something, it doesn't mean the policy is unclear.

For these reasons, the incident report in Comment 15 is insufficient. The problem isn't that the MRSP is unclear, it's that GTS has a faulty process for interpreting the MRSP. Bug 1612389 shows that this is a recurring problem, and GTS' other incidents (especially Bug 1708516) suggest that there is an even deeper problem at GTS with policy compliance in general. But this incident report does not even identify Bug 1612389 as related, let alone the deeper compliance problem, even though this was pointed out to GTS in Comment 5.

The insufficiency of the incident report is very evident in GTS' answer to question 7. Most of GTS' answer is irrelevant, having to do with the submission of GTS' new roots rather than resolving this incident. The only relevant portion (which incidentally lacks a binding timeline as required by https://wiki.mozilla.org/CA/Responding_To_An_Incident) is creating a playbook for changes that affect root certificates. But the next time this issue manifests, it probably won't be related to root certificate modification, just as Bug 1612389 wasn't.

As it stands, there is no reason to believe that GTS won't continue to misinterpret and misapply the BRs and the MRSP and consequentially have further incidents, possibly of a much greater severity than this one. This is deeply troubling for a publicly-trusted certificate authority, and raises the question of whether Mozilla's users are well served by having GTS as a trusted CA.

To ensure there is no misunderstanding about Comment #18:

• Does GTS have any specific text of MRSP they can refer to that they believe is ambiguous and/or supporting the materially incorrect understanding that GTS reached?
• Does GTS have any explanation as to how they failed to identify Bug 1612389 as related, in light of Comment #5?
• Does GTS plan to submit a revised incident report to address the material deficiencies with respect to Question 7, highlighted in Comment #18, to ensure to the community that GTS has taken adequate steps to ensure they do not materially misrepresent or misinterpret root program requirements, particularly those as security critical as the forbidden misuse of a root key?
• Does GTS have any plans to submit a more detailed response to the concerns raised in Comment #18 that suggest there is a serial pattern of issues at GTS that fail to be remediated?

Ultimately, I concur with Comment #18 that the incident report provided does not demonstrate GTS is sufficiently aware of the expectations of publicly trusted CAs, and this is gravely concerning. To attempt to help GTS better recognize these concerns, which to date have unfortunately continued to be misunderstood, and as mentioned in Bug 1708516 Comment #21, a more detailed analysis will be provided to GTS in the forthcoming days that looks to better highlight these concerns.

I shall note that my concerns regarding your (lack of) perception of the MRSP and BR remain, and would like to urge you to implement changes similar to those suggested in the final paragraph of Bug 1708516 Comment 19.

Thank you for the clear and constructive recommendation. We agree with and is why in Bug 1708516 comment 20 we have made it clear that part of the engineering role charter that we have defined is to incorporate more of the engineering organization into the compliance review process. We hope the additional insights brought by expanding the review process and through doing so the introduction of additional compliance training for the whole team will reduce chances of similar incidents in the future.

You did mention the names of the root programs and what topics were discussed, but failed to document or mention the actual responses of the root programs; instead you mentioned that your interpretation of those responses was that you could continue, while the responses of the root programs are left as an exercise to the reader.

Thank you again for the clear and actionable question and recommendation. We agree that this is an issue which is why in the incident response of bug 1709223 we committed to in the future to take such conversations to mdsp. This will ensure that there is both a durable public record of the conversation and that the opinions of the public on the topic can also be considered.

I think that GTS needs to review https://groups.google.com/g/mozilla.dev.security.policy/c/oP8XuNXrANw/m/oIYt70IiAAAJ and redo this incident report. It should acknowledge the problem, and it should take a more holistic approach to find the root cause, including an examination of why the changes made in response to Bug 1612389 failed to prevent this incident.
GTS has not done this in Comment 15. Comment 15 is almost word-for-word a rehash of Comment 0 with Comment 8 added in.

First, we would like to clarify that while Bug 1612389 occurred as a result of a gap in the compliance program it was not misinterpretation or confusion on the intent of the policy. It was, unfortunately, a failure of GTS to identify the requirement in question and ensure its compliance with it.

In that bug we did note that several CAs had the same problem and that in our analysis of those incidents it appeared the wording of the requirement was a contributing factor.

Importantly it was this incident that resulted in us restructuring the compliance program and as part of that introducing the bi-weekly review process described in earlier responses that has helped us identify several issues as noted in Bug 1706967 Comment 11. In the same bug we also noted that changes failed to prevent this incident due to “insufficient engineering representation in the compliance review meetings” and “no automation to monitor policy changes and response times”.

Second, we believe that we have tried to use a more holistic approach this time and we have identified what we believe to be the root causes, which is not the frequency of our meetings or our mix of staffing, but the structure of our program and how we have been approaching compliance. This is why in our remediation we introduced a new role and a plan to expand compliance further into the engineering and operations organizations.

With the goal of making it easier to follow the below table identifies both the identified root causes as well as their mitigations:

We do appreciate that you are judging these as insufficient. We want to ensure we do not miss an opportunity to both improve the way GTS operates as a result of the IR process, and expect to provide the requested clarity so if you could be more specific in what you feel we are missing and what information you are requesting we will do our best to produce it.

Although GTS has provided several reasons why they arrived at their belief that the Mozilla Root Store Policy was unclear about signing a root certificate with SHA-1, none of the reasons have anything to do with the actual text of the MRSP.

To be clear we are not arguing the text explicitly allows SHA-1 usage in the case of Root CAs, instead we are explaining how the allowance in other cases, where the signature is actually relied upon, led us to believe that there was ambiguity which led down the path we are on now.

The only relevant portion (which incidentally lacks a binding timeline as required by https://wiki.mozilla.org/CA/Responding_To_An_Incident) is creating a playbook for changes that affect root certificates.

Thank you for pointing out the omission of the date for this playbook deliverable, its omission in the IR was not intentional and the IR posted in Bug 1706967 Comment 11 should have contained an additional row to better reflect the work being done internally in the IR as well:

To be clear we are attempting to use the best practices for IRs outlined in https://wiki.mozilla.org/CA/Responding_To_An_Incident while focusing on making sure we provide the needed information. If you feel there is specific information that is needed please let us know and we will do our best to provide it.

In your original report you state that "the breadth of the behaviors seen in clients validating X.509 certificates vary enough that there was concern that the algorithm change would introduce potential interoperability issues." However you have yet to identify any client where changing the signature on the root certificate would create an interoperability issue. You later on state that other changes do not worry you:

(ryan_hurst from comment #8)

• We are also not aware of any requirements relating to the ordering of the extensions nor have we seen interoperability issues in the past that were caused by the order in which extensions are represented. As such we did not feel modification of the ceremony tooling to preserve the ordering was warranted.

Unlike the signature algorithm change, it is now that the ordering of extensions does matter to certain clients. certlint/cablint has a specific check that is expressly present due to a known client that has issues with extension ordering. Python versions without commit f06eb46918f11220d13e7170dcb17929498e3294 ignore the first extension when trying to find the subject alternative name extension. This results in the subjectAltName attribute of a certificate being set to None if the SAN extension is the first extension. This then triggers the fallback to check the commonName attribute in the subject.

This clearly does not impact CA certificates, as the code is evaluating the end-entity certificate, but it is an example of where the ordering does matter. I'm sure others can be found, but this is one that easily comes to mind and demonstrates why seemingly innocuous changes do matter.

In your original report you state that "the breadth of the behaviors seen in clients validating X.509 certificates vary enough that there was concern that the algorithm change would introduce potential interoperability issues." However you have yet to identify any client where changing the signature on the root certificate would create an interoperability issue. You later on

While exploring our options for the signature hash algorithm, we researched TLS stack implementations that lack proper support for SHA-2 algorithms.

Some of the examples we found include:

• Payment terminals and ATMs
  • Historically there have been payment processors that had expressed their concerns since they hadn't managed to fully update all terminals.
• Windows versions older than Windows XP with Service Pack 3 (we still see a large number of these users accessing endpoints secured by our certs)
• Android versions older than 2.3 (Gingerbread) (we still see a large number of these users accessing endpoints secured by our certs)
• OpenSSL 0.9.8 for encrypted communications
  • In particular, versions older than 0.9.8o do not support SHA-2 by default
  • Unfortunately, such old versions still exist in embedded systems. (working out exact numbers is tough for this use case as user agents and other signals vary widely, but we see significant number of these devices accessing endpoints secured by our certs)

Although we did not identify any cases that directly would have caused an interoperability issue in this case, the large number and variety of issues in this space had us worried that the change could break some clients. This led us to the aforementioned research on the compliance obligations which prompted us to propose the approach we ended up taking with the modification. We have already covered the fact that there were ancillary changes that we were not as concerned with so I wont cover that again unless you have a specific question.

We would like to reiterate that the use of SHA-1 on the self-signed signature on the root does not mean that any certificates would be issued using the SHA-1 hashing algorithm. We will not issue SHA-1 leaf certificates.

state that other changes do not worry you:

(ryan_hurst from comment #8)

We are also not aware of any requirements relating to the ordering of the extensions nor have we seen interoperability issues in the past that were caused by the order in which extensions are represented. As such we did not feel modification of the ceremony tooling to preserve the ordering was warranted.

Unlike the signature algorithm change, it is now that the ordering of extensions does matter to certain clients. certlint/cablint has a specific check that is expressly present due to a known client that has issues with extension ordering. Python versions without commit f06eb46918f11220d13e7170dcb17929498e3294 ignore the first extension when trying to find the subject alternative name extension. This results in the subjectAltName attribute of a certificate being set to None if the SAN extension is the first extension. This then triggers the fallback to check the commonName attribute in the subject.

This clearly does not impact CA certificates, as the code is evaluating the end-entity certificate, but it is an example of where the ordering does matter. I'm sure others can be found, but this is one that easily comes to mind and demonstrates why seemingly innocuous changes do matter.

Thank you very much for the additional context. During these discussions and through further research we've become aware of these limitations. When we were planning the certificate modification we were expecting the statements in RFC5280 about the ordering of extensions not playing a role in certificate validation to ensure few or no clients would have issues. We moved forward without ensuring that the same ordering still applies as we thought standards' expectations were clearly defined.

I think this bug can be closed, and I will review it again on this Friday 4-June-2021 for closure.

Ben: I'm not confident that GTS has sufficiently addressed the concerns in Comment #18 / Comment #19, and in particular, I do not believe this incident provides a sufficient demonstration that we will not encounter the same root causes again. As noted on the other GTS bugs, I believe this should be kept open until GTS is able to demonstrate that, which seems like it is currently waiting on me to provide that demonstration to GTS about the unaddressed root causes.

We believe we have both identified the root cause of this incident as well as proposed mitigations that would prevent similar classes of issues while broadly improving our compliance program and our community interactions moving forward. In this particular case, we feel the commitment we have made to bring future interpretation conversations to the community will substantially help in preventing similar incidents.

We understand that not everyone agrees, which could be a function of us failing to effectively communicate or maybe there is a remaining gap in our response. We are happy to address any specific feedback or questions from the community, in this and other bugs, but otherwise do not have any further information to provide in this bug.

(In reply to Ryan Hurst from comment #20)

I shall note that my concerns regarding your (lack of) perception of the MRSP and BR remain, and would like to urge you to implement changes similar to those suggested in the final paragraph of Bug 1708516 Comment 19.

Thank you for the clear and constructive recommendation. We agree with and is why in Bug 1708516 comment 20 we have made it clear that part of the engineering role charter that we have defined is to incorporate more of the engineering organization into the compliance review process. We hope the additional insights brought by expanding the review process and through doing so the introduction of additional compliance training for the whole team will reduce chances of similar incidents in the future.

I've read that comment, and although I do agree that it is a step in the right direction, it still is generally unknown what GTS does on the topic of conformance, and that doesn't spark any confidence. I'll add more context on that issue.

Google Trust Services is monitoring this thread for any additional updates or questions.

Google Trust Services is monitoring this thread for any additional updates or questions.

In the light of Comment 5 and Comment 19 (2), and the apparent lack of response to this question earlier:

Did Google Trust Services evaluate the suggestions made by Ryan in Bug 1612389 Comment 9 for implementation (specifically, the public discussion about unclear, ambiguous or confusing sections in requirements), and does Google Trust Services have an explanation as to why these suggestions were not implemented until after this issue was raised and these suggestions were re-iterated to Google Trust Services?

This is of course under the assumption that Google Trust Services by now has actually implemented processes to publicly discuss any sections of the BR and MRSP that Google Trust Services thinks are ambiguous or confusing (as mentioned in Comment 20), and not only changes in the CA profiles (be they Root CA or sub-CA) as Google Trust Services seems to talk about in Comment 0 and Comment 15.

First, we would like to clarify that while Bug 1612389 occurred as a result of a gap in the compliance program it was not misinterpretation or confusion on the intent of the policy.

This is contrary to your statement in the Incident Report in the Bug 1612389 : "At the time, we still felt our erroneous interpretation of 2.6.1 was valid".

If your latest statement should be taken as valid: If this was not based on a misinterpretation, and not on confusion regarding the requirements (which would still mean a misinterpretation), then was it a deliberate decision to misissue? I'm not quite sure other options remain. Sure, the root cause can be a gap in the compliance program, but somewhere in the causes was "a non-compliance with the requirements" for some reason, and if the requirements were not misinterpreted (in any shape or form) then the only remaining reason I can think of is intentional (malicious) misissuance.

In [Bug 1706967] we also noted that changes failed to prevent this incident due to “insufficient engineering representation in the compliance review meetings” and “no automation to monitor policy changes and response times”.

This response still fails to address that there was a suggested change to Google Trust Services' handling of ambiguous and/or confusing requirements; a suggestion that was not used, which subsequently led to this issue and a lack of transparency regarding on what basis Google Trust Services decided to proceed with issuing non-compliant certificates.

Google Trust Services is monitoring this thread and will soon provide a response.

Google Trust Services is monitoring this thread and will soon provide a response.

(In reply to Matthias from comment #30)

In the light of Comment 5 and Comment 19 (2), and the apparent lack of response to this question earlier:

Did Google Trust Services evaluate the suggestions made by Ryan in Bug 1612389 Comment 9 for implementation (specifically, the public discussion about unclear, ambiguous or confusing sections in requirements), and does Google Trust Services have an explanation as to why these suggestions were not implemented until after this issue was raised and these suggestions were re-iterated to Google Trust Services?

This is of course under the assumption that Google Trust Services by now has actually implemented processes to publicly discuss any sections of the BR and MRSP that Google Trust Services thinks are ambiguous or confusing (as mentioned in Comment 20), and not only changes in the CA profiles (be they Root CA or sub-CA) as Google Trust Services seems to talk about in Comment 0 and Comment 15.

First, we would like to clarify that while Bug 1612389 occurred as a result of a gap in the compliance program it was not misinterpretation or confusion on the intent of the policy.

This is contrary to your statement in the Incident Report in the Bug 1612389 : "At the time, we still felt our erroneous interpretation of 2.6.1 was valid".

If your latest statement should be taken as valid: If this was not based on a misinterpretation, and not on confusion regarding the requirements (which would still mean a misinterpretation), then was it a deliberate decision to misissue? I'm not quite sure other options remain. Sure, the root cause can be a gap in the compliance program, but somewhere in the causes was "a non-compliance with the requirements" for some reason, and if the requirements were not misinterpreted (in any shape or form) then the only remaining reason I can think of is intentional (malicious) misissuance.

In [Bug 1706967] we also noted that changes failed to prevent this incident due to “insufficient engineering representation in the compliance review meetings” and “no automation to monitor policy changes and response times”.

This response still fails to address that there was a suggested change to Google Trust Services' handling of ambiguous and/or confusing requirements; a suggestion that was not used, which subsequently led to this issue and a lack of transparency regarding on what basis Google Trust Services decided to proceed with issuing non-compliant certificates.

The gap in the review process leading to the erroneous interpretation in Bug 1612389 was that verification of the signing algorithm in the certificate profile was performed by engineering members of the team with lesser degrees of policy knowledge. The engineers who looked at this part of the profile did not see a policy issue, and the policy authority missed this requirement. This led to an oversight in that case. For this reason we saw enough of a distinction between the two incidents to not equate them.

Our use of the word "ambiguity" in that bug was in reference to the conversations that multiple CAs in the ecosystem had in their analysis of their non-conformance with this requirement.

We did evaluate the suggestion in Bug 1612389 Comment 9 and as a result we brought our policy questions to members of the community. As we have previously mentioned, this was, unfortunately, done in direct conversations with the root store programs. We confirm that we will post future questions concerning provisions of the Baseline Requirements, the Mozilla Root Store Policy, and other relevant standards (not only concerning CA profiles) that appear ambiguous or unclear to us to m.d.s.p. to gather community input on interpretation.

Google Trust Services is monitoring this thread for any additional updates or questions.

Google Trust Services is monitoring this thread and will provide a response soon.

Given the summary of issues presented in this comment: https://bugzilla.mozilla.org/show_bug.cgi?id=1708516#c35, is there any reason why this bug cannot be closed?

We have implemented all items in our remediation plan unique to this response. Our plans on improving our incident response process will be presented in Bug 1708516 and can be tracked there. However, we would be happy to answer any additional questions.

We are monitoring this thread for any additional updates or questions.

Ben, can you advise on the steps moving forward per https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/BOwcbWbZTg0/m/ceLXGwgEBwAJ?

Google Trust Services is monitoring this thread for any additional updates or questions.

I will close this on or about Friday, 30-July-2021, unless there are reasons to keep it open. (Mozilla will not be including this particular re-signed R2 CA certificate in GTS' pending inclusion request.)