Over 23,000 users will have their SSL certificates revoked by tomorrow morning, March 1, in an incident between two companies —Trustico and DigiCert— that is likely to have a huge impact on the CA (Certificate Authority) industry as a whole in the coming months.
The entire saga started earlier today when DigiCert, one of the biggest certificate issuers on the Internet, sent emails to over 23,000 customers who obtained their SSL certificates through a UK reseller named Trustico.
DigiCert said that because of a security incident, they had to revoke all certificates issued to Trustico, which Trustico later sold to its own customers. Trustico General Manager Zane Lucas, on the other hand, denied that his company suffered any security incident.
At this point, it all become too complicated, so we'll just lay out a timeline of events, based on statements made by both companies, at the time of writing.
Timeline of events
➩ 1) On February 2, Trustico sent an email to DigiCert, asking DigiCert to revoke all certificates —around 50,000— managed by DigiCert.
➩ 2) Trustico drops its contract to resell Symantec certificates (now part of DigiCert) and starts a partnership with Comodo.
➩ 3) DigiCert denies the request to mass-revoke 50,000 certificates. DigiCert said that industry rules are not clear if a "certificate reseller" can revoke its customers' SSL certs, or only the end customer can do so alone.
➩ 4) Trustico says DigiCert decided to terminate its contract with Trustico on February 25, after Trustico said it would intend "to seek a legal opinion" on the matter.
Speaking to Bleeping Computer today on Twitter, a DigiCert employee confirmed the contractual obligation between the two companies was ending in 30 days.
➩ 5) In regards to the actual certificates, DigiCert says it told Trustico that they could mass-revoke certificates if there was evidence of a security incident during which the customers' private key were compromised.
➩ 6) DigiCert claims that on February 27 it received an email from Trustico containing over 23,000 private keys for Trustico customers SSL certificates.
➩ 7) In accordance with the CA industry rules that mandate that compromised certificates be revoked in 24 hours after a security incident, DigiCert started the certificate revocation process for the 23,000 compromised certs it received via email.
➩ 8) Earlier today, DigiCert sent emails to over 23,000 Trustico customers stating that their certificates would be revoked. It is unclear if DigiCert was allowed to mass-email Trustico's customers.
➩ 9) Several security experts have publicly accused Trustico of allegedly logging copies of SSL certificate private keys. Certificate authorities —the companies that issue SSL certificates— aren't supposed to have copies of these private keys.
When you signed up with them they integrated Client Side Requests into their website - which means they had the private key (which should never leave the client side). They also retained it and emailed to a 3rd party, a HUGE security hole. pic.twitter.com/iFg6MdcFbK
— Kevin Beaumont (@GossiTheDog) February 28, 2018
Turns out Trustico has an online private key generator, and probably logged all the customer private keys generated that way.
— Geoffrey Thomas (@geofft) February 28, 2018
People seem to be burying the lead with the @MrTrustico mass certificate revocation. Trustico was storing private keys for it's customers (something it never should have had, let alone stored,). That's not how CA's are supposed to work. This is insane. 1/n
— Jake Williams (@MalwareJake) February 28, 2018
Even DigiCert's COO —Flavio Martins— showed his surprise that Trustico sent an email containing the private keys of over 23,000 of its customers.
@rinsure we can't speculate on the reason, but we know that @trustico sent us a document with all of the keys, so revocation is required. Not sure why/how they have customer private keys...
— Flavio Martins (@flavmartins) February 28, 2018
The general theory among professionals —unconfirmed at this point— is that Trustico had automated the CSR (Certificate Signing Request) process, a step in the certificate issuance process, and was generating SSL certificates, but also keeping a copy of the private key.
➩ 10) DigiCert notifies Mozilla of the compromise of 23,000 private keys, promising to publish the private keys at a later date, so they can be untrusted by browser makers.
With the private key, the CA can absolutely impersonate you. I mean, any CA compromise is bad, but Trustico's behavior makes this an absolute worst case. It's clear that they don't understand how CA's really work either. Ugh. I'm not a customer and don't recommend. 2/2
— Jake Williams (@MalwareJake) February 28, 2018
➩ 11) Trustico answers DigiCert's report. Trustico says there was no security incident.
"At no time had any private keys been compromised, nor had we ever informed to you that any private keys had been compromised," Lucas said.
Trustico did not explain the origin of the 23,000 private keys. The company also did not reply to a request for comment sent by this reporter.
➩ 12) Trustico says the reason it wanted to revoke the 50,000 DigiCert certificates is because of Symantec. DigiCert bought Symantec's SSL-issuance business. The 50,000 certificates had been issued on Symatec's older network, and not by DigiCert directly. Google announced last year it would distrust all Symantec SSL certificates because of repeated security incidents. Now, Trustico says it lost faith in both Symantec (and indirectly in DigiCert) to manage their infrastructure correctly.
"During our many discussions over the past week we put it to you that we believe Symantec to have operated our account in a manner whereby it had been compromised," Lucas said. "We believe the orders placed via our Symantec account were at risk and were poorly managed. We have been questioning Symantec without response as to concerning items for about a year. Symantec simply ignored our concerns and appeared to bury them under the next issue that arose. "
"In good conscience we decided it wasn't ideal to have any active SSL Certificates on the Symantec systems, nor any that didn't meet our stringent security requirements," Lucas added. "The same management team responsible for that situation is duly employed at DigiCert and are fully managing our account, causing grave concern on our part as it appears to be business as usual with a new name. We were also a victim whereby Symantec mis-issued SSL Certificates owned by us, subsequently we were asked to keep the matter quiet, under a confidentially notice."
➩ 13) Lucas says that the system Trustico implemented so website owners could get a replacement certificate instead of the soon-to-be-revoked Symantec/DigiCert certs has failed today. This means over 23,000 users/companies will have to deal tomorrow with sites and apps that encounter HTTPS security errors.
——
In the meantime, despite Trustico's request, DigiCert has not revoked the certificates of the other 27,000 users for which Trustico wanted certificates revoked, but for which it did not present evidence of a compromise. A Mozilla representative agreed with DigiCert's decision to leave these certificates as "valid."
The entire incident is likely to end with sanctions for one company or the other. Either way, new rules will be voted to deal with the status of certificate resellers and the rights they have over end-customers certificates.
Furthermore, based on the comments by various security researchers, an investigation may be needed into whether or not a SSL reseller logged the SSL private keys of its customers.
Either way, the words "defamatory" and "legal opinion" were thrown around, meaning this issue isn't likely to die out after a few days, and the two companies may meet again, but with lawyers present.
Let's end this article with funny-guy GlobalSign trying to take advantage of this whole debacle for marketing purposes:
We don't keep your keys - so we can't lose them #Trustico > https://t.co/Po0AfLanUx pic.twitter.com/3Jm7oEhPqO
— GlobalSign (@globalsign) February 28, 2018
Comments
the_moss_666 - 6 years ago
I don't know if I should call it a scam, fraud or something else, but one thing is certain: You didn't get what you ordered. Only you shoud have private key, that's the point. If you buy a cash drop safe, you don't expect manufacturer having all the keys, right? CA having copy of privatate keys is something I'd expect from Chinese government, not the CA.
bkrodgers - 6 years ago
It's really not a "spat" when one side is so clearly wrong. To anyone who has a reasonable understanding of how certificates and CAs work, it's very clear that Trustico is completely wrong across the board on this, and Digicert did both what they should do and what they're required to do.
Trustico's initial premise is flat out wrong. Symantec's infrastructure had serious problems that warranted forcing them out of the industry, but that does not mean that every certificate they issued is inherently compromised. It makes sense to move off earlier if possible, but the premise that your site's security is inherently vulnerable in the meantime is simply incorrect.
And of course any process that results in them having the private keys of their customers further shows that they don't understand how the certificate issuing process is supposed to work and have no business being a certificate reseller.
I don't see anything Digicert did wrong in this event. Trustico, on the other hand, issued a statement that attempts to shift the blame to Digicert, but actually reads as an admission of their ignorance of how the business they're in is supposed to work and directly admits practices that are pretty egregious.
Hvergelmir - 6 years ago
Shameful display by Trustico. Indeed, sending private keys via plain text email forced Digicerts hand, but having them in the first place was is the real bummer.
Nathan1111 - 6 years ago
Having been a partner of Trustico, this impacted on around 50 of our customers. This string of blunders by Trustico started with their request to revoke certificates without notifying it's customers.. then sending it's private keys by email, again without authorisation from it's customers.. then cutting off their telephones and chat services so customers can not call in for support.. then issuing coupon codes which do not work.
The end result means we are left with $$$$ bill in new certificates, engineer time, and some very unhappy customers who are not able to connect to web sites, VPN's, camera systems, federation and other such services.
Foorack - 6 years ago
I really recommend looking into LetsEncrypt. Paid certificates are not better just because they cost a lot. LE certificates are equally as secure as other CA's, unless you are buyign OV and EV certificates (which are heavily debated if they help any at all). And by automating the process you could save a lot of engineering time in the future. Don't pay for something which is free. :)
Hvergelmir - 6 years ago
"I really recommend looking into LetsEncrypt. Paid certificates are not better just because they cost a lot. LE certificates are equally as secure as other CA's, unless you are buyign OV and EV certificates (which are heavily debated if they help any at all). And by automating the process you could save a lot of engineering time in the future. Don't pay for something which is free. :)"
Not exactly free from costs, but free of charge for the users receiving the certificates. Using those myself and I am very very happy with how easy it is to get the ACME clients running and renew certificates.
EV TLS certs make sense if the person requesting the cert wants to instill increased trust in the user visiting their service. But at the technical level they make connections no more or less secure than non-EV certs. For code-signing the reasoning is similar.
Nathan1111 - 6 years ago
Having been a partner of Trustico, this impacted on around 50 of our customers. This string of blunders by Trustico started with their request to revoke certificates without notifying it's customers.. then sending it's private keys by email, again without authorisation from it's customers.. then cutting off their telephones and chat services so customers can not call in for support.. then issuing coupon codes which do not work.
The end result means we are left with $$$$ bill in new certificates, engineer time, and some very unhappy customers who are not able to connect to web sites, VPN's, camera systems, federation and other such services.
dgelman6 - 6 years ago
Digicert has known about the Symantec issues for about a year, and basically has done nothing about it but issue press announcements.
Only when Trustico (wrongly) forced the issue did they actually do something.
Plenty of blame to go around.
Hvergelmir - 6 years ago
"Digicert has known about the Symantec issues for about a year, and basically has done nothing about it but issue press announcements.
Only when Trustico (wrongly) forced the issue did they actually do something.
Plenty of blame to go around."
Not really. Said issue doesn't make these certificates less secure, as bkrodgers pointed out already. However, the reseller keeping the secret keys _does_ make all certificates issued to a CSR signed with those private keys less secure. That blame goes right to Trustico. Perhaps Trustico was referring to the fact that _they_ themselves compromised the private keys, by them storing them? That'd make sense and it'd also explain why they were afraid to give an explanation to Digicert about why they wanted these certs revoked. With the facts that have so far transpired all the blame lies with Trustico. No doubt.
dgelman6 - 6 years ago
Symantec's certs are so secure that Google and Mozilla are declaring them not secure in X days... It's nice to think that the digicert people are responsive, but the reality is otherwise.
I'm not condoning Trustico. What they did was wrong. Plain and simple. Just not under the delusion that Digicert is perfect either.
Hvergelmir - 6 years ago
"Symantec's certs are so secure that Google and Mozilla are declaring them not secure in X days... "
Oh are they (Google and Mozilla)? Wasn't even aware of that. Sources please!
What I am aware of, though, is that browsers will start mistrusting the Symantec CA root and intermediate certificates, thereby removing trust for certificates that have been signed with a private key matching these CA root or intermediate certificates from said browsers and thereby making the TLS certs pretty much useless for business uses. I.e. they won't any longer be implicitly trusted by the browsers and all due to illicit practices by the Symantec CA in the past.
However you cannot conflate security and trust. If you know anything about TLS you'll know that implicit browser trust is the convenient cherry on top, but it's got nothing to do with security other - perhaps - than a vague (for non-EV certs) assurance that the domain ownership has been verified. Still more of a trust thing. Creating your own private CA - provided you have appropriate security measures - is a nice way of issuing TLS certs in an intranet by administratively making sure the CA root/intermediate certs are trusted by all clients. The browser vendors aren't at all involved here, because this is a private CA they haven't blessed with the implicit trust that is now going to be withdrawn from Symantec's CA. Nothing, literally NOTHING, makes connections established to intranet sites using TLS certificates signed by such a private CA (still, provided it's run properly) any less secure than it makes these Symantec-issued certificates less secure by Google and other withdrawing the trust.
"It's nice to think that the digicert people are responsive, but the reality is otherwise.
I'm not condoning Trustico. What they did was wrong. Plain and simple. Just not under the delusion that Digicert is perfect either.
Look, I'm not going to apologize for either side. Both are businesses, so they're likely to have cut corners if it helped their profits. In fact Trustico claimed that Digicert pretty much got the management they had to deal with while it was still owned by Symantec. So it's not really far fetched to think that some practices from the Symantec CA may have carried over, right?
However, sending the private keys by email - while one way of forcing Digicerts hand - sure was also a way to _actually_ create very real security issues for the customers affected. Whereas the upcoming removal of trust doesn't have security repercussions at all.