Postmortem: My TTKT Service Outage

Date of Incident: Nov 11, 2025
Duration: 1 hour 43 minutes
Service Impacted: My TTKT
Timezone: All times in UTC

1. Summary

My TTKT experienced a global outage beginning on November 11, 2025, caused by a compromised latest update that contained a backdoor. An attacker was able to use this backdoor to upload a malicious shell and encrypt core middleware source files, causing the service to become unavailable.

Despite the breach, user data remained safe, as all data is stored on the separate MyForum service, which follows stricter update verification processes under bTeam Developer rules.

The incident was fully resolved on November 12, 2025, at 5:09 AM after restoring the last stable configuration and removing all malicious artifacts.

2. Impact

• Global outage: All users worldwide are unable to access My TTKT via website, app, and custom API

• Duration: 1 hour 43 minutes

• Data integrity: No loss or corruption. MyForum hosting protects all stored user content and personal information.

• Ops impact: Required rollback to stable version and removal of malicious files

3. Root Cause

A release deployed by the VDJGO! team bypassed rigorous security and code-review procedures. This version contained a backdoor that allowed an attacker to:

• Gain unauthorized access to the middleware server

• Upload a malicious shell

• Encrypt core source files, disabling system functionality

The insufficient review process allowed the compromised version to be deployed into production.

4. Timeline (UTC)

10:36 AM — Investigating
Service issues detected; the team begins an investigation.

12:41 PM — Identified
Problems traced to the latest update. Rollback initialized.

12:43 PM — Monitoring
Rollback completed; system behavior being observed.

12:44 PM — Update
Fix deployed. Some users (mostly in ideographic-language regions) still report data issues.

12:45 PM — Update
Primary issues resolved; technical report preparation begins.

5:09 AM (Next Day) — Resolved
Incident fully resolved. System restored and validated.

Total outage duration: 1 hour 43 minutes.

5. Recovery Actions Taken

• Restored the system to the latest verified stable configuration

• Rolled back the compromised update

• Cleared caches and removed the attacker-uploaded shell

• Verified full server integrity

• Confirmed that MyForum-hosted data was unaffected, personal user data and secret variable is okay

6. Preventive & Long-Term Action Items

Immediate

• Enforce strict code audit and security scanning before any deployment

• Strengthen mandatory review steps for VDJGO! updates

Near-Term

• Implement robust CI/CD validations:

  • Automated SAST

  • Dependency vulnerability checks

  • Backdoor/signature scanning

Long-Term

• Harden firewall rules and access controls

• Deploy intrusion detection and anomaly monitoring

• Introduce stricter deployment approval separation

• Provide ongoing security training for engineering teams

Postmortem: My TTKT Service Outage

Date of Incident: Nov 11, 2025
Duration: 1 hour 43 minutes
Service Impacted: My TTKT
Timezone: All times in UTC

1. Summary

My TTKT experienced a global outage beginning on November 11, 2025, caused by a compromised latest update that contained a backdoor. An attacker was able to use this backdoor to upload a malicious shell and encrypt core middleware source files, causing the service to become unavailable.

Despite the breach, user data remained safe, as all data is stored on the separate MyForum service, which follows stricter update verification processes under bTeam Developer rules.

The incident was fully resolved on November 12, 2025, at 5:09 AM after restoring the last stable configuration and removing all malicious artifacts.

2. Impact

• Global outage: All users worldwide are unable to access My TTKT via website, app, and custom API

• Duration: 1 hour 43 minutes

• Data integrity: No loss or corruption. MyForum hosting protects all stored user content and personal information.

• Ops impact: Required rollback to stable version and removal of malicious files

3. Root Cause

A release deployed by the VDJGO! team bypassed rigorous security and code-review procedures. This version contained a backdoor that allowed an attacker to:

• Gain unauthorized access to the middleware server

• Upload a malicious shell

• Encrypt core source files, disabling system functionality

The insufficient review process allowed the compromised version to be deployed into production.

4. Timeline (UTC)

10:36 AM — Investigating
Service issues detected; the team begins an investigation.

12:41 PM — Identified
Problems traced to the latest update. Rollback initialized.

12:43 PM — Monitoring
Rollback completed; system behavior being observed.

12:44 PM — Update
Fix deployed. Some users (mostly in ideographic-language regions) still report data issues.

12:45 PM — Update
Primary issues resolved; technical report preparation begins.

5:09 AM (Next Day) — Resolved
Incident fully resolved. System restored and validated.

Total outage duration: 1 hour 43 minutes.

5. Recovery Actions Taken

• Restored the system to the latest verified stable configuration

• Rolled back the compromised update

• Cleared caches and removed the attacker-uploaded shell

• Verified full server integrity

• Confirmed that MyForum-hosted data was unaffected, personal user data and secret variable is okay

6. Preventive & Long-Term Action Items

Immediate

• Enforce strict code audit and security scanning before any deployment

• Strengthen mandatory review steps for VDJGO! updates

Near-Term

• Implement robust CI/CD validations:

  • Automated SAST

  • Dependency vulnerability checks

  • Backdoor/signature scanning

Long-Term

• Harden firewall rules and access controls

• Deploy intrusion detection and anomaly monitoring

• Introduce stricter deployment approval separation

• Provide ongoing security training for engineering teams

Issues fixed. Detailed technical report will be published soon. We are continuing to investigate this incident.

We implemented a fix and are currently monitoring the result. Some users are reporting issues with their data, mainly in countries that use ideographic languages.

We implemented a fix and are currently monitoring the result. Hopefully the rollback will get the system working again.

We are continuing to work on a fix for this incident. Some issues occurred when we upgraded to the latest update. Rollback is in progress.