MindFire Inc. - Notice history

This incident has been resolved.

Incident Summary On November 25, 2024, at approximately 1:30 AM PST, the MindFire Platform and Microsite Access experienced an unexpected service disruption due to an AWS infrastructure issue.

Technical Issue Amazon Web Services (AWS) unexpectedly suspended the Load Balancer* associated with the MindFire Platform and Microsite Access without prior notification.

Definition: A Load Balancer is a critical piece of infrastructure that distributes incoming web traffic across multiple servers to ensure reliable access to our platform.

Impact

• Duration: Approximately 11 hours (1:30 AM PST - 12:30 PM PST)

• Affected Service: MindFire Platform and Microsite Access accessibility

• User Impact: Platform and Microsite access was significantly impaired during this period

Resolution Process

1. MindFire engineering team began investigating the sudden service disruption

2. Initial attempts to implement a new Load Balancer were unsuccessful

3. Escalated the issue to AWS Support for investigation

4. AWS Support acknowledged the issue was on their end and worked to restore service

5. Full service restoration was achieved at approximately 12:30 PM PST

Root Cause AWS has acknowledged they shut down the service without any confirmation or authorization from the MindFire team.

Preventive Measures We are actively working with AWS to:

• Establish better communication protocols for critical infrastructure changes

• Implement additional monitoring systems to detect similar issues more quickly

• Develop more robust backup solutions to minimize future service disruptions

We sincerely apologize for any inconvenience this service interruption may have caused. The unplanned nature and duration of this outage falls below our standard of service, and we are taking steps with our infrastructure provider to prevent similar incidents in the future.

For any additional questions or concerns regarding this incident, please contact your account representative.

AWS Support has rectified the issue on their end and we are continuing to monitor to confirm all is back to normal.

Unfortunately, we are still waiting on AWS Support to resolve the issue. We are in active communication working towards a solution.

We have determined the root of the issue with the support of AWS.

We are currently experiencing connectivity issues affecting our service. Amazon Web Services (AWS) has acknowledged the incident and their technical team is actively working with us to restore normal operations. 

We will provide updates every 30-60 minutes at status.mindfireinc.com. We appreciate your patience while we work with AWS to resolve this interruption.

We have identified a Network Related Issue and are currently in communication with Amazon Web Services.