AWS health issues are affecting our services

Incident Report for Shop-Ware

Postmortem

Shop-Ware uses AWS (Amazon Web Services) for a variety of technical reasons. Beginning at 11:49 AM PDT on 6/13/2023, customers began experiencing errors in Shop-Ware that were a result of a degredation of the AWS service. Some issues reported resulting from the AWS outage were as follows:

Getting logged out of Shop-Ware
Inability to get logged back in
The "Reset Password" mechanism was not working
Photos were not loading in any repair order (e.g. within "notes", etc.)

Please note that the AWS outage affected many other websites, not just Shop-Ware. AWS began recovering from the degradation of service at around 1:38 PM PDT. As a result, internet sites using AWS (such as Shop-Ware) came back online, and services were restored.

Posted Jun 13, 2023 - 14:27 PDT

Resolved

Many AWS services are now fully recovered.
Shop-Ware should be functioning normally at this time.

Posted Jun 13, 2023 - 14:10 PDT

Monitoring

The Amazon Web Services (AWS) team has implemented a fix and appears to now be recovering.
As such, Shop-Ware services are also recovering. We continue to monitor the situation.

Posted Jun 13, 2023 - 13:57 PDT

Update

Amazon Web Services (AWS) continues to experience a service degradation. Shop-Ware services will be restored once the AWS issues are resolved. Updates can be found directly on the AWS status site: https://health.aws.amazon.com/health/status
Issues being reported are as follows:
- Getting logged out of Shop-Ware
- Cannot get logged back in
- "Reset Password" mechanism not working
- Photos are not loading in any repair order (e.g. within "notes", etc.)

Posted Jun 13, 2023 - 13:21 PDT

Identified

The issue has been identified and a fix is being implemented.

Posted Jun 13, 2023 - 12:52 PDT

Update

AWS (Amazon Web Services) continues to experience a service degradation which affects Shop-Ware (and many other internet sites). The AWS team has identified the root cause as an issue with AWS Lambda, and are actively working toward resolution.

Posted Jun 13, 2023 - 12:52 PDT

Investigating

We are being affected by an AWS service issue. This affects a number of sites, not just Shop-Ware.
From Amazon Web Services: Update: Jun 13 12:26 PM PDT
"We have identified the root cause of the elevated errors invoking AWS Lambda functions, and are actively working to resolve this issue."

Posted Jun 13, 2023 - 12:00 PDT

This incident affected: Web Application.