Duration: 2 minutes

Affected Components: Genius Referrals Custom Portals (Clients Websites)

Aug 27, 21:36:01 GMT+0
Postmortem - ### Summary Earlier today, some advocate portal templates experienced an issue that prevented accounts from fully loading. The issue was introduced after a system update. To quickly restore service, we rolled back to the previous application version. This resolved the problem and advocate portals are now working as expected. ### Next Steps Our team is continuing to investigate the update to identify the cause and ensure stronger safeguards are in place for future releases. We appreciate your patience and understanding, and we remain committed to providing a reliable experience..

Aug 27, 19:05:00 GMT+0
Resolved - This incident has been resolved..

Aug 27, 18:30:00 GMT+0
Investigating - We identified an issue affecting some advocate portal templates where advocate accounts may not fully load as expected. Our engineering team is actively investigating the root cause and working on a resolution. While this issue may temporarily impact the display of advocate account information, all referral data and account details remain secure and unaffected. We will provide updates as progress is made and appreciate your patience as we work to restore full functionality. We are currently investigating this incident..

Duration: 5 hours and 6 minutes

Affected Components: Genius Referrals Integration on Zapier (Zapier App), Genius Referrals Developer API (RESTful API), ReferralTrack Insights (Tracking Services), Genius Referrals Custom Portals (Clients Websites)

Aug 9, 16:39:56 GMT+0
Postmortem - Following a Redis database upgrade, our system encountered connection issues that caused service disruptions for some users. The issue was identified and resolved by flushing the Redis database. Services were fully restored shortly after. We are implementing additional checks after future upgrades to prevent similar incidents..

Aug 9, 06:19:43 GMT+0
Resolved - This incident has been resolved..

Aug 9, 03:28:33 GMT+0
Identified - We are continuing to work on a fix for this incident..

Aug 9, 01:13:25 GMT+0
Investigating - We are currently experiencing an issue with our authentication service that is affecting the login and sign-up functionality on certain Advocate Portals. Our team is actively investigating the root cause to address the problem. Rest assured, we are fully committed to resolving this issue as quickly as possible to ensure a seamless experience for all our valued customers. We apologize for any inconvenience this may cause and appreciate your understanding. We will provide regular updates on our progress until the issue is fully resolved. If you have any questions or need further assistance, please don't hesitate to contact our support team. Thank you for your patience and cooperation..

Duration: 1 day and 26 minutes

Dec 12, 02:35:57 GMT+0
Resolved - This incident has been resolved..

Dec 11, 23:18:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Dec 11, 21:00:29 GMT+0
Identified - We are continuing to work on a fix for this incident..

Dec 11, 19:51:53 GMT+0
Identified - We are continuing to work on a fix for this incident..

Dec 11, 16:20:28 GMT+0
Identified - The issue has been identified, and the team is actively working on a solution..

Dec 11, 02:10:00 GMT+0
Investigating - **Description of the Issue:** Some clients are reporting that they are unable to save email notifications in the admin portal. This issue appears to affect multiple users, but the scope and impact are still being determined. **Current Status:** The issue is under investigation. **Impact:** Clients may experience difficulties saving email notifications, which could potentially disrupt their operations or communication workflows. **Steps Taken So Far:** 1. Acknowledged the issue and logged initial client reports. 2. Notified relevant technical teams. 3. Began investigation to identify the root cause. **Next Steps:** 1. Gather additional information from affected clients (e.g., specific error messages, browser details, etc.). 2. Review admin portal logs to pinpoint potential errors or anomalies. 3. Collaborate with the development and support teams to diagnose and resolve the issue. **Estimated Time for Resolution:** 48hrs.

Duration: 42 minutes

Affected Components: ReferralRewards Management (Rewards Services) , Genius Referrals Hub (Website), CashFlowPlus: Efficient Payout Management, SiteCraft: Dynamic Website Editor, Genius Referrals Portal (App) , Genius Referrals Custom Portals (Clients Websites), Referral Redirect Engine (Redirection Service), Genius Referrals Integration on Zapier (Zapier App), Genius Referrals Developer API (RESTful API), Genius Referral Integration for Shopify (Shopify App) , ReferralTrack Insights (Tracking Services), Genius Referrals Notification Engine (Notification Service)

Jul 20, 13:35:16 GMT+0
Resolved - This incident has been resolved..

Jul 20, 13:35:04 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Jul 20, 12:53:39 GMT+0
Investigating - We are currently experiencing issues with some of our services and are actively investigating the cause. Our team is working diligently to identify and resolve the problem as quickly as possible. We apologize for any inconvenience this may cause and appreciate your patience. We will provide updates as soon as we have more information. Thank you for your understanding. Genius Referrals Team.

Duration: 1 hour

Affected Components: Genius Referrals Portal (App) , Genius Referrals Custom Portals (Clients Websites), Referral Redirect Engine (Redirection Service), Genius Referrals Integration on Zapier (Zapier App), Genius Referrals Developer API (RESTful API), Genius Referral Integration for Shopify (Shopify App) , ReferralTrack Insights (Tracking Services), Genius Referrals Notification Engine (Notification Service)

Nov 18, 05:00:00 GMT+0
Completed - Maintenance has completed successfully.

Nov 18, 04:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 18, 04:00:00 GMT+0
Identified - Dear valued clients, We want to inform you of scheduled maintenance that will impact our services. On November 17, 2023, we will be performing essential infrastructure changes, necessitating a complete outage in all our services for at least one hour. This maintenance is crucial to enhance the reliability and performance of our platform. **Maintenance Schedule:** - Date: November 17, 2023 - Start Time: 11:00 PM EST - Expected Duration: At least one hour During this time, all services, including access to our portal, will be temporarily unavailable. We apologize for any inconvenience this may cause and appreciate your understanding as we work to improve our services. Thank you for your continued support. Sincerely, Genius Referrals.

Duration: 1 hour

Affected Components: Referral Redirect Engine (Redirection Service), Genius Referrals Custom Portals (Clients Websites), Genius Referrals Portal (App) , Genius Referrals Integration on Zapier (Zapier App), Genius Referrals Developer API (RESTful API), Genius Referral Integration for Shopify (Shopify App) , ReferralTrack Insights (Tracking Services), Genius Referrals Notification Engine (Notification Service)

Oct 7, 04:00:00 GMT+0
Completed - Maintenance has completed successfully.

Oct 7, 03:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 7, 03:00:00 GMT+0
Identified - Dear valued clients, We want to inform you of scheduled maintenance that will impact our services. On October 6, 2023, we will be performing essential infrastructure changes, necessitating a complete outage in all our services for at least one hour. This maintenance is crucial to enhance the reliability and performance of our platform. **Maintenance Schedule:** - Date: October 6, 2023 - Start Time: 11:00 PM EST - Expected Duration: 11:59 PM EST During this time, all services, including access to our portal, will be temporarily unavailable. We apologize for any inconvenience this may cause and appreciate your understanding as we work to improve our services. Thank you for your continued support. Sincerely, Genius Referrals.

Duration: 54 minutes

Affected Components: Genius Referrals Custom Portals (Clients Websites)

Aug 4, 19:17:53 GMT+0
Resolved - This incident has been resolved..

Aug 4, 18:47:20 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

Aug 4, 18:23:31 GMT+0
Investigating - We are currently experiencing an issue with our authentication service that is affecting the login and sign-up functionality on certain Advocate Portals. Our team is actively investigating the root cause to address the problem. Rest assured, we are fully committed to resolving this issue as quickly as possible to ensure a seamless experience for all our valued customers. We apologize for any inconvenience this may cause and appreciate your understanding. We will provide regular updates on our progress until the issue is fully resolved. If you have any questions or need further assistance, please don't hesitate to contact our support team. Thank you for your patience and cooperation..

Duration: 39 minutes

Affected Components: Genius Referrals Hub (Website)

Feb 24, 16:37:51 GMT+0
Resolved - We just resolved the issue!.

Feb 24, 15:58:43 GMT+0
Investigating - The Genius Referrals Marketing Website is currently unavailable. We are working to resolve this as quickly as possible..

Duration: 5 hours and 24 minutes

Affected Components: Genius Referrals Notification Engine (Notification Service)

Dec 8, 21:41:58 GMT+0
Resolved - We just resolved the issue!.

Dec 8, 20:09:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Dec 8, 16:18:14 GMT+0
Investigating - We have identified delays in message delivery and are actively investigating. Messages are currently queued and will be delivered once we resolve the issue..

Duration: 3 hours and 22 minutes

Affected Components: Genius Referrals Notification Engine (Notification Service)

Sep 21, 16:35:19 GMT+0
Resolved - We just resolved the issue!.

Sep 21, 16:24:12 GMT+0
Monitoring - We implemented a fix and currently monitoring the result..

Sep 21, 15:29:12 GMT+0
Investigating - We are currently investigating this incident..

Sep 21, 13:12:57 GMT+0
Investigating - We have identified delays in message delivery, and are actively investigating. Messages are currently queued and will be delivered once we resolve the issue..

Duration: 32 minutes

Affected Components: Genius Referrals Custom Portals (Clients Websites), Genius Referrals Portal (App)

Sep 1, 02:54:41 GMT+0
Resolved - The Genius Referrals Admin Portal and Clients Websites were down from 10:23 to 10:35 PM EST. A rollback was issued, and this problem is now completely resolved..

Sep 1, 02:35:00 GMT+0
Monitoring - The rollback is complete, and the Admin Portal and Clients Websites are operational again. We are monitoring the situation to confirm full recovery..

Sep 1, 02:28:00 GMT+0
Identified - We have identified the issue and are rolling back..

Sep 1, 02:23:00 GMT+0
Investigating - We are aware of problems loading the Admin Portal and Clients Websites. We are investigating and will update here as soon as possible..

Duration: 6 hours

Affected Components: Genius Referrals Portal (App)

Aug 22, 03:16:00 GMT+0
Resolved - A bad database query caused Genius Referrals degradation from 20:53 UTC to 2:35 UTC. This is now fully resolved. .

Aug 22, 02:45:00 GMT+0
Monitoring - The rollback is complete, and the Admin Portal is fully operational again. We are monitoring the situation to confirm full recovery. .

Aug 22, 02:16:00 GMT+0
Monitoring - We have identified the issue and are rolling back..

Aug 22, 01:16:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Aug 22, 00:16:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Aug 21, 23:16:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Aug 21, 22:16:00 GMT+0
Identified - We are continuing to work on a fix for this incident..

Aug 21, 21:16:00 GMT+0
Investigating - We are aware of problems loading some pages on the Admin Portal. We are investigating and will update here as soon as possible. .

Duration: 19 hours and 58 minutes

Affected Components: Genius Referrals Portal (App)

May 16, 20:22:31 GMT+0
Resolved - We just resolved the issue!.

May 16, 18:27:14 GMT+0
Monitoring - We implemented a fix and currently monitoring the result..

May 16, 14:17:08 GMT+0
Monitoring - We implemented a fix and currently monitoring the result. RESTful API and Clients' Websites Services are back to normal. .

May 16, 06:07:08 GMT+0
Identified - We are continuing to work on a fix for this incident..

May 16, 00:24:54 GMT+0
Investigating - The Genius Portal and our Client's Portal are currently experiencing a higher than normal amount of load, and may be causing pages to be slow or unresponsive. We're investigating the cause and will provide an update as soon as possible. .

Duration: 5 hours and 29 minutes

Affected Components: Genius Referrals Hub (Website), Genius Referrals Portal (App)

May 15, 20:58:47 GMT+0
Resolved - We just resolved the issue!.

May 15, 19:57:07 GMT+0
Monitoring - We implemented a fix and currently monitoring the result..

May 15, 18:57:30 GMT+0
Identified - We are continuing to work on a fix for this incident..

May 15, 17:57:14 GMT+0
Identified - We are continuing to work on a fix for this incident. The API is currently experiencing a higher than normal amount of load, and may be causing pages to be slow or unresponsive. We're investigating the cause and will provide an update as soon as possible. .

May 15, 16:49:56 GMT+0
Monitoring - Our OPS team is working to identify the root cause and implement a solution..

May 15, 15:29:47 GMT+0
Identified - Some customers are seeing intermittent error messages throughout their accounts. We’re aware of the issue and are working on it urgently. We’re really sorry to be holding you up today! Please know our engineering and operations teams are working hard to get everything up and running, and we will update you right here in 60 minutes with the latest information. .