Update - Update - A current summary analysis can be found here: https://community.khoros.com/t5/Support-Forum/Social-Marketing-Approvals-Service-Update/td-p/547877
Oct 10, 09:06 PDT
Monitoring - Monitoring - While the Khoros Social Marketing Engineering team have processed posts affected by Approvals Workflow degradation, some existing messages may still appear stuck in approval status, and may not appear in the Approvals tab.

Any scheduled message will transition to Error status automatically if it has passed its scheduled time by more than one hour.
Oct 8, 16:31 PDT

About This Site

Khoros monitors all of its critical services both internally and through 3rd party services. This page reports on known infrastructure wide incidents affecting Khoros platforms. If you're experiencing an availability or a performance issue that isn't shown here, please Contact Support. Subscribing to updates is the best way to stay informed about any issues affecting Khoros platform availability.

Khoros Communities Operational
AMER & APAC Community Datacenter ? Operational
EMEA Community Datacenter ? Operational
LSI ? Operational
API Proxy ? Operational
CDN ? Operational
Community Mail ? Operational
AWS Community Hosting ? Operational
AWS CDN (CloudFront) ? Operational
Khoros Care Operational
AMER & APAC SMM Datacenter Operational
EMEA SMM Datacenter Operational
SMM Analytics Operational
Facebook API ? Operational
Twitter API ? Operational
Other APIs ? Operational
AMER SMM CRM Connector Operational
EMEA SMM CRM Connector Operational
Stage SMM CRM Connector Operational
SMM and Community Integration ? Operational
WeChat Operational
WhatsApp Operational
Khoros Marketing Degraded Performance
Conversations - APP Degraded Performance
Conversations - VPC Degraded Performance
Conversations - APP3 Degraded Performance
Conversations - Financial Export Service Operational
Intelligence Operational
Experiences Operational
Vault Operational
Promotions Operational
Khoros Corporate Operational
Khoros Community ? Operational
Khoros Case Management & Portal Operational
Khoros.com Website ? Operational
Khoros Developer Documentation Operational
JX Communities Operational
JX Amsterdam (AMS) Datacenter Operational
JX Bunchball Operational
JX Phoenix (PHX) Datacenter Operational
JX Streamonce Operational
JX Cloud Analytics Operational
JX Cloud Search Operational
JX Rewards Operational
JX Video Operational
JX Platform Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Oct 17, 2019

No incidents reported today.

Oct 16, 2019

No incidents reported.

Oct 15, 2019

No incidents reported.

Oct 14, 2019

No incidents reported.

Oct 13, 2019

No incidents reported.

Oct 12, 2019

No incidents reported.

Oct 11, 2019

No incidents reported.

Oct 9, 2019
Resolved - The issue is resolved and DES Data is current now.
Oct 9, 00:55 PDT
Monitoring - The PHX instance of DES is accessible again. Data may not be current as the system works to catch up to the present on the activities performed during the downtime.
Oct 8, 07:25 PDT
Investigating - Attempts to log in to the PHX instance of DES is resulting in login errors, returning 504 responses. We have escalated the issue with the proper resources and are working to have this brought back as soon as possible.
Oct 7, 07:28 PDT
Oct 8, 2019
Resolved - This incident has been resolved. We are caught up with message processing. Accordingly, new messages should have approvals applied properly and approvals should be visible in the requisite tabs.
Oct 8, 14:29 PDT
Identified - Khoros Social Marketing Engineering team have identified and fixed the initial root cause affecting Workflow Approvals, though some existing messages may still appear stuck in approval status, and may not appear in the Approvals tab.
While we work to re-establish missing components of existing messages, newly created messages may continue to appear affected by other residual issues, which we are actively working to resolve.
Any scheduled message will transition to Error status automatically if it has passed its scheduled time by more than one hour.
Oct 8, 13:06 PDT
Oct 7, 2019
Resolved - This incident has been resolved.
Oct 7, 09:16 PDT
Monitoring - We are continuing to monitor Twitter CSAT surveys. We believe it has been resolved on Twitter's end but will continue to monitor this issue over the weekend.
Oct 4, 12:17 PDT
Identified - We have identified an issue with NPS and CSAT surveys not being successfully submitted or sent via iOs and Android. We are notifying our partners at Twitter about these issues directly. We will update as we receive additional information.
Oct 4, 10:55 PDT
Oct 6, 2019

No incidents reported.

Oct 5, 2019

No incidents reported.

Oct 3, 2019

No incidents reported.

Oct 2, 2019
Resolved - All twitter APIs have recovered and should be operating normally.
Oct 2, 08:08 PDT
Investigating - Twitter API is currently experiencing a disruption in service. Khoros conversations may see delays with incoming Twitter content or may experience error messages while attempting to respond to Twitter messages.

https://api.twitterstat.us/incidents/vz25tls0vk4d
Oct 2, 01:38 PDT
Resolved - This issue has been resolved.
Oct 2, 06:48 PDT
Monitoring - The reconciliation of older messages has been completed, and we believe this is now resolved. We are continuing to monitor on our side. Please notify Support if you are still seeing any issues.
Oct 1, 17:15 PDT
Update - You should no longer see Workflow Approval issues with any message created after 5pm Central Time.

We continue to reconcile older messages, and anticipate the majority will be restored within the next 6-8 hours.
Oct 1, 15:53 PDT
Update - We have identified and fixed the initial root cause affecting Workflow Approvals, though some existing messages may still appear stuck in approval status, and may not appear in the Approvals tab.

While we work to re-establish missing components of existing messages, newly created messages may continue to appear affected by other residual issues, which we are actively working to resolve.

Any scheduled message will transition to Error status automatically if it has passed its scheduled time by more than one hour.
Oct 1, 12:39 PDT
Identified - Khoros Social Marketing Engineering and Operations teams have currently identified an issue with Workflow Approvals, preventing posts from publishing due to a blocker in the aforementioned approvals system. All appropriate resources are being applied to resolve this as quickly as possible. As we learn more or resolve this issue we will update this status page with further information
Oct 1, 11:11 PDT
Oct 1, 2019
Completed - The scheduled maintenance has been completed.
Oct 1, 22:27 PDT
Update - Lithium Platform maintenance has concluded. All systems are currently operational, there are no known issues at this time.

We thank you for your support and look forward to continuously improving our services.

Sincerely,
Khoros Team
Oct 1, 22:18 PDT
Update - Lithium Platform maintenance has concluded. All systems are currently operational, there are no known issues at this time.

We thank you for your support and look forward to continuously improving our services.

Sincerely,
Khoros Team
Oct 1, 22:17 PDT
Update - Lithium Platform maintenance has concluded. All systems are currently operational, there are no known issues at this time.

We thank you for your support and look forward to continuously improving our services.

Sincerely,
Khoros Team
Oct 1, 22:08 PDT
Update - Scheduled maintenance is still in progress. We will provide updates as necessary.
Oct 1, 22:05 PDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Oct 1, 16:00 PDT
Scheduled - We will have an extended European maintenance window on October 1st to move certain infrastructure pieces to our new public cloud environment. During this window, the following functionality will be unavailable for use:

- SDK
- Studio
- Studio publish

This impacts both stage and production use of the above features. After maintenance has concluded, these features will be available for normal use - and you may notice some performance improvements related to the SDK, Studio, and Studio Publishing.

Maintenance details below:

Region: EMEA
Date: Tuesday, October 1st, 2019
Time: 4pm – 12am PT / 11:59pm – 8am UTC Standard (10/1 – 10/2)
Platform: Khoros Community
Duration: 8 hours
Impact: Studio, Studio Publishing and SDK unavailable

We apologize for the inconvenience, and understand that these upgrades will have a short term impact on customization productivity, but this is a necessary step for improving the customizing and publishing experience going forward. We thank you for your patience and understanding during this time.

If you have any questions or concerns, please contact support@lithium.com.

-The Khoros Team
Sep 18, 13:13 PDT
Sep 30, 2019

No incidents reported.

Sep 29, 2019

No incidents reported.

Sep 28, 2019

No incidents reported.

Sep 27, 2019

No incidents reported.

Sep 26, 2019
Resolved - The Khoros Care September Release included a fix to address the issue related to slow refresh times for the agent queue.
Sep 26, 10:11 PDT
Update - Our engineering is still working on a fix that improves the performance further.
Sep 20, 08:15 PDT
Update - A partial fix has been deployed late last night. Now, agents will no longer need to refresh the page however, it may take up to 45 seconds for their screens to update.
Our team will continue to work towards a complete fix and eliminate that delay.
Sep 10, 08:57 PDT
Identified - The issue has been identified and a fix is being implemented.
Sep 7, 09:51 PDT
Investigating - We identified an issue related to care's agent view and browsers not being in focus. We are working on a resolution and deployment.
Sep 6, 14:02 PDT
Sep 25, 2019

No incidents reported.

Sep 24, 2019
Resolved - This incident has been resolved by our Care analytics team. Any temporary lingering issues should clear up shortly as our systems catch up.
Sep 24, 15:54 PDT
Investigating - We are currently investigating reports of Care analytics failing to load.
Sep 24, 15:17 PDT
Sep 23, 2019
Resolved - A fix was deployed the evening of 9/20/2019, and data should be collected in LSI from 9/20 onward.
Sep 23, 10:15 PDT
Identified - The root cause has been identified, and a fix is out of QA. We're now in the process of backporting that fix to test against all current versions before pushing it out to impacted customers.
Sep 18, 08:54 PDT
Investigating - We are aware and investigating a potential issue where some data isn't populating in LSI after a certain date. It appears data is not shown from 9/13/2019 onward, but this date may differ for customers. More updates to come as we investigate further.
Sep 17, 15:15 PDT
Sep 22, 2019

No incidents reported.

Sep 21, 2019

No incidents reported.

Sep 19, 2019

No incidents reported.

Sep 18, 2019
Resolved - This incident has been resolved.
Sep 18, 08:46 PDT
Monitoring - The issue has been fixed now. We are still monitoring the affected communities.
Sep 18, 07:25 PDT
Identified - The issue has been identified and the fix is being implemented.
Sep 18, 07:01 PDT
Update - Our engineering team is still investigating the issue. We will keep you posted with further updates.
Sep 18, 06:45 PDT
Investigating - We are seeing multiple communities failing to load with 502 CloudFront error. We are currently investigating this issue and will keep posting the updates.
Sep 18, 05:42 PDT
Completed - The scheduled maintenance has been completed.
Sep 18, 00:00 PDT
Update - Due to unexpected discoveries, we've decided not to proceed with the migration any further and are in the process of reverting all work. Once a new date has been determined for the migration, we'll reach out to customers directly with all the details.

We estimate the rollback will be completed within ~3 hours, at which point publishing functionality will be restored.
Sep 17, 18:35 PDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Sep 17, 16:00 PDT
Scheduled - We will have an extended European maintenance window on September 17th to move certain infrastructure pieces to our new public cloud environment. During this window, the following functionality will be unavailable for use:

- SDK
- Studio
- Studio publish

This impacts both stage and production use of the above features. After maintenance has concluded, these features will be available for normal use - and you may notice some performance improvements related to the SDK, Studio, and Studio Publishing.

Maintenance details below:

Region: EMEA
Date: Tuesday, September 17th, 2019
Time: 4pm – 12am PT / 11:59pm – 8am UTC Standard (9/17 – 9/18)
Platform: Khoros Community
Duration: 8 hours
Impact: Studio, Studio Publishing and SDK unavailable

We apologize for the inconvenience, and understand that these upgrades will have a short term impact on customization productivity, but this is a necessary step for improving the customizing and publishing experience going forward. We thank you for your patience and understanding during this time.

If you have any questions or concerns, please contact support@lithium.com.

-The Khoros Team
Aug 29, 07:14 PDT