Verbit Status Updates
This page will detail status updates and technical bulletins which may affect our caption customers. If you have any questions or concerns, please contact verbitmarketing@verbit.ai.
Recent Updates
2/14 – Caption Outage (1:44 pm ET)
VITAC experienced an outage of our Internet Captioning Service (ICS) on Thursday, Feb. 13. Our team quickly identified the source of the problem – a sudden spike of unexpected data in our Redis database – and worked with Redis Labs to remove the problematic data and restore caption delivery functionality.
We continued to caption all events during the outage and will deliver post-event files and transcripts as requested. Please note that this outage only affected ICS bookings. If you also booked Zoom or scoreboard captioning, those captions were delivered without error.
We will be closely monitoring events to ensure there are no further issues.
Previous Updates
7/19/24 – CrowdStrike Outage (2:30 pm ET)
VITAC was one of many companies around the globe impacted by the widespread CrowdStrike outage. Multiple captioning computers were affected by the outage, resulting in lost connections, as well as one server that handles company applications.
You are receiving this email because we have identified you among those affected by this outage.
We have been able to restore many of our captioning workstations and will continue to closely monitor our systems and the outage throughout the day.
4/8/24 – iCap Captioning Instability (2:40 pm ET)
VITAC’s Realtime Technical Services (RTS) team identified caption stability issues across the iCap network on Thursday, April 4, and Friday, April 5, that impacted several clients. The instability was traced to Ai-Media, which reported intermittent issues with audio and caption delivery due to an improperly balanced load on the relay system. Per EEG, the issue has been resolved (https://statuspage.eegcloud.tv/incidents/2vrl1v3vw17t).
Please contact Ai-Media with any questions: https://www.ai-media.tv/company/contact-us/.
4/3/24 – Caption Outage (2:45 pm ET)
VITAC’s Realtime Technical Services (RTS) team identified a caption outage on Tuesday, April 2, at 5:58 PM ET that impacted several clients. The caption loss was traced to a maintenance update at Amazon Web Services (AWS) that caused a brief outage in our Control Tower caption management system. Caption service was restored at 6:05 PM ET.
3/21/24 -Technical Outage, ICS Captioning Loss (Updated 10 pm ET)
We have corrected the issue causing the outage of our Internet Captioning Service (ICS) and the problem is resolved. ICS will be up and running by Friday morning. Thank you for your patience, and thank you for working with VITAC.
3/21/24 -Technical Outage, ICS Captioning Loss (Updated 5:40 pm ET)
We continue to be experiencing outages impacting our captionedtext website and ICS service. We have sourced the problem as an expired security certificate and unannounced upgrade by our real-time data platform provider. We are working with affected customers on short-term alternative solutions. We expect, but cannot guarantee, the problem to be resolved by Friday morning. We will provide additional updates as they become available.
3/21/24 -Technical Outage, ICS Captioning Loss (2:25 pm ET)
VITAC is experiencing a technical outage impacting our captionedtext website and ICS service, which delivers captions to customer URLs. Our technical team is working to identify the cause and rectify the situation. We are continuing to caption events and will provide post-event deliverables as ordered. Additional information will be shared once we’ve resolved the issue.
2/12/24 – iCap Captioning Instability (Updated 4:55 pm ET)
VITAC’s Realtime Technical Services (RTS) team identified caption stability issues across the iCap network on Sunday, Feb. 11, that impacted several clients. The instability was traced to Ai-Media, which reported several of the platform’s users created excessive load on the system. Per EEG, the issue has been resolved (https://statuspage.eegcloud.tv/incidents/xgg1zrmx6d2s).
Please contact Ai-Media with any questions: https://www.ai-media.tv/company/contact-us/.
11/10/23 – 11/7 Remediation Complete
Remediation of the 11/7 event is complete. Please reach out to your Account Executive or Account Manager for more information.
11/7/23 – Captioning Outage (Updated 8:15 pm ET)
VITAC’s Canonsburg, PA office has returned to full capacity.
We identified the source as our Uninterruptible Power Supply (UPS), which did not fail over during the power outage. That failure took our VxRail virtual servers offline. We worked with vendors to restore connectivity and are now back in production. We continue to investigate the root cause of the outage and will provide additional updates.
11/7/23 – Captioning Outage (Updated 5:25 pm ET)
VITAC’s Canonsburg, PA, office experienced a power outage this afternoon due to a power surge in the community. We’ve rerouted our work to other VITAC offices, but customers have been affected.
Our teams are actively working to restore connectivity and will provide an update as soon as possible.
10/2/23 – iCap Captioning Outage (Updated 2:50 pm ET)
VITAC’s Realtime Technical Services (RTS) team identified caption outages between 6:45-7:40 AM ET on Monday, Oct. 2, that impacted several clients. The outage was a result of connection issues at EEG, which have been resolved (https://status.icap.eegcloud.tv/).
Please contact EEG with any questions: https://www.ai-media.tv/company/contact-us/.
9/15/23 – iCap Captioning Outage (Updated 9:05 am ET)
VITAC’s Realtime Technical Services (RTS) team identified a caption outage on Wednesday, Sept. 13, at 11:50 PM ET that impacted several clients. The outage was a result of an iCap connection stability issue at EEG (https://statuspage.eegcloud.tv/). Services were restored at approximately 12:25 AM ET.
Please contact EEG with any questions: https://www.ai-media.tv/company/contact-us/.
We’d also like to take this moment to kindly remind our partners that we suggest backup connections remain available for use in such situations. We would be happy to coordinate testing of backup connections at your convenience.
6/22/23 – Live Captioning Outage (Updated 9:00 pm ET)
On Thursday, June 22, between 3:30 and 5:20 pm ET, several VITAC live captioning customers were affected by an outage tied to our HAProxy load balancer. We were alerted that the load balancer was unstable and rebooted it. The system unexpectedly returned with a different IP address – it is set to a static IP and we’ve rebooted before without issue.
- We reconnected our Scheduler, Intranet, and other programs to the new DNS address and rebooted systems across the company.
- Outage time varied by customer and captioner but does not appear to be significant.
- We are meeting with our solutions architect to understand why this occurred and ensure it does not happen again, and will provide further updates as they are available.
6/15/23 – Data Center Migration (Updated 11:00 am ET)
We are upgrading our physical network connections, Wi-Fi, and other equipment within the data center. All updates feature the newest and most up-to-date technology and are configured with the industry best practices for redundancy.
During scheduled maintenance periods:
- Customers will be contacted by Realtime Technical Services (RTS) to arrange maintenance windows with backup connectivity if necessary.
- Customers using iCap will not be affected.
The work is expected to be completed by June 26.
6/12/23 – Phone System Modernization (Updated 10:00 am ET)
VITAC’s operations department will be moving to a new internet-based phone system, enabling customers to better reach our support team. The new platform promises increased reliability and improved integration and tracking, allowing us to more easily track any issues and outages.
Inbound calls to VITAC:
Please continue to contact us using the same phone numbers you have always used to contact us.
During very busy times, you will have an option to leave a voicemail or wait for the next available employee. Please do not hesitate to leave a message. Your message will be routed immediately with urgent status to the entire RTS team, and you can expect a reply ASAP.
Outbound calls from VITAC
Note that while we port our numbers over to the new system, calls from VITAC to you will appear different than they have in the past. You will see these numbers on your phone:
- RTS Colorado: 833-540-0552
- RTS Pittsburgh: 314-332-1761
- Sports Supervisors: 332-230-2081
- Client Sales and Services (CSS): 724-401-4468