As of 9:00am Eastern time Monday Jan 8 the system continues to perform as expected. All issues related to this incident have been resolved.
If you have any questions regarding this incident, please contact support@validic.com.
Thank you,
Validic Technical Support
Posted Jan 08, 2018 - 09:38 EST
Monitoring
As of 8:30pm Eastern time Sunday the issue is resolved.
Earlier today, Validic engineers implemented several additional updates designed to further offset the performance challenges created by the patches needed at Amazon Web Services to address the Meltdown and Spectre security vulnerabilities. The updates were successful and our system is now performing normally. All realtime updates are processing as expected and all data that was delayed or missed earlier has been captured.
We will continue to monitor the progress of the system overnight and send a final notice Monday morning Eastern time unless there are any new issues to report.
Thank you for your patience as we worked through this issue.
Validic Technical Support
Posted Jan 07, 2018 - 20:34 EST
Identified
Recently, numerous worldwide computer systems were affected by the Meltdown and Spectre security vulnerabilities. Like many hosting providers, Amazon Web Services (AWS) has patched servers that potentially contained those security holes. Like most other service providers AWS have seen some performance degradation in the patched servers. Validic utilizes Amazon Web Services and our engineers believe performance loss since the servers have been patched is the primary cause of the recent Fitbit latency issues.
Although Validic is continuing to process Fitbit data, some of the records are delayed. To overcome any performance issues caused by the security fix our engineers have already implemented software changes designed to lessen the demand on the affected servers. These updates have already helped minimize the impact of the issue. In addition our engineers are actively adding additional server capacity, rewriting certain routines to take advantage of the new servers and updating scheduled process to more efficiently utilize the new processes.
We do not expect any data loss related to this issue. However, given this issue is causing significant delays in some data records, Validic is treating this with high urgency.
We will continue to provide further updates via this status page until we have verified complete resolution. Another update will be provided no later than 10:00pm Eastern time.
Thank you,
Validic Technical Support
Posted Jan 07, 2018 - 14:23 EST
Update
As of 8:30am Eastern time, the Fitbit data catchup process is continuing to execute. Much of the records not retrieve on Jan 4 and 5 have been restored. Engineers are currently taking additional steps to increase the speed of processing to ensure all records are caught up as quickly as possible.
We will provide another update this this site no later than 4:00pm Eastern time.
Thank you,
Validic Technical Support
Posted Jan 07, 2018 - 08:43 EST
Update
As of 9:00pm Eastern time Saturday Jan 6 all new incoming Fitbit data is being handled normally. Validic engineers are working to finalize the capture of some Fitbit data there were either delayed or not retrieved on Jan 4 and 5. This process was originally intended to be complete by 11:00pm Eastern time today. However, we now expect this process to take several hours longer.
We will provide another update no later than 9:00am Eastern time Sunday Jan 7.
Thank you,
Validic Technical Support
Posted Jan 06, 2018 - 21:17 EST
Update
New incoming Fitbit data continues to be processed normally. A reconciliation process is currently running to detect and recapture any data that were delayed or missing. It is expected the reconciliation process will be fully complete by midnight Eastern time this evening.
We will provide further updates on this status page.
Thank you,
Validic Technical Support
Posted Jan 06, 2018 - 16:07 EST
Monitoring
Engineers have determined the root cause of the issue and have put a fix in place. As of 10:15PM ET new incoming data is processing normally. For data that was previously delayed, it may take several hours before everything is caught up.
We will continue to monitor the system and provide updates via this status page.
Thank you for your patience,
Validic Technical Support
Posted Jan 05, 2018 - 22:22 EST
Investigating
Validic engineers have determined Fitbit data retrieval is occurring more slowly than normal. This issue may affect new and historical data. Validic is currently investigating the issue and will provide updates as they become available.
If you have questions regarding this issue, please contact support@validic.com.