PLEASE READ FULLY IF YOU SEE "QUEUED" EXPENSE REPORTS IN YOUR EXPORT PAGE
While the syncing and exporting outage for NEW syncs and exports has been verified as resolved, there is a significant set of expense reports that reached the "queued" status during this outage. Some of these reports have been successfully exported to QuickBooks while others have not.
The update to this status is dependent on Tallie receiving and processing a message from QuickBooks that indicates an export was successful. Due to the outage, any success message that was sent by QuickBooks during our outage was missed, and thus, we have no way of programmatically determining whether a report showing this state in Tallie has been successfully exported. Additionally, because we have no way of determining whether the export was successful, re-exporting any data that was previously exported successfully will result in duplication of data in QuickBooks.
Any attempt at programmatic resolution of this would result in one of the following scenarios for our customers: - Programmatically marking these reports as "Ready for export" would result in duplicated data in QuickBooks for any report that was previously successfully exported and exported again. - Programmatically marking these reports as "Exported" would require a customer to dig through their "exported" reports and QuickBooks to determine which of them were successfully exported or not and update them accordingly.
As such, we have elected to leave these reports in their "Queued" state. We STRONGLY RECOMMEND for any customer experiencing this issue to reach out to our support team for assistance in rectifying this situation.
Please note that our support team is acutely aware of this situation and prepared to handle these circumstances, however, this is a wide-ranging issue that affects many customers, and thus resolution on a customer-by-customer basis will take a bit of time. Please continue to be patient with them as they work through resolving the fallout of this outage.
Finally, thank you again for your patience as we have worked through this outage. We understand deeply that it was and continues to be a very impactful outage and are working to ensure we avoid such outages in the future.
Posted Aug 22, 2022 - 12:36 PDT
Update
We are seeing evidence of strong progress toward resolution of this issue, but have not yet reached full resolution.
Currently, it appears that syncing between both QuickBooks Online and QuickBooks Desktop is working, however, exporting data from Tallie to QuickBooks appears to still be a work in progress.
We have evidence to suggest that in some cases, exports that were stuck in the "queued" status have successfully exported, and their statuses in Tallie have been updated to reflect this completed status appropriately. There is also evidence of some reports exporting successfully after having been stuck in "queued" without updating appropriately in the user interface.
We are expecting that in many cases, the user interface will continue to update appropriately, in which case, there will be no need for further intervention.
If, however, all reports in a given export have successfully exported to QuickBooks, but the status is still showing "queued" in Tallie, those cases will require intervention by our team to set the status appropriately.
We are continuing to monitor this incident closely.
Posted Aug 22, 2022 - 09:12 PDT
Update
As of 9pm Pacific Time on Sunday, August 21, the previously referenced queue of messages has processed approximately 80% of the backlog.
The current pace suggests that we should be able to get through the queue of messages around the time we expect our customers in the Eastern Time Zone to begin logging on Monday morning.
We are optimistic that this means normal service for our QuickBooks users can resume with the start of the new work week, but we will continue monitoring the situation and provide updates as necessary.
Posted Aug 21, 2022 - 21:11 PDT
Identified
We have identified what we believe to be the root cause of this outage and are continuing to work through the backlog of messages referenced in our previous updates.
The current backlog of messages is diminishing at a significantly faster rate than previously, but we are working to make it even faster and are working to fully resolve this outage prior to the new work week's business hours.
We will provide another update as we approach the work week. Thank you for your patience.
Posted Aug 21, 2022 - 11:10 PDT
Update
The message backlog referenced in yesterday's update is continually decreasing, however, the speed at which the backlog is decreasing leads us to expect the observed issues to persist for most, if not all, customers that have experienced syncing and exporting issues to this point.
We are still investigating the root cause of the outage and attempting to provide a fix that will resolve the cause rather than just the symptoms.
We will continue to provide updates throughout the weekend.
Posted Aug 19, 2022 - 15:41 PDT
Update
We have identified that the issues manifesting for customers are caused by a severe backlog of messages that are being processed by our systems. Our previous update had the intended effect of diminishing this backlog, however, the rate at which the queue is clearing currently is at a slower pace than we had hoped for.
We will be shipping code changes later today that we expect to expedite the timeline for processing this backlog of messages, We will monitor the speed at which these messages begin processing after the change is made and provide an expected timeline of resolution based on that.
With that in mind, we have not yet identified the cause for the buildup of these messages in the first place.
As such, along with this change, we will also be adding more logging around the processes that appear to be leading to this backlog in hopes that it will result in an understanding of the root cause, so that we can avoid any such issues in the future.
Again, thank you for your patience as we work to resolve this outage.
Posted Aug 18, 2022 - 17:56 PDT
Investigating
The patch we applied was unsuccessful at resolving this outage. We are continuing to investigate and will attempt to apply additional fixes soon. Thank you for your patience.
Posted Aug 17, 2022 - 17:16 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Aug 17, 2022 - 15:26 PDT
Investigating
We have received several reports of syncing and exporting failing for both QuickBooks Online and QuickBooks Desktop. This issue is currently under investigation.
Posted Aug 12, 2022 - 11:36 PDT
This incident affected: QuickBooks Desktop Integration and QuickBooks Online Integration.