We experienced intermittent FloatHub system outages between the hours of 3 am and 11 am (UTC) today (Dec. 8, 2021). This was related to a major system failure at Amazon Web Services in their US-east zone where much of our infrastructure is located (see Bloomberg News more details). No data should have been lost, but systems were unavailable during most of the outage and it will take a little while for devices to catch up to current with our backend systems.
This is the second major Amazon outage in the last 18 months. After the first, we deployed redundant systems at other infrastructure centers in an attempt to ensure no data would be lost. We are currently looking at both better status systems (so end users can find out faster when there is an issue) and logic for improved automatic failover to other infrastructure options.
