System unavailability
Incident Report for RATELIMITED
Resolved
Our Newsroom is currently in the process of being restored. Otherwise, service fully operational.
Posted Dec 26, 2019 - 14:07 UTC
Update
We are continuing to monitor for any further issues.
Posted Dec 23, 2019 - 08:19 UTC
Monitoring
After an investigation, it was found that our backup was taken approximately 18 days ago.

We would like to apologize to our users about this data loss, and please expect an e-mail regarding this soon.

We are currently investigating to see as to what was the exact root cause of the issue.
Posted Dec 23, 2019 - 08:17 UTC
Identified
It appears that the LVM pool managed to get corrupted once again.

We are assessing the situation, and have temporarily restored service from a backup.

Some data might be missing, we are currently trying to identify how much data was lost during the backup restore.
Posted Dec 22, 2019 - 05:17 UTC
Investigating
We are currently investigating an incident where the server is inaccessible.

What we know so far:
NGINX is up, but extremely slow. (Resulting in gateway timeouts)
PVE through the VM's NGINX 502s.
SSH is inaccessible.
Posted Dec 21, 2019 - 11:58 UTC
This incident affected: APIs (File Uploads, File Storage (Minio), Users - Deletion, Users - Password Updating, Users - E-Mail Updating, API Keys - Creation, API Keys - Deletion, API Keys - Renaming, API Keys - Regeneration), Frontend (Frontend (Origin)), Services (Newsroom), Service Functions (File Delivery), and Internal Tools (Frontend (Staging)).