Today in Edworking News we want to talk about In 2017, they wrote a blog post on how they store billions of messages. they shared the journey of how we started out using MongoDB but migrated our data to Cassandra because we were looking for a database that was scalable, fault-tolerant, and relatively low maintenance. They knew we’d be growing, and they did! They wanted a database that grew alongside us, but hopefully, its maintenance needs wouldn’t grow alongside our storage needs. Unfortunately, they found that to not be the case — our Cassandra cluster exhibited serious performance issues that required increasing amounts of effort to just maintain, not improve. Almost six years later, they're changed a lot, and how they store messages has changed as well.
Our Cassandra Troubles
They stored our messages in a database called cassandra-messages. As its name suggests, it ran Cassandra, and it stored messages. In 2017, they ran 12 Cassandra nodes, storing billions of messages. By the beginning of 2022, it had 177 nodes with trillions of messages. Despite the growth, the system was high-toil — our on-call team was frequently paged for issues with the database, latency was unpredictable, and maintenance operations were becoming too expensive to run.
Image Description: A visual representation of a Cassandra cluster with nodes and data flow.
### Changing Our Architecture
Our messages cluster wasn’t our only Cassandra database. They had several other clusters, each exhibiting similar faults. We were intrigued by ScyllaDB, a Cassandra-compatible database written in C++. Its promise of better performance, faster repairs, and stronger workload isolation via its shard-per-core architecture was appealing. By 2020, they had migrated every database but one to ScyllaDB. The last one? Our friend, cassandra-messages.
Data Services
With Cassandra, they struggled with hot partitions. High traffic to a given partition resulted in unbounded concurrency, leading to cascading latency. To control the amount of concurrent traffic to hot partitions, they wrote intermediary services called data services that sit between our API monolith and our database clusters. They chose Rust for writing our data services due to its fast speeds and safety features. Rust’s libraries were a great match for our needs, and its emphasis on safety made it easy to write safe, concurrent code.
Image Description: A diagram showing the flow of data services between API and database clusters.
### A Very Big Migration
Our migration requirements were straightforward: migrate trillions of messages with no downtime, and do it quickly. They provisioned a new ScyllaDB cluster using our super-disk storage topology. Initially, our migration plan was to dual-write new data to Cassandra and ScyllaDB and migrate historical data behind it. However, they decided to rewrite the data migrator in Rust, which significantly sped up the migration process. We managed to migrate data at speeds of up to 3.2 million messages per second.
Image Description: A graph showing the migration speed and completion status.
### Several Months Later…
They switched our messages database over in May 2022, and it’s been a quiet, well-behaved database since then. We reduced our nodes from 177 Cassandra nodes to just 72 ScyllaDB nodes. Our tail latencies have also improved drastically. For example, fetching historical messages had a p99 of between 40-125ms on Cassandra, with ScyllaDB having a steady 15ms p99 latency. This performance improvement has unlocked new product use cases for us.
At the end of 2022, during the World Cup, they observed that goals scored showed up in our monitoring graphs. This was a testament to our system's robustness, as it handled the increased traffic without breaking a sweat.
Remember these 3 key ideas for your startup:
Scalability and Maintenance: When choosing a database, ensure it can scale with your growth while keeping maintenance needs manageable. Migrating to a more efficient database like ScyllaDB can significantly reduce operational toil and improve performance. For more insights, check out how to determine realistic goals for a project.
Data Services and Concurrency: Implementing intermediary data services can help manage high traffic and reduce database load. Using a language like Rust can provide the speed and safety needed for such tasks. Learn more about task automation and why you should use it.
Efficient Migration: A well-planned migration strategy can save time and resources. Rewriting critical components in a performant language can drastically speed up the process, as seen with our Rust-based data migrator. Discover the best productivity hacks to get your work done.
Edworking is the best and smartest decision for SMEs and startups to be more productive. Edworking is a FREE superapp of productivity that includes all you need for work powered by AI in the same superapp, connecting Task Management, Docs, Chat, Videocall, and File Management. Save money today by not paying for Slack, Trello, Dropbox, Zoom, and Notion.
I
For more details, see the original source.
