https://discord.com/blog/how-discord-stores-billions-of-messages
<aside> 💡 From billions to trillions, Discord managed their database and structures to maintain their stable service state. But how? The answer is surely in their blog article, but what I tried was to follow their thoughts and methods in my own experiment table and take a deeper thought about unexplained theory for myself and junior engineers.
</aside>
My article will be written in simple way, and…
Discord team’s article in this way.
Let's get to the point.
The messages were stored in a MongoDB collection with a single compound index on channel_id and created_at. Around November 2015, we reached 100 million stored messages and at this time we started to see the expected issues appearing: the data and the index could no longer fit in RAM and latencies started to become unpredictable. It was time to migrate to a database more suited to the task.
I’ve done a simple simulation, to make sure index size exceeding RAM size makes problem.
simulated codes are in mongodb_ram_cache.py .
Left table used 0.4GB of cached RAM, right one used 4GB.
| 20000000, 1 index | 20000000, 5 indexes | |
|---|---|---|
| Total Index Size | ||
| 220729344 | ||
| 693288960 | ||
| Max Byte Configured | ||
| 428867584 | ||
| 428867584 | ||
| Max Elapsed Time (ms) | ||
| 17.32039 | ||
| 73.91405 | ||
| Min Elapsed Time (ms) | ||
| 6.11401 | ||
| 7.01022 | ||
| Avg Elapsed Time (ms) | ||
| 9.43092 | ||
| 12.27970 | ||
| 20000000, 1 index | 20000000, 5 indexes | |
|---|---|---|
| Total Index Size | ||
| 217350144 | ||
| 694005760 | ||
| Max Byte Configured | ||
| 4294967296 | ||
| 4294967296 | ||
| Max Elapsed Time (ms) | ||
| 8.38494 | ||
| 13.02290 | ||
| Min Elapsed Time (ms) | ||
| 2.57015 | ||
| 2.66004 | ||
| Avg Elapsed Time (ms) | ||
| 3.05736 | ||
| 3.31723 | ||
As you can see, when the total index size is over the maximum byte configured for MongoDB, read/write time occasionally goes too high, and it is higher than in other cases on average, too. But if there was much more RAM provided, the result would be different.
The experiment found that the average time elapsed was quite similar for this case, and the maximum elapsed time increased was lower than the 0.5GB cached RAM case. Considering this was done in a RAM + SSD environment, using HDD might result in even more difference.
So the larger the data size, the more RAM should be provided for stable DB service management. That would be expensive, as RAM is. To solve this problem, MongoDB suggests sharding, but the Discord team didn’t use it.
Everything on Discord was stored in a single MongoDB replica set and this was intentional, but we also planned everything for easy migration to a new database (we knew we were not going to use MongoDB sharding because it is complicated to use and not known for stability).