Maximizing Efficiency in Scalable Systems: Part I — Reducing costs of consumption-based state stores by up to 97% when using Dapr actors at scale
Scalable systems are really good when dealing with high volumes of data. They ensure smooth operations without failovers or the need for human intervention, such as manually increasing RAM or adding more servers — a practice that harks back to the 2008 software development era.
However, the convenience of scaling often comes with a hefty price tag, potentially impacting your return on investment. This article isn’t about building systems at scale in Dapr, but rather, it’s about making them more cost-effective. If you want to read about building systems at scale with Dapr, you can find my colleague’s blog here: Building event-driven systems at scale in Kubernetes with Dapr.
Today, we’ll explore how to reduce your bill when using a consumption-based state store with Dapr. In my case, we’re using CosmosDB and this will be my basis.
The Power of Dapr and CosmosDB in Handling High Volumes
Dapr was instrumental in helping my team to build a scalable, event-driven system capable of handling approximately 320 million events per day. This included the usage of a scalable state store database, for which we used Azure CosmosDB, provisioned with 600K RU/s. While the system operates 24/7, handling all events without fail, it’s important to remember that the benefits of a scalable system, such as zero downtime, come at a cost.
The Cost of Scalability: A Closer Look
Focusing on CosmosDB, provisioned with 600K RU/s, we found that the costs were substantial. With CosmosDB, billing is based on RU usage, which is determined by volumes and the size of the documents you interact with. For instance, reading a 75KB document from CosmosDB uses 75 RU/s, and if that document grows to 100KB and is written back to CosmosDB, it uses 100 RU/s. This totals 175 RU/s per I/O operation. Multiply this by thousands of requests per second, and you end up with a hefty cloud bill.
A Solution: Compression — Cutting Costs by 97%
After thorough analysis and exploration of various cost-reduction strategies, we decided to compress the actor state before saving it to CosmosDB and decompress it after retrieval. This simple act of compression reduced the document size from 300KB to 9KB, resulting in a roughly 97% reduction. Consequently, the RU consumption plummeted from 600K RU/s to an average of 20K RU/s in a 24 hour period. With roughly 580K RU/s consumption saved, you can now provision your CosmosDB to auto-scale between 10K — 100K RU/s and still process the same volume of data, and will likely sit towards the bottom end of that scale most of the time.
Despite the additional processing time required for compression and decompression, we found that the overall response time for each request remained relatively the same or even less. This is because the compressed state is significantly smaller, making the I/O operation quicker compared to handling large uncompressed actor state. This effectively compensates for the time loss during compression and decompression.
The Pros and Cons
Pros
- Compression saves a substantial amount of money per month by reducing RU consumption
- Reduced costs enhance product viability and ROI
- Compression can be applied to any document storage database (e.g. CosmosDB, MongoDB, etc.)
- CosmosDB is unlikely to rate limit Dapr state management calls
Cons
- Compressed state is not human-readable, necessitating a custom tool or endpoint for viewing uncompressed state
- Querying compressed state with state state properties is not possible; queries can only be made by actor id (This should not be a significant issue as Dapr interacts with state stores in a key-value fashion)
Conclusion
While this compression-decompression process can be implemented outside the Dapr SDK, it would be beneficial to see this functionality incorporated into the Dapr itself. This could be achieved by extending the SetStateAsync method to include a new parameter (e.g. compressState) that can be set to true or false.
Keep an eye on part 2, where I will be taking you through the actual code implementation of compression.
