Performance and scalability of in-memory databases | Big Data
In the times when a growing number of applications and services demand more and more resources for high availability and performance purposes, scalability is one of the biggest challenges for database designers. Demand for traditional databases that keep data residing on disk is going down as performance and time are the key values in every business that relies on database solutions. One of the paths to follow in order to achieve database access speed within the range of microseconds instead of merely milliseconds is to switch from disk-based solutions to much faster in-memory ones.
There are a number of such databases already available on the market that address the need for performance gain and scaling by keeping data completely in memory and providing means to scale horizontally (out) and vertically (up). In-memory data grids such as GigaSpaces and Oracle Coherence as well as in-memory databases including MySQL Cluster, IBM solidDB or Oracle TimesTen seem to meet business needs.
Quoting benchmarks
Multiple performance benchmarks show that in-memory databases are fully scalable solutions with near-linear scalability. ExtremeDB in-memory database (flagship of McObject company) claims linear scalability in the benchmark that was run on the installation consisting of 160 64-bit processor cores while using over 1 terabyte (requires registration) of data stored completely in memory. On the other hand, the open-source MySQL Cluster tested on a 16-node cluster achieved 500,000 reads per second whereas if the number of nodes had increased to 32 the performance gain was 3 times that amount. But scaling up is not the only possibility here.
In-memory databases give you more than that. They give you scaling out which basically means adding more nodes to the system. Operations of replication and partitioning may be added to the picture if more hardware is available to the system. Almost all in-memory solutions offer some kind of replication to ensure the availability of the data as well as load balancing.
As a proof of concept, this benchmark on MySQL Cluster shows scaling from two to four data nodes configuration with various numbers of servers in the system and on average 7x performance speedup.
Final thoughts
There is a host of competitive technologies and solutions that meet the performance and scalability needs of modern mission-critical applications. Choosing the right one depends on the particular problem and the purpose that it is going to serve but if you want a real-time, extremely fast, and reliable database management system, an in-memory database is a solution worth considering.
