We are approaching 7 years since the term “NoSQL” re-entered the popular tech vernacular, and 7 years since I wrote the post “Is the Relational Database Doomed?”. During this time, we have experienced a tidal wave of non-relational data management technologies. So, time for an update to my prior article.
At the start of the decade, when the NoSQL buzz was in its heyday, some were predicting the end of the dominance of the relational database platform (RDBMS) within a decade. The reason for this seemed somewhat sound. That being, the relational database is based on what is now 40+ year technology and things are so much more advanced now than back then, so clearly this was a technology ripe for disruption.
So how has this disruption gone? Well, all my metrics show there are more relational databases in existence today than at any point in history. It may be hard for many people to comprehend the volume. Often, mid-size enterprises operate hundreds of relational databases. Many large enterprises have thousands to tens of thousands. These represent the data stores of everything from ERP's, financial systems, content sharing apps, IT tools and so on.
So despite the noise surrounding NoSQL, in a head to head comparison of volume of use, NoSQL use seems so very small. At a guess, I would predict that for every NoSQL database in existence there would be at least 1000 relational databases. Probably more. You would be forgiven for thinking NoSQL use was almost insignificant.
So why has there been so little disruption?
- The relational database has such a massive legacy. The IT world is full of people whose front of mind solution for a new data management requirement is a relational database. To demonstrate this I looked on LinkedIn. My search showed that over 1,000,000 people list “SQL” as a skill in their profile. In comparison only 16,000 listed NoSQL and 30,000 listed MongoDB. That’s a massive skills gap.
- The RDBMS is very general purpose. There are very few day-to-day data management requirements that cannot be met without a run of the mill RDBMS. So why would you not go with what you know, if what you know is suitable?
- Relational databases are solving very complex problems in a balanced approach. There is 40+ years of learnings on how to balance consistency, concurrency, scale and performance. Many NoSQL initiatives focus on improving some objectives (such as scale or performance) at the expense of others (such as consistency or redundancy) solving their own problems but also lacking a general purpose appeal.
- Relational database vendors have also kept innovating. With some RDBMS vendors you can now combine SQL and XPATH, support JSON natively and support other non-structured data types. Also, many RDBMS platforms now support in-memory databases and others are quickly adding this support.
So with the continued dominance of the relational database, what future is there for the NoSQL alternatives? Well that is clear, the same opportunity as they have been filling over the last 7 years. Edge cases. Sure, enterprises have many routine data in/data out applications and these belong on the RDBMS but modern enterprises are trying to do more with data than ever before and leverage data in new ways for a competitive advantage. Maybe they need data to be captured at a massive scale, much greater than what is possible with a traditional RDBMS. Maybe they are looking to deep mine data to identify complex relationships between entities, or make predictions about how scenarios will transpire. Maybe they are trying to learn from large and diverse data sets and discover key new ways to improve productivity. This new world of requirements is where new world platforms have the opportunity to shine, focus on improving a specific set of key objectives, potentially at the expense of others, and then find the market that needs those objectives.
To summarise, I cannot see a world in the near future where any non-RDBMS gains any dominance in supporting the data management needs of most applications. The vendors people use may change, and the location of those databases may change (on premise to cloud) but they will be relational. However, what are the edge cases of today will of course become more mainstream. When this happens it will not be at the expense of the RDBMS, but instead they will be in addition to it. The playing field is getting bigger. Organisations desire to do more with data, via big data or data science initiatives that is fuelling a market ripe for vendors with clever, yet tightly focused, data management solutions.
So as it turns out, relational (SQL) and non-relational (NoSQL) technologies were not at war at all. They are in fact allies, working together to deliver organisations both general purpose and special purpose data management solutions.