Building A Distributed Database Step by Step…(Part 1)
Have you ever wanted to make a custom database? In this talk we will deep dive into how distributed databases and Blockchains make consensus. We will review how Raft and Paxos compare to Byzantine fault-tolerant systems in Blockchains such as Ethereum. You will learn how to build your own Database
A distributed database is a collection of multiple interconnected databases, which are spread physically across various locations that communicate via a computer network.
The Project is to build a distributed database, it's my hobby project. After completing this project we will learn the following and many more...
- What is a distributed database?
- The practical working of Distributed Database.
- Algorithms for Sync all Database (Consensus Algorithm)
- Building our Query Language for Retrieval of Data.
- Create a log and Try to add ACID ( Atomicity, Consistency, Isolation, and Durability) Property and Transaction Management
- Building a highly concurrent system and learn how to achieve Concurrency.
- Learn about GO Programming Language (GO make concurrency easy)
- We may use any other programming language as required ….
- Pick up one of the blockchain Consensus Algorithm like Raft etc.
- And many more we will discover later and it’s a long journey...
Abstract Layout of Distributed Database what we try to build
Let’s Discuss the very basic idea of Distributed Database and Design Technique…
Distributed Database Management System
A distributed database management system (DDBMS) is a centralized software system that manages a distributed database in a manner as if it were all stored in a single location.
Features
- It is used to create, retrieve, update and delete distributed databases.
- It synchronizes the database periodically and provides access mechanisms by the virtue of which the distribution becomes transparent to the users.
- It ensures that the data modified at any site is universally updated.
- It is used in application areas where large volumes of data are processed and accessed by numerous users simultaneously.
- It is designed for heterogeneous database platforms.
- It maintains the confidentiality and data integrity of the databases.
Advantages of Distributed Databases
Following are the advantages of distributed databases over centralized databases.
Modular Development − If the system needs to be expanded to new locations or new units, in centralized database systems, the action requires substantial efforts and disruption in the existing functioning. However, in distributed databases, the work simply requires adding new computers and local data to the new site and finally connecting them to the distributed system, with no interruption in current functions.
More Reliable − In case of database failures, the total system of centralized databases comes to a halt. However, in distributed systems, when a component fails, the functioning of the system continues may be at a reduced performance. Hence DDBMS is more reliable.
Better Response − If data is distributed in an efficient manner, then the user requests can be met from local data itself, thus providing a faster response. On the other hand, in centralized systems, all queries have to pass through the central computer for processing, which increases the response time.
Lower Communication Cost − In distributed database systems, if data is located locally where it is mostly used, then the communication costs for data manipulation can be minimized. This is not feasible in centralized systems.
Letter We will talk more about the distributed database and design techniques
Our main vision to build a distributed database top of famous SQLite DB
Actually main idea is to build Distributed Database for IoT and Wireless Sensors Networks Which will Support RDBMS(Relational Database) not just key value pairs NoSQL Data.
We may able to use this Lightweight Database for microservices which support high availability and scaling.
Use case of our Custom Made Distributed Database(We will call it RLDB)
Rapid Light Weight Distributed Database (RLDB)
Because traditional database required very high computational power and primary storage compare to SQLite DB. We will try to solve this problem by our lightweight distributed SQLite DB.
IoT and Wireless Sensors networks are made of low powered and low processing systems and installing a traditional database is a very bad choice for that. So a traditional solution for that is to build a centralized database.
But our solution is to build a low power and low computational enable distributed database for IoT and Wireless Sensors Networks.
Later we discuss more use case of our Custom Made Distributed Database and how this database solves some existing problem.
Stay tuned and I hope it will be fun and challenging … 😎
Next Part 1.1.1
