introducing vapusdata

Introducing VapusData

Author: Vikrant Singh
Published On : 19th June, 2024

" Everything is Data & Data is Everything "

This is VapusData's first blog article. In it, we will introduce the concept, our goal, important ideas, and other features. Let us first discuss our vision's motto, which is mentioned in the phrases above.

What do we mean by "Everything is Data"?

In today's digital world, data takes myriad forms and exists in countless places. It can be:
Let's start the journey of VapusData.

Introduction to VapusData

When it comes to decentralizing and managing data throughout a company, VapusData is a platform that offers strong and customizable Data Mesh frameworks. With VapusData Platform, data owners are brought closer to ETL (extraction, transformation, loading). In a similar vein, data product marketplaces (based on data catalog concept) make it simple to browse, find, and use published data products.

Data consumers can request data products by searching for data-store metadata across domain nodes. Relevant domain owners can then accept, configure, and publish the requested data products.

VapusData's unified query interface makes it easy for developers to code and create apps based on any database they want, without requiring them to deal with data drivers or intricate conditional statements.

Our Vision

overview of vapusdata
VapusData is an all encompassing platform that was developed with the purpose of safely decentralizing and managing data for organizations. The following is an exhaustive rundown of our vision:
  • Create a Unified query interface that supports all major datastores, including SQL and NoSql based data stores, data streams, and caching, and that functions based on data intake routes. A single interface that will let you build an application, remove the maze of conditional statements from your CRUD codebase, and take it to new heights.

    To put it simply, Unified Query Interface will enable you to store and control the CRUD activities of the datastores utilizing data routes and a basic SDK in a language of your choice. In addition to many other things, it will automatically modify and index the data based on usage, cache the fields based on usage, and much more. We will talk more about this fascinating idea in other papers and articles.
  • Give domain owners a strong foundation on which to construct, maintain, and validate their data containers. and make it possible for such data containers' attestation to serve as the foundation for reliable data products.
  • Create a data catalog that enables data consumption, product requests, and discovery across several data stores housed on different cloud providers' or self-hosted infrastructure. Customers of data do not have to seek for their items in separate data catalogs.
  • Provide an adaptable, safe, and reliable attribute-based control to oversee the development, management, shipping, and consumption phases of the data product life cycle.

What problems we are solving?

Data Management has always been an issue since the data has existed, specially when it comes to data sharing, discovery and governance, in further posther posts we will deep dive into each problem statements and how VapusData aims on solving that. But before that lets create a P2P summary of those problems in our mind map.

Bad Data Governance

In traditional designs, it can be difficult to guarantee consistent data governance and compliance across a wide variety of data sources and teams.

Data Discovery

It is common for traditional data architectures to result in data silos, which are situations in which data is kept separate inside particular teams or departments, thereby restricting both accessibility and cooperation.

Data Silos

Data Management has always been an issue since the data has existed, specially when it comes to data sharing, discovery and governance, in further posther posts we will deep dive into each problem statements and how VapusData aims on solving that. But before that lets create a P2P summary of those problems in our mind map.

Scalability Problems

Data teams that are centralized have the potential to become bottlenecks, which can slow down data access, processing, and creativity throughout the firm.Centralized data systems have trouble keeping up with the growing amount, variety, and speed of data.

Domain Ownership

Without clear ownership, no specific team is accountable for data quality, leading to inconsistencies and errors.Centralized teams struggle to keep up with rapidly changing business requirements, delaying insights and decision-making.

Data Audit & Security Scanning Cost

There is no single interface that exists where all of the databases are collectively connected or configured with their metadata, which makes database and data auditing for security and audit teams a highly lengthy and expensive operation. This is because there is no one interface that, exists.

Database Driver Management

The database drivers that the data and domain team uses are a constant source of concern for them, regardless of whether or not they are official and up to date.
In addition to this, when an application makes use of numerous databases, the coding layer of the databases begins to get increasingly cluttered with a large number of conditional statements.

Inconsistent Data Integration

The integration of data from a variety of sources necessitates processes that are both complicated and time-consuming, which ultimately results in inefficiency.

Key Components

First, let us go over the fundamental ideas behind the VapusData platform so that we may get a better understanding of it.
KEY COMPONENTS OF VAPUS DATA

Data Containers

Data containers The platform brings ETL (Extraction, Transformation, Loading) processes close to data owners, ensuring that data is processed and managed by those who understand it best.This proximity enhances data accuracy and relevancy, as data owners can directly oversee the transformation and loading processes.

Data Governance and Access management

Data governance policies are built into the data products themselves. This makes sure that the organization's rules are followed when accessing and using data.Policies cover things like data protection, security, keeping data, and following compliance like GDPR.
Fine-grained access limits make sure that only people who are allowed to can get to certain data products.To limit who can do what, VapusData uses attribute-based access control (ABAC) in data product access interfaces.

Data Product Marketplace

VapusData features an intuitive data product marketplace based on the data catalog principal, providing an easy-to-use interface for searching, discovering, and consuming published data products.
This marketplace allows data consumers to efficiently find the data they need and integrate it into their workflows.Metadata management is crucial for discovering and understanding data products. Each data product includes comprehensive metadata detailing its schema, quality metrics, lineage, and usage policies.VapusData helps domain owners in this field by automating the management of metadata across the platform.

Unified Query Interface

VapusData offers a unified query interface that simplifies the development of applications.Developers can easily code and develop applications based on their preferred database, without the need to manage data drivers or complex conditional statements.This interface abstracts the complexities of data management, enabling faster and more efficient application development.

Vapus Observability

Consumers can request specific data products by searching through data-store metadata across various domain nodes.Relevant domain owners can then review these requests, configure the necessary data products, and publish them for consumer access.

Data Product Request

It is important to keep watching and being able to see data all the time in order to maintain its quality and integrity. Anomalies can be found, data usage can be tracked, and SLAs can be made sure of with metrics and logs.Lineage tracking and automated checks for data quality make the data products clear and trustworthy.

In addition, VapusData also provides a single plane of glass for all the connected data store auditing, compliance and Secops review.

Robust and Configurable Frameworks

VapusData offers adaptable frameworks that make it possible to decentralize and manage data in a secure manner across a wide range of business areas.
These frameworks are designed to be extremely flexible, which enables domain owners to adjust data management processes to their particular requirements.

Whats Next?

In the next blog, we will discuss more about the key concepts of vapusData and dive deeper into the key concepts of VapusData and what they do.
And we will go through the architecture of the VapusData platform as well. So stay tuned for an interesting discussion over the architecture.

References