Skip to main content

DataSQRL Compiles Data Pipelines

Implement your data processing in SQL and define your data API in GraphQL. DataSQRL compiles an optimized data pipeline powered by Kafka, Flink, and Postgres.

Pick an Example: 
IMPORT datasqrl.tutorials.clickstream.Click;  -- Import data
/* Find next page visits within 10 minutes */
VisitAfter := SELECT b.url AS beforeURL, a.url AS afterURL,
a.timestamp AS timestamp
FROM Click b JOIN Click a ON b.userid=a.userid AND
b.timestamp < a.timestamp AND
b.timestamp >= a.timestamp - INTERVAL 10 MINUTE;
/* Recommend pages that are frequently co-visited */
Recommendation := SELECT beforeURL AS url, afterURL AS rec,
count(1) AS frequency FROM VisitAfter
GROUP BY url, rec ORDER BY url ASC, frequency DESC;

Step 1: Implement in SQL

Develop a recommendation engine for your customers based on visits to your website.

DataSQRL builds on the SQL you already know for a low learning curve.

type Query {
Recommendation(url: String!): [Recommendation!]
}

type Recommendation {
url: String!
rec: String!
frequency: Int!
}

Step 2: Define API

Specify the GraphQL schema for the API. That's how external or internal customers access the processed data.

DataSQRL compiled data pipeline

Step 3: Compile to Pipeline

DataSQRL compiles SQL + GraphQL schema into an optimized data pipeline integrating Apache Flink, Kafka, Postgres, and API layer into a robust, scalable, and easy to maintain data product.

AWS
AWS
GCP
GCP
Azure
Azure
Docker
Docker
Kubernetes
Kubernetes
Confluent Cloud
Confluent Cloud

Step 4: Deploy Anywhere

DataSQRL builds optimized executables for each component that run efficiently on your preferred cloud, managed service, or self-hosted.

Use the services and technologies you already trust to run your data pipeline.

Saves You Time

Saves You Time

DataSQRL allows you to focus on your data processing by eliminating the data plumbing that strangles your data pipeline implementation with busywork: data mapping, schema management, data modeling, error handling, data serving, API generation, and so on.

Easy to Use

Easy to Use

Implement your data processing with the SQL you already know. DataSQRL allows you to focus on the "what" and worry less about the "how". Import your functions when SQL is not enough - DataSQRL makes custom code integration easy.

Fast & Efficient

Fast & Efficient

DataSQRL builds efficient data pipelines that optimize data processing, partitioning, index selection, view materialization, denormalization, and scalability. There actually is some neat technology behind this buzzword bingo.

Fully Customizable

Fully Customizable

Open Source

Open Source

Robust & Scalable

Robust & Scalable

Use Cases

Data Mesh

Data Mesh

DataSQRL empowers domain teams to develop streaming data products autonomously. Build a self-service data platform with existing skills.

Event-Driven Microservices

Event-Driven Microservices

Efficiently process events in realtime and expose the results through consumable APIs in an event-driven architecture.

Observability & Automation

Observability & Automation

Build tailored observability platforms that turn your metrics into insights. Automate your processes with custom rules and AIOps.