The DataSQRL Process
Across many organizations, we observed that data products commonly struggle or fail because of:
- Complex Technology: The technologies and data architectures used are very complex and require specialists which causes delays, budget overruns, and missed requirements.
- Misaligned Process: The process used to implement the data product is not focused on the needs of the customer.
- Political Interference: Internal and external politics and governance requirements add significant friction or outright derail the project.
DataSQRL simplifies the technology for building data products. To address the other two problem areas that frequently strike data product implementations, we developed the DataSQRL Process.
The basic idea behind the DataSQRL process is to focus on value delivery and limit the impact of external factors. In the context of building data products, these goals can be surprisingly difficult to attain because a team's energy and attention is frequently drawn to implementation, orchestration, and planning issues. And managing the political ramifications of data projects can be an outright Kafkaesque experience.
Key Principles
The DataSQRL process is based on three key principles:
- Customer-focused: Focus on customer satisfaction through early and continuous delivery of valuable data products.
- Responsive: Harness changing requirements and creative input from all stakeholders for competitive advantage.
- Integrated: Integrate with existing software development processes, tools, and frameworks.
We distilled these principles from our work on a broad range of data product implementations. We think of these as "light houses" that we use regularly to make sure a data product implementation is oriented correctly and reposition if necessary.
Maybe it's just us, but we find having a bit of light in the fog of a data product implementation to be very reassuring.
Adopt the DataSQRL Process to Your Organization
The DataSQRL process is a framework and not a prescriptive process implementation. In line with key principle #3, we found that processes that are closely aligned with an organization's existing software development processes are the most likely to be successful.
We recommend developing a data product implementation process by applying the key principles to your existing software development process. The goal is to strike a balance between accommodating the unique characteristics of data product implementations and aligning with your existing development workflows.
That does sound a bit wishy-washy. Unfortunately, there is some art in it. We will try to nail it down more as it matures. Until then, we can help you with that.
Why Do We Need a New Process?
The unique requirements of data products - like data acquisition, model building, or data architecture design - often do not fit into an organization's existing software development process, which leads to the adoption of specific data science or data engineering processes.
While those processes are well-suited to address these unique requirements, they often lose focus of customer needs and value delivery because of their complexity, lengthy planning cycles, and high implementation cost. In addition, the misalignment with an organization's development process results in high friction and operational overhead which can further delay value delivery. You win the battle but loose the war - and usually a lot of money.
The goal of the DataSQRL process is to shift the focus back to the needs of customers and value generation, while accommodating the unique requirements of data products and staying aligned with the existing development process. In other words, removing the distractions from building with data, so we can all have some fun again.
The DataSQRL process is a work in progress, and we continuously refine it as we learn from our community and customers. Please share your thoughts, opinions, and ideas by joining the DataSQRL community or working with us.