The Data Product Platform Navigator

Level 1: Provisional Platform – The “No Platform” Platform

A provisional platform for data products and data mesh is characterized
by an ad-hoc and chaotic approach to technology and provisioning. To
illustrate this, let’s use a bit of domain storytelling.

Consider a first data product developer who sets out to create a new
data product. Lacking a central self-service platform, they must navigate
the organization through personal networks. They start by talking to a
DevOps engineer [1] to request a source control repository, and that
engineer creates a Git repository [2] on the Git platform. Next, the
developer needs a build and deployment automation pipeline, so they talk
to a second DevOps engineer [3], who creates a CI pipeline [4] on the CI
system. For data storage, the developer talks to a cloud engineer [5], who
creates an S3 bucket [6] on AWS Cloud. With the basic infrastructure in
place, the developer sets up their own data processing tools: they create
an Iceberg database [7] on an Iceberg catalog and query engine, and create
a Spark job [8] running on a Spark cluster. By manually assembling these
disparate pieces, they produce something that resembles a data
product.

Sometime later, a second data product developer is tasked with building
another data product. Having no platform documentation or self-service
interface to guide them, they talk to the first developer [9] to ask how
to get started. The first developer refers them to the DevOps engineer who
helped them earlier. The second developer then talks to that same DevOps
engineer [10], who creates another Git repository [11] on the Git
platform. Next, the developer talks to the second DevOps engineer [12],
who creates another CI pipeline [13] on the CI system.

When choosing their analytical technology stack, however, the second
developer takes an entirely different direction. Rather than using Iceberg
and Spark, they create a dataset [14] on Google BigQuery, run a dbt model
[15] on dbt Cloud for transformations, and create a view [16] on a data
virtualization tool to serve the data.

This scenario highlights the core failure modes of a provisional
platform. Data technology is requested and provisioned entirely ad-hoc,
with human bottlenecks at every step where developers must “know someone
who knows someone that knows how to do something” to make progress. There
is no explicit concept of a data product or data contract. There are only
isolated, manually stitched pieces of technical infrastructure. The result
is too high a cognitive load, deep technological fragmentation and
complexity, and increasing shadow IT across the organization.

Level 2: Operational Platform – The “Fragmented” Platform Experience

At the operational level, there is an effort to standardize technology
and track requests centrally. However, the experience for data product
developers remains fragmented and complex.

To introduce order into the chaos, organizations typically establish
dedicated infrastructure teams (e.g. a DevOps Team, a Cloud Team, and a
Data Platform Team). Requests are then channeled through a centralized
ticketing system like Jira.

Continuing the story, the data product developer now has to request
every component through formal tickets:

The developer creates a Git Repository Ticket [1] in the ticketing
system. Sometime later, the DevOps Team reads the ticket [2] and
provisions a Git repository [3] on the Git platform. The developer then
creates a CI Pipeline Ticket [4], which the DevOps Team reads [5] to
provision a CI pipeline [6] on the CI System. For storage, the developer
creates an S3 Bucket Ticket [7], which the Cloud Team reads [8] to
provision an S3 bucket [9] on AWS Cloud. Turning to data infrastructure,
the developer creates an Iceberg Database Ticket [10], which the Data
Platform Team reads [11] to provision an Iceberg database [12] on the
Iceberg Catalog & Query Engine. Finally, the developer creates a Spark Job
Ticket [13], which the Data Platform Team reads [14] to provision a Spark
job [15] on the Spark Cluster.

While this introduces central visibility and standardizes the
technology catalog, it replaces personal tapping-on-shoulders with queues
and handoffs. Waiting for multiple teams to fulfill tickets takes
significant time. More critically, once all these individual components
are provisioned, the developer is still left entirely alone to glue them
together into a functioning data product. Frustrated by the delays and
integration friction, developers are often tempted to bypass the platform
altogether, managing [16] their own shadow IT solution (“My Data
Platform”) to deliver results faster.

At the operational level, automation can be introduced through typical
Infrastructure-as-Code (IaC) scripts (e.g. Terraform) to speed up the
provisioning of individual components. This will avoid ticketing systems
and human bottlenecks.

In this alternative setup, each
specialized team provides reusable scripts that developers can execute
independently. The data product developer executes the
createGitRepository script [1], which automatically
provisions a Git repository [2] on Git. The developer then executes the
createCIPipeline script [3], which provisions a CI pipeline
[4] on the CI System. For cloud storage, the developer executes the
createS3Bucket script [5], provisioning an S3 bucket [6] on
AWS Cloud. For data assets, the developer executes the
createIcebergDatabase script [7], provisioning an Iceberg
database [8] on the Iceberg Catalog & Query Engine, and executes the
createSparkJob script [9], provisioning a Spark job [10] on
the Spark Cluster.

While this script-based automation eliminates ticketing delays, the
developer is still left with the complex task of gluing everything
together to create a data product. The cognitive load remains high, and
there is a knowledge bottleneck, as developers need to know how to compose
data products from these different components (e.g. configure and
integrate networking, security roles, storage, pipelines, secrets, etc).
When errors occur, developers run into knowledge gaps and still need [11]
direct human help from the DevOps Team, Cloud Team, and Data Platform
Team.

Ultimately, this operational approach leads to a lack of uniformity:
organizations end up with “Frankenstein” data products that look similar
on the surface because they use the same underlying technologies, but are
architecturally inconsistent under the hood. Exhausted by the complexity
of stitching everything together, developers frequently retreat to shadow
IT, continuing to manage [12] their own local platforms (“My Data
Platform”).

Level 3: Scalable Platform – Platform as a Product

A scalable data mesh platform represents a major shift in the
developer experience. The interface changes completely, moving from
individual technologies to a higher-level, data product-centric
abstraction.

In this scenario, the organization establishes a
dedicated Data Product & Data Mesh Platform Team that treats the
platform as a product. Rather than handling tickets or providing
disconnected scripts, this team provides a self-service developer
experience consisting of a Data Product Developer Portal and an underlying
Data Mesh Platform Backend.

The domain
story is much simpler. In summary, a data product developer just clicks
‘Create Data Product’, and the platform wires the technology and
infrastructure together as a Data Product. In the domain storytelling
diagram above, the developer creates a Data Product [1] on the Data
Product Developer Portal by defining its architectural specification
such as input and output ports, data transformation code setup, and data
contracts for the structure and guarantees on the data. The portal
captures this intent and instructs the Data Mesh Platform Backend to do
the provisioning and integration of components.

From this single
declaration, the Data Mesh Platform Backend automatically orchestrates
the underlying technical resources across all infrastructure domains: it
provisions a Git repository [2] on Git, provisions a CI pipeline [3] on
the CI System, and provisions an S3 bucket [4] on AWS Cloud. In the data
infrastructure domain, the backend provisions an Iceberg database [5] on
the Iceberg Catalog & Query Engine, and provisions a Spark job [6] on
the Spark Cluster.

Instead of just spinning up isolated components, the backend
automatically wires and glues them together into a cohesive,
standardized data product.

This integrated self-service tooling transforms the developer
experience. The cognitive load on developers drops significantly because
they interact with the abstraction of a data product rather than
wrestling with underlying technologies. It removes the knowledge
bottleneck: developers no longer need to be infrastructure plumbing
experts who know how to hand-stitch security permissions, pipelines, and
storage configurations. Furthermore, because the platform backend
enforces architectural standards, every data product exhibits more
technical uniformity, which prevents “Frankenstein” data products. When
every data product adheres to the same standardized architectural
structure, the platform can enable automated governance by injecting
automated policy checks, audit logging, access controls, and data quality
verification across all data products. By making data product creation
easy, straightforward and frictionless, it encourages voluntary
adoption, naturally discouraging shadow IT.

By removing the knowledge bottleneck, the platform empowers data
product developers with more autonomy, which is crucial for scaling a
data mesh and data product management practice across multiple domain
teams.

Level 4: Optimizing Platform – Platform as an Ecosystem

At the optimizing level, the platform becomes deeply integrated
into the daily developer workflow. Rather than requiring developers to
switch contexts and go to a platform portal, the platform comes
directly to the developer’s working environment and context.

This shift is achieved in the developer’s local context by making
the tools they live in every day (e.g. IDE or CLI) “data product
aware.” The domain story illustrates how this streamlines the entire
data product lifecycle:

The data product developer creates [1] a local Data Product
directly from their day-to-day workspace. Through an IDE Plugin or a
dedicated CLI, the developer can create, develop, deploy, and monitor
the data products from within their familiar environment without
disruptive context switching.

The local tools communicate with the platform and create [2] the
Data Product on the centralized Data Mesh Platform Backend (which is
maintained by the Data Mesh Platform Team).

From there, the
Data Mesh Platform Backend automatically provisions and integrates the
necessary technologies needed for the data product. A Git Repository
is provisioned on Git [3] and a CI Pipeline on the CI System [4]. In
AWS Cloud, an S3 Bucket is created [5] to store data in the data
product. Structure is given to the data using an Iceberg Database [6]
on an Iceberg Catalog & Query Engine. And finally, the data product
comes alive when a Spark Job is running in the Spark Cluster [7].

Just as in Level 3, the platform backend automatically handles all
the wiring and integration so the provisioned resources are glued
together to function as a single, cohesive data product.

Another defining dimension of the optimizing level is the emergence
of a federated platform ecosystem. The central platform team no longer
acts as a bottleneck for new capabilities. Instead, other teams across
the organization can contribute directly to the platform backend:

• An engineer from a Domain Team can contribute a domain-specific extension,
  such as Platform Capability X (e.g. specialized data product templates), making
  it available to the entire organization [8].
• An engineer from a Specialized Team (such as security, compliance, or data
  governance) can contribute platform capabilities, such as Platform Capability Y
  (e.g. automated PII masking, fine-grained access policies, or audit logging)
  [9].

An optimizing platform for data products and data mesh minimizes
context switching for developers, manages the entire data product
lifecycle, and enables an ecosystem of contributors. Managing the
entire data product lifecycle empowers the organization to execute
migrations of data products when underlying technologies evolve (e.g.
upgrading an Iceberg or Spark version, breaking changes in data
contract formats, etc), without requiring every team to rewrite their
data products. Actively tracking which platform capabilities and which
data products are actually consumed allows teams to prune unused data
products and decommission obsolete platform features, continually
refining and evolving the platform to meet organizational needs.