Create Data Products from Query Logs

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

You can create a data product from a source’s query history instead of from a blank specification. Alation analyzes how the source is actually queried, finds the tables that are frequently used together, and suggests a starting set of tables, classified by data architecture layer where it can recognize one. This analysis reads the query logs directly and does not require lineage to be enabled or any additional BI tooling. You review the suggestion, choose how to package it, and continue through the rest of the Data Products wizard.

This flow is one way to start a data product. For the full creation workflow and the other ways to create a data product, see Get Started with Creating Data Products.

Note

This feature is in Beta. The Recommend (from Query History) option appears in the create flow when the feature is enabled for your environment.

In this topic:

Prerequisites

  • You have permission to create data products in the Data Products App.

  • The source has query log ingestion (QLI) configured, so that query history is available to analyze.

Note

You do not need published or sample queries to generate recommendations. Published and sample queries are used later to enrich the data product with metrics, but clustering and table recommendations do not depend on them.

Understand Recommendation from Query History

After you select Recommend (from Query History) in the Data Products wizard and choose a data source, Alation builds the suggestion automatically. You do not configure these stages.

  1. Gather usage evidence: Alation reads the source’s query history over the scan window, which is the last 30 days by default. It parses and anonymizes the queries — literal values are replaced with variables, so only table and column names are used, never actual data values — and converts them into query templates.

  2. Cluster co-queried tables: From the templates, Alation identifies the tables that are most frequently queried together and proposes that subset. It does not pick a fixed number of tables. Tables that are rarely queried are not suggested, but you can add them yourself later.

  3. Classify by architecture layer: Alation sends the suggested tables and their titles to an AI model. The model infers each table’s typical usage and assigns an architecture label based on the table’s name. Classification uses only the table name and title and not the table’s data and not query statistics.

Because the suggestion is built from real usage, the tables and relationships reflect how analysts queried the source during the scan window.

Interpret the Architecture Labels

Alation recognizes several common data architectures from table naming conventions and labels the suggested tables accordingly. These are standard industry conventions, not classifications that Alation defines:

  • Medallion: bronze, silver, and gold layers. Bronze is raw, as-ingested data that mirrors the source, with little or no transformation. Silver is cleansed and conformed data: deduplicated, type-cast, joined across sources, and quality-checked. Gold is curated, business-ready data, aggregated and denormalized for analytics, reporting, and consumption, and is what end users typically query. Alation infers the layer from naming patterns, for example a GLD_ prefix for gold, SLV_ or SLVR_ for silver, and BRZ_ or BRNZ_ for bronze.

  • Star schema: fact and dimension tables.

  • Kimball: master, transactional, and aggregate tables.

In addition to the architecture, each table can show the following labels:

  • Table type: A role in the cluster, such as snapshot (point-in-time captures such as inventory levels), reference (lookup tables), history (tables that track changes over time), or staging (raw staging tables).

  • Confidence: A level, such as strong or moderate, that indicates how certain Alation is about the table’s type and its fit in the cluster.

  • Not recommended: A marker Alation applies when a table’s type is typically not useful in a data product, such as a staging, vault, bridge, or intermediate table. You can still include a not-recommended table if you need it.

Keep the following in mind when you read the labels:

  • The labels are derived only from table names and titles, so they are best-effort suggestions rather than a verified classification of the data.

  • If the tables do not follow a recognizable naming convention, Alation shows them without an architecture layer, and you select tables freely.

  • Alation analyzes one schema at a time. It cannot reconstruct an architecture that is split across multiple schemas. For example, if gold, silver, and bronze tables live in separate schemas, Alation does not detect the layers together.

Create a Data Product from Query Logs

  1. Access the Data Products App.

  2. Open the Data Products wizard to begin creating a new data product. For details, see Get Started with Creating Data Products.

  3. On the asset type step, under What do you want to build from?, click Recommend (from Query History).

  4. Select the data source with query history that you want to use. Alation lists only data sources that have query history available.

  5. Confirm or change the Scan window. The default is the last 30 days.

  6. If you change the scan window, click Rescan to regenerate the suggestion.

Alation analyzes the query history and presents the suggested data products.

Note

The description you enter on the initial creation step is not used by the query-history recommendation, so you can leave it blank when you use this flow.

Review the Suggested Tables

Alation presents one or more suggested data products, each a cluster of tables that are frequently queried together. The screen shows a count, such as Found 15 suggested data products, and lists each cluster with the tables it contains.

  1. Review the suggested data products. Each entry lists the tables in the cluster, labeled where Alation detected an architecture or a table type.

  2. Select the cluster you want to use, then click Review relationships.

  3. Review the tables and the relationships Alation inferred between them. The relationships are AI-inferred from query history, not verified data lineage, so confirm them before you rely on them.

  4. Adjust the selection: clear any suggested table you do not want, and add tables that were not suggested.

Note

The suggestions and relationships are AI-generated. Review the tables, types, and relationships before you continue.

Package the Suggested Tables

Before you continue, choose how the tables are packaged.

  1. Choose one of the following:

    • One per layer — Create a separate data product for each detected layer. This option is available only when Alation detected architecture layers.

    • Single data product — Combine all of the suggested tables into one data product.

  2. Click Continue with these tables.

Alation carries the selected tables and relationships into the Data Products wizard, where you complete the remaining steps, such as documentation and visibility, as you would for any other data product. For details, see Get Started with Creating Data Products. After you create the data product, its lifecycle (versioning, listing, and editing) is the same as that of any other data product.