The medal matters to me. The more useful story for an engineer, though, is what sat behind the demo: governed metrics, a real data pipeline, a 30-question evaluation suite, constrained server APIs, and product decisions that let a user inspect why an AI-generated answer should be trusted.

01What ChicagoPulse does with Databricks Genie

ChicagoPulse is a Databricks App that turns governed City of Chicago open data into plain-language answers, neighborhood comparisons, maps, and transparent data-health signals. It is built for residents, community organizations, and civic-data practitioners who want local context without first learning Socrata APIs, SQL, or the structure of several municipal datasets.

The app has three connected surfaces:

1. Ask ChicagoPulse answers natural-language questions through Databricks Genie. A user can inspect the generated SQL, chart or table, source identifiers, request provenance, and suggested follow-up questions.
2. Neighborhood Pulse covers all 77 official Chicago Community Areas, with a choropleth, completed-month metrics, 12-month trends, and comparisons for up to four areas.
3. Data Health shows whether the pipeline actually completed, when each source was ingested, what period the app reports, and how much source data is available.

ChicagoPulse currently models four official City datasets:

The product question was never merely, "Can Genie answer a question?" The harder question was, "Can someone understand the answer, inspect its evidence, and decide whether the underlying data is current enough for their purpose?"

02A civic-data chatbot is not enough

City open-data portals contain valuable records, but access is not the same as usability. A resident asking how 311 demand changed in Austin should not need to discover the correct dataset, decode its columns, learn Chicago's Community Area identifiers, choose a complete reporting period, and write a defensible aggregation before seeing an answer.

Natural-language querying removes much of that friction. It also creates a new failure mode: a fluent answer can hide the wrong metric, an incomplete month, stale data, or a plausible query built at the wrong grain.

That is especially uncomfortable in a civic context. ChicagoPulse is not emergency, legal, or policy decision support, but the product still needs to show its work. I treated evidence as part of the interface, not as an implementation detail buried in logs.

03The architecture behind the chat box

The deployed frontend and API run together as a Databricks App. React and TypeScript provide the interface, while FastAPI is the normalization and security boundary between the browser and Databricks services.

The data layer is more than a collection of uploaded CSV files. Fifteen notebooks cover setup, Bronze ingestion, Silver transformations, Gold metrics, spatial enrichment, semantic-base construction, Unity Catalog Metric Views, incremental refresh, and validation. The daily workflow has explicit ingestion, transformation, and validation stages.

That separation matters. Genie should reason over defined business measures, not infer metric semantics from raw operational columns on every question. ChicagoPulse Metric Views expose measures such as total 311 requests, open building violations, permit fees, and month-over-month change with governed dimensions for month and Community Area.

The app then binds three Databricks resources at deployment time: a Genie space, a SQL Warehouse, and the daily refresh job. It authenticates as a dedicated app service principal instead of putting a personal token or resource credential in browser code.

04Why the browser cannot send arbitrary SQL

One of my most important choices was to make ChicagoPulse deliberately less flexible.

Genie owns the natural-language analytical conversation inside its governed space. The app's other data endpoints, such as neighborhood maps, comparisons, and Data Health, do not accept arbitrary SQL from the browser. FastAPI maps those requests to allowlisted, parameterized query templates. Resource selection also stays on the server, so a client cannot choose a different Genie space, warehouse, job, notebook path, or set of job parameters.

The trade-off is real. A user cannot turn the neighborhood endpoint into a general-purpose analytics API. I accepted that constraint because the route has a specific product job. It should return a neighborhood pulse, not execute whatever SQL arrives over HTTP.

This split also made the public API simpler. Raw Databricks payloads and state messages are normalized before they reach the browser. Empty, failed, cancelled, and expired Genie messages become explicit application states. Upstream errors do not leak credentials or internal payloads.

05A 30-question benchmark keeps Genie changes honest

A polished demo question proves very little. It usually proves that someone tested the demo question.

ChicagoPulse has a 30-question Genie benchmark with verified SQL. The questions are grouped across six categories:

• basic metrics;
• neighborhood comparisons;
• 311 drilldowns;
• time trends;
• cross-dataset reasoning; and
• freshness and coverage.

The suite ranges from straightforward prompts such as "How many 311 requests were recorded across Chicago in the last completed month?" to cross-dataset questions such as identifying Community Areas where 311 requests and building violations both rose by more than 10 percent.

The benchmark is part of the maintenance loop. Answer feedback does not automatically retrain Genie or mutate its instructions. Instead, maintainers review recurring feedback, verify proposed corrections against governed data, turn proven patterns into instructions or verified SQL examples, and run the benchmark before and after the change.

That process is slower than feeding every negative rating directly back into the system. It is also auditable. User feedback is evidence to investigate, not ground truth by default.

06Data Health makes trust visible

Many data products display a recent timestamp and call the pipeline healthy. ChicagoPulse does not infer end-to-end success from MAX(_ingested_at).

An ingestion timestamp answers one question: when did records last arrive? It does not prove that transformations completed, Gold tables were rebuilt, semantic views remained valid, or quality checks passed. Data Health therefore separates source freshness from pipeline success.

The surface shows the last successful refresh, latest completed reporting month, source coverage, row counts, ingestion timestamps, and links back to the official City datasets. A maintainer can explicitly start the one bound refresh job after a confirmation step, then inspect the real Jobs API state for ingestion, transformation, and validation. The interface never fabricates progress.

Two smaller display rules prevent surprisingly large analytical errors:

• Neighborhood metrics use only fully completed calendar months, so a partial current month is not compared with a full prior month.
• Missing source coverage renders as N/A, not zero. Zero is a measured result; unavailable means the app does not have enough evidence to make that claim.

These details are not visually dramatic. They are the details that determine whether someone misreads the chart.

07The hardest problems were semantic, spatial, and operational

The chat interface was not the hardest part of ChicagoPulse.

Time semantics were harder than they looked. "Latest" is ambiguous when a source is updated during the current month. I standardized the product on the latest completed month for neighborhood metrics and made the reporting period visible.

Geography needed explicit enrichment. City records do not all arrive with equally usable Community Area fields. The pipeline includes official Community Area boundaries and spatial enrichment for building violations so that neighborhood comparisons share a common geographic vocabulary.

Genie is asynchronous. A conversation response can be pending, complete, empty, failed, cancelled, or expired. The FastAPI layer normalizes those states so the React app has a stable contract instead of coupling every component to upstream response shapes.

Free Edition forced useful discipline. The production snapshot runs on Databricks Apps Free Edition, where access to the live app may require permission in the connected Databricks account. That constraint reinforced the need for deterministic mock providers, local lifecycle testing, and a production fail-closed rule that refuses to start when mock mode is enabled.

None of those choices appears in a screenshot of a successful answer. Together, they are most of the reason I trust the screenshot.

08What ChicagoPulse does not claim

ChicagoPulse summarizes public administrative data for neighborhood-level exploration. It is not real-time, address-level, emergency, legal, or policy decision support. Source records can be revised by the City, and the live experience may require Databricks account permission.

The Data Health catalog identifies food inspections, crimes, traffic crashes, and affordable rental housing developments as possible future sources. They are visibly marked as future scope. They are not currently ingested, modeled, or available to Genie.

That distinction is intentional. A roadmap label must not look like current data coverage.

09What winning changed for me

The contest result validated more than a UI or a single Genie response. It rewarded the whole path from source records to a governed semantic layer, from a natural-language question to inspectable SQL, and from an ingestion timestamp to evidence that a pipeline actually completed.

If I were starting again, I would define the benchmark categories and trust states before polishing the interface. Evaluation questions expose semantic gaps early. Explicit states expose product ambiguity early. Both are cheaper to address before users build expectations around the happy path.

I would also preserve the boundary between conversational flexibility and deterministic application queries. It is tempting to route every data interaction through one AI interface. ChicagoPulse is better because it does not. Genie handles open-ended questions over governed semantics; fixed product journeys use constrained APIs.

The Gold banner is the visible outcome. The reusable lesson is less glamorous: a Genie-powered app becomes credible when governance, evaluation, provenance, and operational truth are designed into the product from the start.

10Try ChicagoPulse and inspect the evidence

• Read the official Databricks Community winners announcement.
• Open ChicagoPulse on Databricks Apps. Access may require permission in the connected Databricks account.
• Explore the four official City datasets linked above and compare their raw structure with the questions the app supports.
• Review the ChicagoPulse source code on GitHub for the full React, FastAPI, and Databricks pipeline notebooks.
• [Coming Soon: ChicagoPulse demo video URL.]

If you are building with Databricks Genie, start your next design review below the chat box. Ask what defines each measure, which questions form the regression suite, what evidence a user can inspect, and how the interface distinguishes stale, missing, partial, and failed data. Those answers will matter long after the demo ends.

To be upfront: Packt sent me a reviewer copy, and I read it as someone who writes DAX for actual Power BI work, not as a first-timer. I recently built powerbi-dashboard-generator, so I went in with specific scars and specific questions. This review is about how the book teaches, and whether the scenarios hold up when you sit down to build them.

The short version: this is the strongest advanced DAX resource I have read, precisely because it refuses to be a function reference. It is a book of hard problems, solved well.

★★★★★★★★★★4.5 / 5

Quick verdict: Extreme DAX, Second Edition is a scenario playbook disguised as a DAX book. Its real subject is how experienced practitioners think through hard modeling problems. Best for intermediate-to-advanced analysts who are done with tutorials and stuck on the real thing.

AuthorsRozema, Stikkelorum, Vlootman

PublisherPackt (2026)

Length554 pages · 14 chapters

FormatBook + PBIX repo

01What is Extreme DAX, Second Edition?

Extreme DAX, Second Edition is an advanced Power BI and DAX book (Packt, 2026) that teaches the language through 13 real business scenarios rather than a catalog of functions. It runs 554 pages across 14 chapters and is written by Michiel Rozema, Madzy Stikkelorum, and Henk Vlootman, with a foreword by Jeroen ter Heerdt, who was DAX Product Manager for Power BI at Microsoft until March 2026.

The book assumes you already write DAX. It deliberately skips the "here is what SUM does" material, on the reasonable grounds that a search engine can tell you the arguments of a function. What it spends its pages on instead is the gap between knowing the functions and solving the problem: the modeling decisions, the context gymnastics, and the "why does the visual show this" moments that no function list prepares you for.

02The thesis: procedural learning, not another reference

Jeroen ter Heerdt's foreword frames the book better than any blurb could, using a distinction from learning science: declarative learning (concepts, rules, theory) versus procedural learning (doing, practice, pattern, intuition). DAX has plenty of the former and very little of the latter.

That is an accurate description of how it reads. In ter Heerdt's words, "reading it is close to the experience of looking over the shoulder of someone who has solved your exact problem before." That is exactly the resource that is hard to find, and it is why the scenario format works.

03Why the structure is the argument

The book is built so the foundation carries the harder material, and that ordering is deliberate:

• Chapters 1 to 4 set the base: analyzing data with DAX, model design, using DAX, and the one that everything else leans on, context and filtering (row, query, and filter context, CALCULATE, table functions, and variables).
• Chapters 5 to 9 are the first wave of real scenarios: security with DAX, dynamically changing visualizations, inventory analysis, alternative calendars, and working with Auto-Exist.
• Chapters 10 to 13 are the deep end: recursion in DAX, DAX-driven waterfalls, benchmarking with window functions, and real-estate investment planning.

Read that way, the "advanced" label is earned. Each scenario is a self-contained project with a business case, a model, and a worked solution, and most ship with a downloadable PBIX so you can open the file and see exactly how a measure behaves inside a visual.

04The scenarios that stuck with me

These are the chapters I expect to actually reach for, not just the ones that read well.

The line from the preface that captures the second edition best is a quiet flex:

"Many of the scenarios from the first edition of Extreme DAX can be, and should be, solved in another way. We can even go beyond to things that were practically impossible three years ago."

05What changed from the first edition

If you owned the 2022 edition, this is not a cover refresh. The problems are familiar, but the solutions are rebuilt around features that have landed since: field parameters, calculation groups, custom calendars, visual calculations, and user-defined functions. Several chapters that used to lean on awkward workarounds now have cleaner answers, and the recursion and waterfall chapters in particular exist because the language finally supports them well.

The framing changed too. DAX semantic models are now positioned as the center of the Microsoft Fabric platform rather than a Power BI feature, which is the right call for where the stack is heading.

06Where it falls short

No book earns a real review without the criticism, so here is mine.

It is genuinely advanced, and it will lose beginners fast. The authors say plainly that they do not explain most functions and arguments, because you can find that online. That is the correct decision for the audience, but it means someone still learning CALCULATE will drown by Chapter 5. This is a second book, not a first.

Some scenarios use preview features. The book notes that a few features were not generally available at the time of writing, so you may need to enable preview options to follow along. Useful now, but a reminder that the newest material is also the least stable.

Depth over breadth cuts both ways. Because each chapter goes deep on one business case, the book is not a lookup you skim for a quick pattern. You get the most from it by building the PBIX alongside the text, which is a bigger time commitment than a reference.

It is Power BI and Fabric specific. The modeling instincts transfer, but the solutions assume the Microsoft semantic model engine. If you work in another BI stack, the thinking helps and the code does not.

07How Extreme DAX compares to other DAX books

08How it connects to everything else I have read and built

The most interesting thing about this book is not any single measure. It is a pattern I keep seeing across the tools I work with: the interesting work is moving out of the raw engine and into the layer of control, governance, and reusable logic wrapped around it.

DAX getting user-defined functions, recursion, and goal-seeking is the same move, in miniature, that AI agents make when they wrap a model in tools, memory, and verification. I wrote about that shift in my reviews of Principles and Patterns of Building AI Agents and 30 Agents Every AI Engineer Must Build: the value lives in the architecture around the primitive, not the primitive itself. Chapter 10's UDF-driven recursion reads like DAX growing the same procedural escape hatches.

The security chapter lands the same way against the platform trend. Value-level security, aggregation security, and the Fabric-centric framing are all about turning the semantic model into a governed layer that feeds both dashboards and, increasingly, AI. That is exactly the direction I traced in my Databricks Well-Architected Framework and Databricks Data + AI Summit 2026 write-ups: both ecosystems are converging on a governed semantic and metrics layer as the source of truth. Reading Extreme DAX next to those, the Microsoft answer to that problem comes into focus.

And on the practitioner side, three chapters map straight onto problems I hit building powerbi-dashboard-generator: field parameters for user-driven visuals (Chapter 6), Auto-Exist surprises when a visual quietly drops rows (Chapter 9), and calculation groups for keeping time intelligence sane across metrics (Chapter 8). Those are not academic examples. They are the exact places a real dashboard breaks.

09Who should actually read it

Read this if you already write DAX for a living, or close to it, and you keep hitting the wall between "I know the functions" and "I cannot make this specific thing work." Security beyond RLS, custom calendars, recursion, benchmarking, and financial modeling in DAX are all here, worked end to end.

Skip it, or shelve it for later, if you are still learning the basics. It will not hold your hand, and it is not trying to.

10Final word

Extreme DAX, Second Edition is a book of hard problems solved by people who have clearly solved them for real. The scenario format is the right one for advanced DAX, the second edition earns its update by rebuilding old solutions with new language features, and the PBIX-per-chapter approach means you can actually feel the difference between reading a pattern and running it. It loses half a point only for being genuinely inaccessible to beginners and leaning on a few preview features. 4.5 / 5, and it is going on the shelf next to the reference I reach for when I am stuck.

Disclosure: Packt sent me a reviewer copy of this book. Thank you to Shruthi Shetty for reaching out and getting it to me, and a real thank you to Michiel Rozema, Madzy Stikkelorum, and Henk Vlootman for writing it, and to Jeroen ter Heerdt for a foreword that framed the whole thing.

You can find the book on Amazon and via Packt, and the companion Power BI files on GitHub. If you read it, I would like to know which scenario you built first.

I found this out the practical way. I used the PWAF to design and submit a Databricks solution accelerator for WorldLink, and the framework was genuinely useful, but the most valuable thing in it was not a pattern or a diagram. It was a single sentence buried in the Customer Managed deployment page.

This post is the guide I wanted when I started: what the PWAF is, which of the three partner tracks you belong to, what the four pillars actually demand, how to choose between the four deployment models, and which requirements are hard gates versus suggestions.

"This badge is awarded to people who complete the ISV Partner Well Architected Frameworks Training and Knowledge Check with a score of 80% or higher. This course is designed to teach ISVs the basic knowledge of Partner Well Architected Frameworks, as well as give you the main resources and tools for building solutions utilizing PWAF."

Partner Well Architected Frameworks Solution Development Knowledge Check Assessment ISV Partnership Skills Resource Utilization

Verify this credential

IssuerDatabricks Academy

CredentialPartner Training, ISV Partner Well Architected Frameworks

IssuedJuly 31, 2026

The training is worth the hour, but it teaches you the map, not the terrain. What follows is the terrain.

01What is the Databricks Partner Well Architected Framework?

The Partner Well Architected Framework (PWAF) is Databricks' architecture guide for technology partners building products, integrations, and data shares on the Databricks Data Intelligence Platform. It is published by Databricks Labs at databrickslabs.github.io/partner-architecture and organized around three partner types and four architectural pillars.

The framing matters. Databricks explicitly positions the PWAF as an extension of the cloud Well Architected Frameworks (AWS, Azure, GCP) and the Databricks Lakehouse Architecture, applied to problems those frameworks do not cover: multi-tenant SaaS architectures, customer data integration, marketplace distribution, and partner attribution.

That last one, attribution, is the tell. The cloud frameworks care whether your system is reliable and secure. The PWAF also cares whether Databricks can see your product working, because partner economics depend on measurable joint impact. Read the whole framework through that lens and a lot of otherwise-arbitrary requirements start making sense.

One practical note before you start: the PWAF is deliberately built for AI-assisted development. Every page has a copy-link action and an AI assistant dropdown with pre-loaded prompts, and the site publishes two machine-readable context files, an llms.txt site map and an ai-context.txt guidance file. If you are going to work through the framework with Claude or Cursor, point the tool at those files first instead of pasting pages one at a time.

02Who the PWAF is for: the three partner tracks

Before reading anything else, work out which track you are in, because the technical standards differ per track and you will waste time reading requirements that do not apply to you.

The boundaries blur in practice. A notebook-based solution accelerator that a customer runs in their own workspace is closest to the Built-On track under the Customer Managed deployment model, even though it feels nothing like a SaaS product. That mismatch is worth thinking about early, and I will come back to it.

One thing the PWAF is clear about: the framework is technical guidance, not membership. To get go-to-market benefits, co-sell, and program incentives you have to be an active member of the Databricks Partner Program, and validation requests go through the Partner Portal. Building well and being a partner are separate steps, in that order or not.

03The four PWAF pillars, and which ones actually gate you

Databricks defines four pillars:

1. Architectural Best Practices. Design principles, repeatable patterns, reference architectures, and implementation guidance.
2. Defined Technical Standards. Per-partner-type requirements that must be met for validation and program participation, with a stated path to validation.
3. Measurement and Attribution. Continuous measurement of adoption and impact, and partner attribution through telemetry.
4. Operations and Lifecycle Management. Monitoring, automation, runbooks, access management, and continuous improvement workflows.

Here is the part nobody tells you: engineers spend roughly all of their effort on pillar 1 and almost none on pillar 3, and pillar 3 is closer to a gate.

Pillar 1 is fun. It is diagrams, patterns, and design debates. It is also where you have the most latitude, because "good architecture" has many valid shapes.

Pillars 2 and 3 are checklists with binary outcomes. Did you embed a programmatic User-Agent? Did you register your data assets in Unity Catalog? Is OAuth implemented, or are you still passing personal access tokens around? These are yes-or-no, and they are cheap to do early and expensive to retrofit.

Pillar 4 is the one that gets deferred to "after we ship," which is why so many partner solutions have no runbook, no documented upgrade path, and no support boundary. The Customer Managed guidance is blunt about this: define clear boundaries for support, provide clear versioning for all artifacts, document upgrade paths and breaking changes, and consider backward compatibility for long-term customers. That is a pillar-4 to-do list, and it takes a day.

04Connected ISV: the requirements that are genuinely mandatory

If you are connecting an existing product to Databricks, the integration requirements page is the highest-value page in the entire framework. It is short and it is specific. These are the hard requirements.

OAuth 2.0 is mandatory, and personal access tokens are not a strategy

OAuth is mandatory for all ISV partner integrations and is the standard mechanism for secure production authentication. Two flows are supported:

• User-to-machine (U2M). A browser-based OAuth flow where the user authenticates with their own identity and your app acts on their behalf. Relevant docs cover JDBC, ODBC, OIDC U2M, and token federation.
• Machine-to-machine (M2M). Client credentials with service principals, so jobs, services, and scripts reach Databricks with no signed-in user. This covers Workload Identity Federation, account token federation, and Databricks OIDC M2M.

The security and lifecycle guidance is standard but worth quoting as a checklist, because reviewers will look for it: request least-privilege scopes, persist tokens securely, rotate refresh tokens, use PKCE, never log tokens and ensure redaction, support revocation, and provide a clear disconnect and re-auth experience.

That last item is the one I see skipped most. Plenty of integrations can connect. Far fewer can cleanly disconnect.

Unity Catalog registration is not optional if you touch data

The rule is broad: if an ISV integration reads, writes, stages, or manages any type of data asset, the integration must register and operate on those assets in Unity Catalog. "Data asset" is defined widely too, covering structured and tabular data (tables, views, materialized views, metric views, UDFs), unstructured files (images, documents, logs, binaries, ingestion files), and ML models.

Practically, this means three-level namespacing (<catalog>.<schema>.<table>) everywhere, no side-channel storage that bypasses governance, and metadata access that complies with the Databricks Acceptable Use Policy.

Telemetry: one string, programmatically set, verifiable in a system table

This is the requirement that most directly serves the Measurement and Attribution pillar, and it is a small amount of work with outsized consequences.

You must programmatically embed a stable partner User-Agent identifier in every Databricks API call, driver or SQL connection, SDK or connector call, job, CLI invocation, or other integration request originating from your product. The format is built from isv-name, product-name, and product-version, separated by underscores. Each product or integration needs a distinct value.

Two details matter more than the format:

You cannot outsource it to the customer. The framework says it plainly: partners must set this value programmatically in the connection code path, and you cannot rely on joint customers to configure it themselves. It calls this a strict requirement.

You can verify it yourself. The Source column in Query History reflects the User-Agent used by the partner application, and for full coverage across workloads you query the audit system table:

SELECT *
FROM system.access.audit
WHERE event_time > current_timestamp() - INTERVAL 2 days
  AND lower(user_agent) LIKE '%<your-user-agent>%';

Run that before you submit anything. If it returns nothing, your attribution does not exist, whatever your architecture diagram says.

Genie attribution has specific UI requirements

If your product surfaces Genie output, four things are required: a visible "Powered by Genie" label wherever Genie-generated content appears, a citation linking directly to the Genie Space that produced the result (constructible from the space_id returned by the Genie API), any links in the Genie response surfaced and navigable, and the same attribution in product surfaces like settings or integration dialogs and in your documentation.

And a hard one for AI products: agent products must invoke Genie via MCP, and agent workflows that do not use the Genie MCP will not be validated. If you built an agent that talks to Databricks through a bespoke SQL path, that is a rewrite, not a tweak.

Documentation is a deliverable, not an afterthought

Partners must provide written documentation that lets a customer connect the product to Databricks, and the framework specifies its contents: prerequisites, the Databricks privileges the connecting user or service principal needs, which authentication types are supported, and step-by-step setup for each supported auth type. PDF is accepted at submission, live docs on your site are preferred.

I would go further and write this document first. It forces you to answer "what privileges do we actually need" before you have hard-coded an over-privileged service principal into a demo.

Product categories change the specifics

Beyond the common requirements, the framework splits Connected ISV guidance into five product categories: Data Engineering and Ingestion, AI Agents and Tools, Business Intelligence, Governance and Observability, and Apps and Dev Tools. Find your category and read its page, because category-specific best practices are additive to the shared requirements, not a substitute.

05Data Collaboration: productizing data instead of shipping software

The Data Collaboration track assumes your data already exists in Databricks and is registered in Unity Catalog. It does not cover upstream ingestion or ETL, which is a useful scoping decision.

Sharing breaks into named patterns:

The engineering advice in this track is less about protocols and more about making data usable, and it lands on metadata. Unity Catalog lets you attach table and column descriptions, tags, business terms, and relationships, and the framework ties this directly to AI readiness: when Genie can read complete and accurate metadata, users can query and explore your data in natural language.

That is a real incentive change. Metadata used to be documentation hygiene you did for humans who mostly ignored it. Now it is the interface that determines whether natural-language querying works on your data product. If you share a table with cryptic column names and no descriptions, you have shipped a dataset that AI tooling cannot use well, and your consumers will notice.

Two more operational notes worth acting on: enable Change Data Feed where incremental reads and automation matter, and plan schema evolution and backward compatibility so your updates do not break consumer workloads. A shared table is an API. Breaking changes behave like breaking changes.

06Built-On ISV: the five-layer application architecture

If Databricks is your backend, the framework expects a specific application shape. For most Built-On products you build an application that sits between your end users and the platform, in five layers:

1. Client Layer        Browser frontend (React, Angular, Vue), renders UI,
                       makes authenticated API requests
2. Authentication      OAuth 2.0 / OIDC with your identity provider,
                       user auth and session management
3. Backend Layer       Server-side API routes, validates sessions, manages
                       tokens, proxies ALL Databricks API calls
4. Data Layer          Application database (PostgreSQL, MySQL): users,
                       organizations, encrypted credentials
5. Databricks          Unity Catalog, SQL Warehouses, DBFS, Jobs, and other
                       services via REST APIs

The design goal is stated explicitly: Databricks credentials never reach the client, authentication boundaries are clear, and each layer scales independently.

The load-bearing word there is "all." Every Databricks call is proxied through your backend. It sounds obvious written down, and it is the first thing that gets violated when someone wants a fast dashboard and calls the SQL Warehouse straight from the browser with a token in local storage. If you take one structural rule from the Built-On track, take that one.

Layer 4 is the one people under-plan. You need somewhere to store users, organizations, and encrypted customer credentials, which means you are running an application database alongside Databricks and inheriting all the key-management responsibility that comes with storing customer credentials at rest.

07How to choose a Databricks deployment model

This is the decision the rest of your architecture hangs off. Databricks documents four models.

The framework gives five selection criteria, and they are good questions in the order they are asked:

1. Data residency. Does the customer's data need to stay in their environment?
2. Operational capacity. Do you want to manage infrastructure, or does the customer prefer to?
3. Customer sophistication. Does the customer have a platform team capable of managing Databricks?
4. Regulatory constraints. Do compliance requirements dictate where data can reside?
5. Scale and economics. Which model gives you the best unit economics?

Databricks also notes that many partners offer more than one model, typically Partner Hosted for customers who want simplicity and Hybrid for enterprise customers with data residency or compliance requirements. Treat the models as a product-line decision, not a one-time architectural choice.

Side Car deserves a closer look because it is the least familiar of the four. Partners build and govern assets centrally, then deliver them into customer environments via OpenSharing and Clean Rooms, keeping control of asset quality, versioning, and IP while the customer keeps authority over their data and execution environment. Customers can consume shared datasets from non-Databricks systems and run compute wherever they like, while Databricks centralizes policy enforcement and asset governance. If your value is the asset rather than the application, this is the model to read twice.

08The deployment model trap: Customer Managed limits your program benefits

Here is the sentence I mentioned at the top. On the Customer Managed page, Databricks states:

Due to limitations in telemetry and application management, program benefits are limited for Customer Managed deployments.

Sit with that for a second, because it connects two things that look unrelated.

The Measurement and Attribution pillar exists so Databricks can observe adoption and attribute joint impact to you. Customer Managed means the customer owns the workspace, the infrastructure, and the runtime. You ship notebooks, Databricks Asset Bundles, Terraform modules, or Python packages, and then you are not in the execution path. There is no product of yours making tagged API calls, because the customer's own workspace is making them.

So the model with the lowest operational burden for you is also the model where attribution is weakest, and attribution is what partner benefits are built on. That is not a hidden penalty, it is a structural consequence, and Databricks documents it.

Now consider what most consulting-built solution accelerators look like. A repo of notebooks, a job definition, a config file, a deployment guide. The customer clones it and runs it in their workspace. That is Customer Managed, by default, without anyone having chosen it.

If you are building an accelerator and you want program benefits, you have three honest options:

• Accept the tradeoff. Build Customer Managed because it fits the buyer, and treat the accelerator as a services differentiator and credibility asset rather than a route to partner benefits.
• Move up a model. Wrap the same logic in a Partner Hosted or Hybrid application so there is a product in the execution path that can carry a User-Agent and be managed and versioned by you.
• Go Side Car. If the output is a governed asset (a metric view, a curated dataset, a model), deliver it via OpenSharing so you keep central control of quality and versioning while the customer keeps their data.

What you should not do is build Customer Managed by accident and then be surprised when the attribution story is thin. I would have made a different architectural choice at the start if I had read that page before designing rather than during.

09Firefly Analytics: read the reference implementation, not just the prose

The single most useful resource attached to the PWAF is Firefly Analytics, an open-source SaaS application published by Databricks Labs specifically as a PWAF reference implementation. Source is at github.com/databrickslabs/firefly, it runs live at firefly-analytics.com, and the architecture docs are at firefly-analytics.com/docs.

Its stack is documented rather than implied:

• Frontend: Next.js 15, React 19, TypeScript, Tailwind CSS, shadcn/ui, TanStack Query
• Backend: Next.js API routes, PostgreSQL with Drizzle ORM, Better-Auth
• Databricks: Unity Catalog, SQL Warehouses, DBFS, Volumes, Databricks Apps
• Deployment: Vercel for the frontend, Docker, VM, or serverless for the proxy

The patterns it demonstrates map one-to-one onto the framework's abstract guidance: SSO-SPN authentication, Databricks identity handling, workspace-catalog binding for tenant isolation, application-tier auto-scaling, hybrid tenancy, and organization and user onboarding. It also demonstrates six things Built-On partners commonly build, including embedded Databricks Apps via a proxy, notebook and code and SQL editors, a data catalog, and a pipeline editor.

Two ways to use it that are better than reading it start to finish:

As a diff target. Pick the one pattern you are least sure about, credential isolation between tenants, say, and read only Firefly's implementation of it. Compare against yours. The gap is your work item.

As AI context. The framework explicitly suggests pointing AI tools at Firefly's source, which carries inline comments explaining PWAF principles in context. Combined with the llms.txt and ai-context.txt files, you can ask concrete questions ("how does Firefly bind an organization to a catalog?") and get answers grounded in working code instead of paraphrased docs.

Do not deploy it as your product. It is a reference implementation provided for learning, with no support commitment beyond documentation and your Databricks partner team.

10Field notes: mapping a real accelerator onto the PWAF

Abstract frameworks get concrete when you run something through them. Here is the accelerator I built at WorldLink US, the Semantic Model Migration Accelerator, described the way the PWAF would want it described.

What it does. Converts Power BI semantic models exported as TMDL into governed Unity Catalog metric views, with deterministic DAX-to-SQL translation, flagged-measure review bundles, and KPI parity validation against the Power BI baseline.

The pipeline. Five stages: parse .tmdl files for tables, measures, relationships, and partition sources; translate using configurable DAX-to-SQL, format string, and relationship mapping rules; generate metric view YAML plus model summaries and review bundles; deploy by registering metric views as Unity Catalog objects through the SQL Statement Execution API; validate by comparing KPI results against the Power BI baseline and producing reviewer-ready quality reports.

The numbers. A moderately complex model takes an engineer 2 to 3 days by hand. Automated conversion runs in under 2 minutes per model. Conservative time savings are 60 to 80 percent versus manual conversion, with internal benchmarks reaching over 95 percent on supported star-schema models. The repo carries 156+ automated tests passing in CI, 23 sample TMDL packs across topologies, and a reference workload migrating 12 measures.

Now the honest PWAF mapping, pillar by pillar.

Two things I would keep exactly as they are.

The explicit non-goals. The accelerator documents what requires additional scope as loudly as what works: Power BI Import-mode models (data must be materialized as Delta first), calculation groups and many-to-many relationships (roadmap), row-level security (reapplied directly in Unity Catalog), and composite models (only DirectQuery portions migrate automatically). Naming the boundary is a credibility feature. Reviewers and customers both trust a scoped tool more than a tool that claims everything.

Determinism as the supported path. Optional, bounded Mosaic AI review handles edge cases, but the production path stays deterministic. For a migration tool where the deliverable is numerical parity with an existing report, "the LLM will figure it out" is not a validation story. Parity is measurable, and a >95 percent parity target is a claim you can be held to.

The thing I got wrong was upstream of all of this: I designed a genuinely good Customer Managed solution without registering that Customer Managed is where program benefits are thinnest. The architecture was not the problem. The model selection was made implicitly by the delivery format.

11How to get started with the PWAF

In the order that avoids rework:

1. Join the Databricks Partner Program. The framework is public, but co-sell, go-to-market benefits, and validation submissions require active membership via the Partner Portal.
2. Take the ISV PWAF training. It is short, and it gives you the vocabulary and the resource map. The knowledge check needs 80 percent or higher.
3. Identify your track. Connected ISV, Data Collaboration, or Built-On. Read your track's requirements page in full before designing anything.
4. Choose your deployment model explicitly, and write down why. Use the five criteria. Record the decision with its tradeoffs, including the Customer Managed benefits limitation if that is where you land.
5. Implement the hard requirements first. OAuth, Unity Catalog registration, and the programmatic User-Agent. These are cheap now and expensive later.
6. Verify attribution with a query, not a hope. Run the system.access.audit check above and confirm your User-Agent shows up.
7. Read Firefly for the pattern you are least sure about. Diff its implementation against yours.
8. Write the customer connection documentation. Prerequisites, required Databricks privileges, supported auth types, step-by-step setup per auth type.
9. Produce the pillar-4 artifacts. Runbook, support boundary, versioning scheme, upgrade path, breaking-change notes.
10. Ask for architectural review before you submit. Partner Engineering takes technical questions through a Partner Support request in the Partner Portal. Use that earlier than feels necessary.

12PWAF best practices, and the mistakes that cost the most

Choose the deployment model before the tech stack. It determines your attribution ceiling, your operational burden, and which program benefits are even available. Deciding it implicitly through delivery format is the single most expensive mistake in this whole process.

Treat technical standards as acceptance criteria. OAuth, Unity Catalog, User-Agent, Genie attribution, MCP for agents. Put them in your definition of done, not your backlog.

Instrument attribution on day one. It is one string in your connection code path. Retrofitting it across every driver, SDK call, job, and CLI invocation later is tedious and easy to do incompletely.

Do not let customers configure your telemetry. The framework prohibits it, and even if it did not, customer-configured telemetry is telemetry you do not have.

Define the ideal customer profile and the explicit non-goals. A tool that says "star-schema models with SUM, DISTINCTCOUNT, DIVIDE, AVERAGEX, and time intelligence" is more trustworthy than one that says "any semantic model."

Make quality measurable. Parity percentages, test counts, benchmark workloads. "Significantly faster" is not a claim; "under 2 minutes per model versus 2 to 3 days manually" is.

Write the docs before the demo. Documenting required privileges first prevents you from quietly building on an over-privileged service principal.

Build a demo that works on someone else's data. Bundled reference workloads and multiple sample topologies exist for exactly this reason. A demo that only runs on your dataset is a video, not a solution.

Keep AI on the optional path when correctness is the deliverable. Deterministic core, bounded AI assist for edge cases, clearly labelled. This is easier to validate and easier to defend.

Plan for schema evolution if you share data. A shared table is an API contract, and CDF plus backward compatibility planning is how you avoid breaking consumers.

13Sources

Everything in this post is drawn from the official framework. If you only open three of these, make them the integration requirements, the deployment models overview, and the Customer Managed page.

14Related reading

If you want the platform context these partner patterns sit on top of, my breakdown of Databricks Data + AI Summit 2026 covers Genie, Unity Catalog semantics, and metric views in depth, which is most of what the PWAF assumes you already understand.

If you are building an accelerator right now, do one thing before you write more code: open the deployment models page, pick your model deliberately, and write down the tradeoff you are accepting. That paragraph is worth more than the next week of implementation.

To be upfront about what this review is: I read both cover to cover as an engineer trying to keep a mental model current, not as someone who shipped a Mastra app to production. So I am reviewing the ideas and the teaching, not the framework's runtime. That distinction matters, and I will come back to it.

The short version: this is one series in two volumes, and reading them as a pair is the point. Principles tells you what to build. Patterns tells you how to keep it alive once real users touch it.

★★★★★★★★★★4.25 / 5 as a series

Quick verdict: The most efficient on-ramp to agent engineering I have read. Principles (4 / 5) is a fast, honest primer. Patterns (4.5 / 5) is the one you will keep on your desk, because it is about the messy middle between a demo and production.

AuthorsSam Bhagwat and Michelle Gienow

PublisherSelf-published, 2025 to 2026

Length134 + 92 pages

PriceFree digital, paid print

01What are Principles and Patterns of Building AI Agents?

Principles of Building AI Agents is a short, conceptual guide to modern agents, now in its 3rd edition (March 2026). It runs about 134 pages across 33 chapters and ten parts: prompting an LLM, building an agent, tools and MCP, graph-based workflows, RAG, multi-agent systems, observability and evals, development and deployment, coding agents, and a grab-bag "everything else" on multimodal. It is deliberately pocket-sized.

Patterns for Building AI Agents is Volume 2 (October 2025), co-written with Michelle Gienow. It is roughly 92 pages of 22 numbered patterns grouped into four working parts: configure your agents, engineer agent context, evaluate agent responses, and secure your agents. Where Principles is textbook, Patterns is a collection of lessons learned from teams pushing agents into production.

Both are written by Sam Bhagwat, CEO of Mastra, the open-source TypeScript agent framework, and previously a co-founder of the Gatsby React framework. Both are free.

02The thesis: what to build, then how to build

Bhagwat states the relationship between the two books plainly, and it is the most useful framing in either one:

That is not marketing filler. It is an accurate description of how the two books actually feel to read. Principles gets you fluent. Patterns is where the scar tissue lives.

03Principles: a primer that respects your time

The best decision in Principles is its length. Bhagwat says in the foreword that it "should fit in your back pocket," and that you should be able to get something working in a day or two. He keeps that promise. There is no throat-clearing.

A few things it does genuinely well:

It anchors on a clean definition. It borrows Simon Willison's line that an agent "calls tools in a loop to achieve a goal," then lays agency out as a spectrum, from binary choices in a decision tree, to memory and retries, up to planning, subtask decomposition, and self-correction across long horizons. That ladder is a better mental model than the usual "agents are magic" hand-waving.

It has a spine of engineering judgment. The line I kept coming back to is "make it work, make it right, make it fast/cheap, in that order." It shows up when choosing a provider, when picking model size, and when deciding whether to move workloads to cheaper open models later. It is the kind of advice that sounds obvious and that people ignore constantly.

Tool design gets the emphasis it deserves. The chapter on tool calling argues that designing your tools is the single most important step, and it should happen on paper before you write code. The illustration, investor Alana Goyal's book-recommendation agent, is a clean before-and-after: dumping every book into the context window failed, and breaking the problem into specific tools ("get books by genre," "get recommendations by investor") made the agent behave like a real analyst. Think like an analyst, then write each operation as a tool.

Memory is treated as a design choice. Principles names three memory types (working memory, semantic recall, and observational memory) and spends real pages on observational memory, where an "observer agent" compresses raw sessions into structured observations and a "reflector agent" garbage-collects when they overflow. That is a more honest picture of production memory than "just stuff it in the context window."

Where Principles is weakest is the flip side of its virtue: it is thin. Some chapters are two pages. If you already build agents, several sections will feel like a table of contents with commentary. That is a fair trade for a free primer, but it is why I score it a notch below its sequel.

04Patterns: the half worth paying for (even though it is free)

Patterns is the one I would hand to a team about to ship. It is organized as problem, then solution, then a real example, and the examples are specific enough to be useful rather than decorative. Three parts stood out.

Context engineering, treated as its own discipline

The context section is the clearest short treatment of context engineering I have read. It names five concrete failure modes rather than waving at "the model got confused":

The example that sells it: a Google Gemini team benchmarking a Pokemon-playing agent watched accuracy start degrading around 125k tokens, even with a 500k-token window. By filtering with RAG, pruning irrelevant context, and assembling a compiled context string before every call, they took a research agent from 34% to reliably over 90% accuracy. The book's blunt summary is that "context is not free," and every token influences behavior for better or worse.

Evals as engineering, not vibes

The evals part is built around a recurring, believable case study: a medical necessity-review agent for health insurers, whose north-star metric is "false approvals." The workflow it lays out is the honest version of quality work: list your failure modes and why they happen, connect engineering metrics to a business metric, cross-reference the two, then iterate against a benchmark in CI so a fix "here" does not silently break something "there."

The pattern I found most quietly important is "Have SMEs Label Data." The argument is that software engineers are not your domain experts, so they should not be the ones judging whether a medical approval was correct. Outsourcing annotation breaks the loop between seeing a failure and understanding it. That is a governance point disguised as a labeling tip.

"Benchmarks are the difference between engineering and experimentation."

Security, written for the way agents actually break

The security part opens with the right observation: traditional security assumes humans click buttons and code runs deterministically, and agents break both assumptions by interpreting instructions from anywhere and acting across systems.

Its centerpiece is Simon Willison's "lethal trifecta": access to private data, exposure to untrusted content, and the ability to communicate externally. Combine all three and a prompt-injection attack can read your secrets and exfiltrate them. The GitHub MCP server is the worked example, and it covers the whole trifecta at once. The practical advice is to remove one leg, usually the exfiltration path, via input processors. The rest of the part is equally grounded: sandbox untrusted code (with fast-start runtimes like E2B or Daytona, since a Docker cold start is too slow), grant granular just-in-time access rather than broad roles, and add real-time guardrails on both inputs and outputs.

05The patterns that stuck with me

These are the ideas I expect to actually reuse, pulled from across both books.

06Where the series falls short

No book earns a real review without the criticism, so here is mine.

Breadth beats depth, and Principles feels it most. This is the same complaint you will find on Goodreads, and it is fair. Principles skims. It is a wide, shallow tour by design, so anyone hoping to master RAG or multi-agent orchestration from it will be disappointed. It is a map, not the territory.

It is Mastra's book, and it shows. The concepts are portable, but the code and several APIs are Mastra-specific and TypeScript-first. That is reasonable for a free book from a framework company, and the plugs are examples rather than hard sells, but Python engineers should know the snippets will not copy-paste.

The stack will age faster than the ideas. Both books name specific models, token thresholds, and tools. Useful now, dated in eighteen months. The durable parts are the distinctions: single-threaded versus parallel, generation versus verification, autonomy versus guardrails.

Patterns is a field guide, not a manual. At 92 pages it is dense and original, but each pattern is a strong sketch, not an end-to-end implementation. You will still need docs and code to actually ship any one of them.

07How the two books compare

08Who should actually read it

Read Principles if you are a developer, ideally in TypeScript, who wants a fast and jargon-light on-ramp to how agents work. You can finish it in an evening and immediately hold better conversations about agents.

Read Patterns if you have a working prototype and are staring down the gap between "it demos well" and "it survives real users." That is the book's whole reason to exist, and it delivers there.

If you want a deeper, Python-first, and far more exhaustive treatment of agent architectures, the two Mastra books pair well as the lean counterpart to a heavier text. I wrote about one such book, a 542-page pattern library, in my review of 30 Agents Every AI Engineer Must Build. Read together, the contrast is instructive: one is a comprehensive reference you study, the other is a pocket guide you keep open while you work.

09Final word

Bhagwat's series wins on a rare axis: respect for the reader's time. Principles (4 / 5) gets you fluent in an evening. Patterns (4.5 / 5) is the one I would keep open while building, because it is about the part everyone skips: what happens after the demo works. That the whole thing is free, and this good, is faintly ridiculous.

Thank you to Sam Bhagwat and Michelle Gienow for writing them, and for giving them away. You can download both at mastra.ai/book, or grab print copies of Principles and Patterns on Amazon. If you read them, I would like to know which pattern you reached for first.

The short version: the title undersells the book. It sounds like a listicle in hardcover. It is not. This is a book about agent engineering as a discipline, and the "30 agents" are the vehicle, not the point.

★★★★★★★★★★4.5 / 5

Quick verdict: 30 Agents Every AI Engineer Must Build is a pattern library disguised as a catalog. Its real subject is the architecture around the model, not prompting tricks. Best for engineers moving from LLM features to production agent systems.

AuthorImran Ahmad, PhD

PublisherPackt (2026)

Length542 pages · 17 chapters

FormatBook + executable repo

01What most reviews get right, and what they miss

The public conversation around this book is still thin, which makes sense for a 2026 release. Packt and Amazon list a 4.5 rating, and the recurring praise is fair: people call it practical, architecture-first, and production-minded. Goodreads notes that it moves cleanly from basic to advanced multi-agent thinking.

All true. But that generic praise misses the actual thesis, and the thesis is the reason the book is worth your time.

Most agent books teach you to call a model well. This one teaches you to build the system around the model. The preface says it plainly: the shift from LLM apps to agents is architectural, not incremental, and raw model calls are not enough. Everything after that earns the claim.

02The hidden idea: the agent is the architecture, not the model

If you take one sentence away from the book, make it this:

Ahmad keeps returning to a single core loop, the cognitive loop of perception, reasoning, planning, action, and learning, and treats everything else (interoperability, lifecycle, prompting, deployment, security, governance) as part of agent engineering rather than as cleanup at the end.

That is why the first third of the book matters more than the "30 agents" framing suggests. Chapter 1 gives you the vocabulary: what actually separates an agent from a script (autonomy, persistence, reactivity, proactiveness, adaptability, goal orientation), the reactive/deliberative/hybrid families, protocols like MCP and A2A for connecting to tools and other agents, and an Agentic AI Progression Framework for classifying maturity from manual operations up to learning agents.

By the time you reach the domain chapters, the point is unmistakable. Healthcare, finance, law, education, and robotics do not need "magic agent prompts." They need different control structures, memory models, validation gates, and escalation rules.

03Why the structure is the argument

The book is built so the foundation carries the rest, and that ordering is deliberate:

• Chapters 1 to 4 build the operating system: agent foundations, the toolkit (LangChain, LangGraph, LlamaIndex, CrewAI, AutoGen, vector databases, cloud platforms), prompting, and deployment with security and responsible AI.
• Chapters 5 to 10 are the reusable building blocks: cognitive architectures, retrieval and knowledge, tool orchestration, analysis and verification, software agents, and conversational or content systems.
• Chapters 11 to 16 adapt those blocks to hard domains: multimodal perception, ethics and explainability, healthcare, finance and law, education, and embodied or physical-world systems.
• Chapter 17 looks forward to agent societies, governance, and organizational roadmaps.

Read that way, "30" is less a count and more a curriculum. You learn what kinds of agent problems exist, which architecture each one needs, and what breaks when you reach for the wrong one.

04The patterns that actually stuck with me

These are the parts I kept coming back to while building, not the parts that just read well.

Two lines from the book capture the tone well:

"This transformation is architectural, not merely incremental."

"Raw LLMs alone are not enough."

The companion GitHub repository is worth pulling down. It ships executable, per-chapter code (with a simulation mode so you can run patterns without wiring up every external service) and it is the fastest way to feel the difference between reading a pattern and running it.

05Where it falls short (the honest part)

No book earns a real review without the criticism, so here is mine.

Breadth wins over depth, and you can feel it. Across 542 pages and 17 chapters spanning software, healthcare, law, finance, education, multimodal systems, and robotics, some later domain chapters read more like strong reference architectures than end-to-end implementation manuals for someone already deep in that field. That is a defensible tradeoff for a pattern library, but go in expecting a map, not a fully paved road, in the vertical chapters.

The stack will age faster than the ideas. The book names concrete frameworks, models, and versions. Useful today, dated in eighteen months. The architectural distinctions (reactive versus deliberative, single agent versus chain-of-agents, generation versus verification, autonomy versus governance) are the part that will still hold.

Case studies are illustrative, not field-validated. The companion material notes that several "real-world" scenarios use fictional companies navigating real constraints. Good for teaching, but read the business outcomes as scenario design, not independent proof.

The hands-on path leans LangChain and LangGraph. Not a flaw so much as a bias worth knowing before you start.

06How it compares to typical AI agent books

07Who should actually read it

Read this if you already know Python and basic ML and you are trying to move past "prompt in, answer out" into systems that plan, act, verify, and operate under real constraints. If you build agents that touch production, or you want a mental model that survives your current framework, this is one of the more grounded books on the topic.

Skip it, or borrow it, if you only want a quick single-framework tutorial or a weekend of copy-paste demos. That is not what this book is trying to be.

If agent infrastructure is your world, it pairs well with where the platforms are heading. I wrote about that shift in my Databricks Data + AI Summit 2026 breakdown, where the story was almost entirely about operating and governing agents at scale, which is exactly the muscle this book trains.

08Final word, and a thank you

30 Agents Every AI Engineer Must Build is really a field guide to agent engineering wearing a catalog's cover. The "30 agents" are 30 reusable architectures you can combine, and the enduring value is the engineering judgment underneath them. It is not a perfect book, the breadth costs it some depth, but it is an unusually honest and useful one. 4.5 / 5, and it has earned a permanent spot on my shelf.

Thank you, Dipali Malwatkar, for reaching out and sending the book. I appreciate the trust, and I hope this review does it justice by going past the blurb.

And a real thank you to Imran Ahmad for writing it. It is clear every pattern was pressure-tested against the production realities of latency, cost, reliability, and security, and that care shows on the page.

You can find the book on Amazon and via Packt, and the code on GitHub. If you read it, I would genuinely like to hear which of the 30 patterns you reached for first.

This is the guide I wish I had when I was fixing my own resume and helping friends fix theirs. It moves in the order you should actually work: build a strong draft, find the gaps, tailor it to the role and company, then make it survive the machines. At the end there is a matching cover letter section, a final checklist, and the two tools I keep coming back to.

01The mindset: who actually reads your resume

Before you write a word, understand the audience. A hiring manager spends an average of about 6 to 7 seconds on the first scan, so your strongest, most relevant information has to be visible at the top without hunting. But that human is now the last step, not the first.

Here are the four readers your resume has to satisfy, in the order they see it.

The resumes that get interviews are built for all four at once. A beautiful Canva template that a parser turns into gibberish gets rejected before a human sees it. A plain keyword dump that passes the parser but bores the recruiter gets skipped at the next step. You need both.

A note on the numbers in this guide: figures like "roughly 99 percent of Fortune 500 companies use an ATS" are well established, but some widely quoted parse-rate and preference percentages come from resume-tool vendors testing their own products, not peer-reviewed studies. I have flagged those as illustrative. The direction is reliable even where the exact number is not.

02Step 1: Build the resume from scratch

Start with structure, not styling. Get the right sections in the right order, then fill them with evidence.

Section order depends on where you are

Order your sections by what is strongest and most relevant. For students and early-career applicants, education and projects usually earn a spot near the top. Once you have real work history, experience leads.

Header

Your name is the largest text on the page. Under it, put a single contact line with items separated by a vertical bar: location or relocation status, email, phone, LinkedIn, GitHub, and portfolio, each hyperlinked. Leave out your full mailing address, photo, date of birth, and marital status. Keep contact details in the body of the document, never in the page header or footer, because parsers often drop those.

Summary

Two to four sentences, or a tight two-bullet version. Lead with your target title, years of experience, and core specializations, then close with a standout credential. Skip generic adjectives like "hardworking" and "team player." A two-bullet structure works well:

• Overview: "Data analyst with 1.5 years across fintech and retail, focused on data wrangling, visualization, and stakeholder reporting."
• Proof: "Built a customer churn model in Python and Tableau that cut churn by 15 percent."

Experience: the accomplishment formula

This is the section that decides most interviews. Each entry is a header line (role, organization, dates tied to the title) followed by 4 to 6 achievement bullets. Write every bullet with a strong action verb and this formula: action verb + what you did + tool or method + quantified result.

Anyone can list duties. Strong bullets answer four questions: what did you do, why, how, and what was the impact. A simpler way to remember it is Problem + Action + Result: the situation you faced, what you did about it, and the outcome. Quantify about 80 percent of your bullets with a number, percent, timeframe, or scale, and bold the metrics so they survive the 6 second scan.

Use past tense for previous roles and present tense for your current one. Order bullets from most to least impressive. If you need help with verbs, group them by category (led, orchestrated, spearheaded for leadership; engineered, optimized, automated for technical work) and vary them so no two bullets start the same way.

Two writing rules that trip up most first drafts: drop the personal pronoun "I" (start with the verb: "Built," not "I built") and avoid passive voice ("Reduced latency," not "latency was reduced"). For technical roles, name the languages, frameworks, platforms, and tools in each experience entry, not just in your projects, so a reviewer can see your stack in context. One more filter worth applying to every line: could you confidently talk about it in an interview? If not, cut it or rewrite it until you can defend it.

Skills

Group skills into labeled categories and put one category per line, with the category name in bold and a comma-separated list after it. Order both the categories and the items by relevance to the target role. Skills-first, categorized sections parse more accurately and let a recruiter scan your stack in seconds. Keep soft skills out of here; prove those through your experience bullets instead.

• Languages: Python, SQL, R
• Cloud and data: AWS, Databricks, dbt, Airflow
• Visualization: Power BI, Tableau, Looker

Projects, education, and certifications

For students and career switchers, projects do heavy lifting. Give each a title, start and end dates, and a repo or demo link, then 1 to 2 bullets written like experience bullets: what you built, the stack, and a measurable result. Make it easy to find the code or a deployed version, and if you made a critical technical decision (why this database, why this architecture), say so briefly. That reasoning is exactly what an interviewer will ask about.

For education, use one line per degree, most recent first. Include your GPA (cumulative or major, whichever is stronger) only if it is above about 3.3; if both are below that, leave it off. Bootcamp and non-university technical coursework counts, so list programs like CodePath, and add relevant coursework only when your projects and experience are still thin. Include your high school only if you are a college freshman, or if you earned a standout honor like valedictorian. For certifications, list current, verifiable credentials with a "show credential" link, and prioritize proctored ones over self-paced course certificates.

03Step 2: Find the gaps (self-audit)

Before you send anything, run your draft through the same five checks the four readers apply. This is where most resumes quietly fail.

Common gaps to hunt for: duties instead of outcomes, bullets with no numbers, an objective statement where a summary should be, a skills block buried at the bottom, inconsistent date formats, and part-time or volunteer roles dropped entirely instead of reframed as accomplishments. Do not omit retail, food service, or gig work: reframe "provided customer service at a store" as "advised 30+ customers daily and trained 3 new hires on store procedures."

04Step 3: Enhance by role and company (tailoring)

A generic resume sent to every job is the lowest-return move in the search. Tailoring is not swapping a few words; it is translating your experience into the specific language a given team uses about its own work. Budget about 10 to 15 minutes per application once you have a strong master resume.

The three moves that matter:

1. Match vocabulary. If the posting says "stakeholder management" and your resume says "worked with cross-functional teams," the ranker does not know those are the same thing. Use their exact terms, not synonyms.
2. Reorder by emphasis. The first 3 to 4 responsibilities in the job description are what the team cares about most. Your top 3 to 4 bullets in your most recent role should map to those, in the same order.
3. Strengthen with specifics. Swap generic bullets for ones that show the tool, the scope, and the metric, written the way the company writes about its own work.

A fast, repeatable workflow: build a T-chart. In the left column, list the requirements from the posting. In the right column, write one specific example of how you have demonstrated each. Then pull those examples into your summary, skills, and top bullets, and use your 3 to 5 strongest for stories in the cover letter.

One rule that cannot bend: rewrite, do not fabricate. Never add a skill you do not have. It gets caught in the interview, and it gets caught in the reference check. Tailoring reorders and rephrases what is true; it does not invent.

05Step 4: ATS optimization (beat the bots without gaming them)

An ATS (Applicant Tracking System) first parses your document into fields, then ranks it against the job description. Roughly 99 percent of Fortune 500 companies use one, so this is the default case, not an edge case. The good news: the same choices that pass the parser also make your resume clearer for the human on the other side.

File type: submit what the application asks for. A standard .docx is the safest choice across enterprise systems and is the better bet for legacy parsers like Oracle Taleo. Text-based PDFs preserve your layout and parse cleanly on modern platforms like Greenhouse and Lever. The one hard rule: never submit an image-based or Canva-flattened PDF, because parse rates for those collapse (some vendor tests put them near 4 percent; treat the exact figure as illustrative, but the risk is real). Test yours by trying to highlight the text.

Semantic matching, not stuffing: modern rankers infer meaning, so keyword stuffing is penalized. Integrate each major keyword at most once or twice per bullet, woven naturally. Aim for a 70 to 80 percent thematic match with the posting rather than a copy-paste, and include adjacent skills (a data scientist listing "data visualization" alongside "stakeholder communication") to signal real-world application. Keep roughly 80 percent of your experience bullets carrying a concrete metric, because AI screeners weight verifiable results far above vague claims.

On AI and fairness: no major enterprise ATS rejects a resume for being AI-assisted, and clean, well-structured resumes tend to do better, not worse. Regulation is also moving toward human oversight: GDPR Article 22 gives EU candidates the right not to be subject to purely automated decisions with significant effects, and the EU AI Act treats recruitment AI as high-risk. The practical takeaway is simple: a cleanly formatted, semantically aligned, truthful resume is the most likely to reach human eyes.

06Step 5: The matching cover letter

The cover letter is your chance to connect the dots the resume only lists. Keep it to one page and three to four short paragraphs, reuse your resume header so the two look like a set, and address a specific person whenever you can. Structure it as Hook, Proof, Close.

• Hook: Open with a brief story, a standout result, or a genuine connection to the company's mission, then name the role. If someone referred you, put their name in the first sentence. Skip "I am writing to apply for."
• Proof: Map your 3 to 5 strongest, quantified achievements to the job description. Do not restate the whole resume; pick the wins that match what this team needs and show them as bullets.
• Close: Restate your fit in a sentence, give your email and phone for easy follow-up, and thank the reader sincerely. End on a confident, human note.

Match the cover letter's font and header to your resume, export it as a selectable-text PDF named clearly (for example FirstName_LastName_CoverLetter.pdf), and write a fresh one for every application. A tailored letter that mirrors the posting's priorities is what separates a memorable application from a form submission.

07Iterate: the final pre-submit checklist

Treat every send as a small experiment. Tailor, submit, and note what earns replies, then adjust.

• One page for students and most candidates, two only with 10+ years of relevant experience.
• Consistent formatting, no typos, every link working and pointing to the right place.
• Every bullet starts with a varied action verb (no "I", no passive voice) and follows Problem, Action, Result with a number.
• Every line is something you could confidently explain in an interview.
• 3 to 5 job-description keywords mirrored naturally, tested for a 70 to 80 percent match.
• Standard headings, round bullets, dates tied to titles, no images or tables.
• Titles and dates consistent with your LinkedIn and portfolio.
• Saved in the requested format and named LastnameFirstname.
• A tailored, one-page cover letter addressed to a real person.

08Resources

These are the two I recommend starting with, plus a few more that speed up the work.

A few more tools worth keeping in the kit:

• Jobscan for testing your keyword and match score against a specific job description before you submit.
• Simplify Copilot (browser extension) for autofilling applications and surfacing keywords from postings.
• RenderCV and the FAANGPath simple template if you want alternative ATS-friendly LaTeX layouts.

Bonus: mine your own LinkedIn first

Remember the fourth reader, the verifier, and the checklist item about keeping your resume consistent with your LinkedIn. The fastest way to close that gap is to actually look at your own LinkedIn data, and most people never do. Your profile already holds years of roles, skills, and connections you can turn into quantified bullets and a tighter skills section, and reviewing it side by side with your resume is how you catch the title and date mismatches before a recruiter does.

That is the itch I built this next tool to scratch.

Request your data export from LinkedIn (Settings, then Data Privacy, then Get a copy of your data), drop the ZIP into the app, and use what surfaces to fill the gaps from the self-audit. The code is open source on GitHub if you want to see how it works or run it locally.

09The one thing to remember

Your resume will not get you the job. It will get you the interview. Every choice above serves that single goal: be readable to the machines, be clear and specific to the humans, and be honest at every layer. Build the strong draft, find the gaps, tailor it to the role, pass the ATS, then send it and iterate.

New here? You might also like why I built this blog.

This post is a structured, source-backed walkthrough for data engineers, analytics engineers, AI engineers, and enterprise architects. Where helpful, I've embedded the official Databricks keynote and session videos so you can go straight to the source.

Note on sourcing: This recap prioritizes official Databricks announcements and reputable coverage. Confirmed product announcements are stated plainly; analyst interpretation and keynote-only details are flagged as such. Full links are in the source table at the end.

Quick answer: Databricks Data + AI Summit 2026 (DAIS 2026) was Databricks' annual data and AI conference, held June 15 to 18, 2026 at the Moscone Center in San Francisco with more than 30,000 in-person attendees. Its headline launches were Genie One, Unity AI Gateway, Lakeflow, LTAP with Lakebase, and the Lakewatch security lakehouse.

01TL;DR: the five things that matter

• Event: DAIS 2026 ran June 15–18 at Moscone Center, San Francisco, with 30,000+ in-person attendees, tens of thousands virtual across 150+ countries, and 800+ breakout sessions.
• Genie One: Genie evolves from AI/BI chat into a governed agentic coworker (web, iOS, Android, Slack, Teams, MCP), backed by Genie Ontology, Genie Agents, App Builder, and ZeroOps.
• Unity AI Gateway + expanded Unity Catalog: runtime governance for models, agents, and tools: contextual policies, hard spend caps, unified tracing, semantics (Glossary, Domains, Metrics), and cross-cloud/region governance.
• Lakeflow: unified, agentic data engineering: Lakeflow Designer (GA), 100+ Connect connectors, Zerobus ingest, Real-Time Mode for Spark Declarative Pipelines, and Lakeflow Jobs as a platform orchestrator.
• LTAP + Lakebase + Lakehouse//RT: Lake Transactional/Analytical Processing unifies OLTP and OLAP on a single storage copy, with serverless Postgres (Lakebase) and a millisecond-latency engine (Lakehouse//RT).
• Lakewatch + Panther: an "agentic SIEM" security lakehouse, plus the intended acquisition of Panther to bring AI SOC workflows into the platform.

If you internalize only three things for 2026–2027:

1. LTAP + Lakebase aims to collapse the OLTP/OLAP split and remove CDC/ETL between operational and analytical systems.
2. Genie + Unity AI Gateway + Unity Catalog define the runtime control plane for agents, models, and AI/BI, including spend and policy enforcement.
3. Lakeflow + Lakewatch signal that data engineering and security operations will increasingly be run by agents on the lakehouse.

02Event overview

• Dates & location: June 15–18, 2026, at Moscone North/West/South, San Francisco, with hybrid virtual attendance.
• Scale: 30,000+ in-person and tens of thousands virtual participants from 150+ countries, across 800+ breakout sessions spanning data engineering, governance, AI, applications, and agents.
• Audience: Data engineers, ML/AI engineers, data scientists, BI developers, architects, app developers, and executives.
• Tone vs. prior years:
  • 2024–2025: "How do we build agents and RAG?"
  • 2026: "How do we operate agents and AI apps reliably, govern them, control cost, and run on unified data?"

Analysts consistently noted the shift from proof-of-concept demos to operating models, agents, and security at production scale.

Watch: the Day 1 keynote

The full Day 1 keynote is the best single source for the announcement sequencing, Unity AI Gateway, Genie Ontology/One/Agents, Lakewatch, CustomerLake, Reyden/Lakehouse//RT, LTAP, and Lakeflow.

▶ Data + AI Summit Keynote 2026, Day 1 Databricks · YouTube

Watch: the Day 2 keynote

The Day 2 keynote dives deeper into the developer and research side: open source, AI engineering, and the technical underpinnings of the agentic platform.

▶ Data + AI Summit Keynote 2026, Day 2 Databricks · YouTube

03AI and the agentic platform

Genie One, Ontology, Agents, App Builder, ZeroOps

• Genie One evolves Genie from AI/BI chat into a data-smart agentic coworker for business users, available on web, iOS, Android, Slack, Teams, and as an MCP app for other assistants. It connects via Lakehouse federation, Lakeflow Connect, and 50+ workplace apps (Google Drive, Jira, Slack, Confluence, SharePoint, email), and can schedule tasks, send alerts, create docs, and call tools.
• Genie Ontology is an automatically learned enterprise context graph that extracts meaning from tables, queries, dashboards, pipelines, and apps, ranks authoritative definitions (PageRank-style), and enforces ACLs via Unity Catalog.
• Genie Agents turn curated "Spaces" into reusable agents that reason over structured and unstructured data and take autonomous actions via tools, MCP connections, and write-backs, all governed by Unity AI Gateway/Unity Catalog.
• Genie App Builder (private preview) is a vibe-coding environment: upload context, and Genie produces a build plan and a live app against governed data, enforcing Unity Catalog permissions.
• Genie ZeroOps is a background agent that watches pipelines, tables, jobs, and models, performs root-cause analysis using logs, metrics, and lineage, proposes fixes, tests them in a sandbox, and lets humans approve.

Pricing/packaging: Genie moves to usage-based, no-seat pricing, with up to $10 of free usage per user per month; organizations pay only for AI consumed (token-based).

▶ Introducing Genie One: the AI coworker that understands your data (with demo) Databricks · YouTube

Unity AI Gateway and agent governance

Unity AI Gateway is the runtime governance and telemetry layer for models, agents, tools, and MCP services:

• One place to govern all AI assets, register Databricks-hosted and external models, agents, skills, and MCP services alongside data, with the same access controls, discovery, lineage, and auditing as Unity Catalog tables.
• Contextual Service Policies (Beta), runtime policies that allow/deny/require approval for actions like pushing code to GitHub, writing to certain paths, blocking PII in responses, and guarding against prompt injection.
• AI budgets + hard caps, central budgets now cover external BYO-key providers; hard caps stop requests when a budget is hit, not just alert.
• Unified tracing, model and MCP traces feed into Lakewatch for investigation, giving full auditability of agent tool calls.

Analysts argue that Unity AI Gateway + Unity Catalog together make the catalog "the control plane for enterprise AI, not just data."

Agent Bricks and Mosaic AI

• Agent Bricks is Databricks' agent platform, integrated with Lakewatch and hands-on training; official materials emphasize Vector Search, AI Playground, and Foundation Model APIs to build RAG agents with MLflow Tracing–based observability.
• Lakewatch, Genie, and Lakeflow all rely on Mosaic AI components (models, vector search, evaluation, tracing) under the hood, but most Mosaic-specific updates were folded into these products rather than launched under separate branding. (Inference based on keynote/product positioning; no standalone "What's new in Mosaic AI" Summit blog was found in the sources reviewed.)

▶ How Agent Bricks gives developers choice, context and control (with demo) Databricks · YouTube

For ML engineers specifically, Genie Code gained upgraded intelligence for ML workflows, feature engineering, training, and evaluation, with native integrations across the Databricks ML platform.

▶ Agentic machine learning with Genie Code (includes demo) Databricks · YouTube

04Data engineering, Delta, streaming, and LTAP

Lakeflow: unified, agentic data engineering

The Lakeflow update is the most material announcement for data engineering teams.

▶ Unlocking agentic data engineering with Lakeflow + Genie Databricks · YouTube

Key pillars:

• Lakeflow Designer (GA): a visual, AI-powered no-code canvas for Spark Declarative Pipelines; flows compile down to real SDP code, so engineers can refine without translation gaps.
• Genie Code + Lakeflow: Genie Code is embedded to generate connectors, pipelines (Python/SQL), and job DAGs with triggers and dependencies, using full end-to-end context.
• Lakeflow Connect: 100+ native managed connectors to apps, databases, and file stores; query-based CDC for all DB and federation sources, plus a "Community Connectors" OSS framework. Free tier: 100 DBUs/day, supporting up to 100M records/day.
• Zerobus Ingest (GA): Kafka-free ingestion for high-volume events: near-real-time writes (<5s), up to 100 MB/s per stream and >10 GB/s per table, with a Kafka-compatible API (Beta), gRPC + REST (GA), SDKs for Python/Java/Rust/Go/TypeScript, and OpenTelemetry (public preview).
• Spark Declarative Pipelines Real-Time Mode (Public Preview): ~5 ms end-to-end latency on a Spark-only stack (no separate Flink cluster), on classic and serverless compute.
• Lakeflow Jobs: a native orchestrator with data-aware triggers (jobs fire when SQL-expressed "ready" conditions are met), external orchestration (operators for Snowflake, REST APIs, Slack, PagerDuty, etc., with 40+ examples), and Unity Catalog–governed credentials and audit.

Customer proof points: Panasonic reported a 50% reduction in Power BI refresh times after consolidating legacy ETL onto Databricks via Lakeflow; Meta cut end-to-end pipeline latency to under a minute with Zerobus + SDP Real-Time Mode.

LTAP, Lakebase, and Lakehouse//RT

• LTAP (Lake Transactional/Analytical Processing) unifies transactions, analytics, streaming, and operational data on a single copy of data in the lake, built on Lakebase and the Lakehouse. It writes OLTP data directly into Delta/Iceberg in Unity Catalog with Postgres semantics and full ACID, eliminating the CDC/ETL duplication between OLTP and OLAP, with strict isolation between transactional and analytical workloads.
• Lakebase (serverless Postgres on object storage) now serves thousands of customers and reportedly 12M database launches/day, adding cross-cloud/region disaster recovery, git-style branching and snapshots, and autonomous DB operations (agents monitor health, propose indexes, assist recovery).
• Lakehouse//RT + Reyden: a real-time warehouse engine delivering millisecond-scale query latency and tens of thousands of concurrent users directly on Delta/Iceberg, meant to replace separate serving layers and caches. SiliconANGLE reports customers seeing up to 16x better performance vs. existing real-time serving stacks, positioned as a drop-in upgrade for Lakehouse SQL warehouses (beta).

Analysts characterized LTAP + Lakehouse//RT as Databricks "declaring the end of pipelines" between applications and analytics, particularly for agent-heavy workloads.

▶ Introducing LTAP: a new data processing architecture Databricks · YouTube

▶ Introducing Lakehouse//RT and Reyden, Reynold Xin, Co-founder & Chief Architect Databricks · YouTube

05BI and analytics (including Power BI / Fabric relevance)

Genie for AI/BI and business workflows

Genie One extends beyond conversational analytics into doc/report generation, dashboards, scheduled briefs, and workflow automation grounded in governed data. It's embedded in Slack/Teams, has native mobile apps, and can be called as an MCP tool by external assistants, turning Databricks into a shared AI/BI backend.

Customers: Uplight uses Genie One to "explore, discover and innovate" faster; Foot Locker uses Genie Agents as a central AI insights space for executives across banners; Albertsons uses Genie within a Merchandising Intelligence initiative for explainable decisions across the 4Ps.

Unity Catalog Metrics and semantics for BI

• Metrics enhancements: multi-fact relationships (Public Preview in Dashboards), LOD calculations, parameterized metrics, richer window functions, and metric materialization (public preview) for faster dashboards and agent queries.
• Agentic/UI-driven authoring: a visual designer plus agents that propose metric definitions, lowering the barrier to semantic modeling.
• Import from third-party tools (Beta): import models from Power BI and Tableau directly into Databricks, bootstrapping Metrics from existing BI semantics.
• Domains and Glossary: Domains (Public Preview) and an upcoming Glossary organize assets into business categories and feed Genie Ontology and agents.
• External lineage (GA): end-to-end lineage now spans upstream sources and downstream BI tools.

This is a critical bridge for Power BI/Fabric customers: you can keep existing semantic models while migrating workloads or running BI across platforms.

Real-time BI

For BI teams, Lakehouse//RT + LTAP mean direct millisecond queries from BI tools against Delta/Iceberg without a separate serving database, and "fresh" analytics that sit directly on operational data (no ETL lag).

06Governance and security

Unity Catalog: control, context, choice

Unity Catalog's roadmap is framed around control, context, and choice in the agentic era:

• Control: Unity AI Gateway; Governance Hub (Private Preview) for a centralized posture dashboard; ABAC expansions (grant policies for models, identity/context attributes, tag propagation); and opt-in RBAC (Public Preview soon).
• Context: Glossary, Domains, and Metrics build an open, OSI-ready semantic layer accessible via SQL, APIs, MCPs, and agents; External Lineage (GA) and Table Insights feed Genie Ontology.
• Choice: cross-cloud/region governance with a four-level namespace (metastore.catalog.schema.table); external engine writes to UC-managed Delta tables (Spark, Flink, DuckDB) via catalog commits; FILE type (Beta) for governing unstructured assets (PDFs, images, audio, video); and geospatial types (GA) in Delta and Iceberg v3.
• OpenSharing: announced as a Linux Foundation project, extending Delta Sharing to models, skills, and unstructured data.

Platform security and compliance

• Automatic Identity Management (AIM): Entra ID (GA on AWS/GCP) and Okta (Public Preview) auto-provision users, groups, and service principals.
• Context-Based Ingress (Public Preview): zero-trust-style policies combining network, identity, and scope, plus expanded Inbound Private Link.
• Private Network Gateway (Private Preview on Azure) and Private Link for Lakebase (GA on AWS, Public Preview on Azure).
• Compliance expansion: HITRUST across AWS/Azure/GCP, ISMAP (Japan), extended KSA on GCP, expanded AWS GovCloud coverage for AI features, and forthcoming FedRAMP High on Azure Commercial.

Lakewatch and the Panther acquisition

• Lakewatch (agentic SIEM), formally launched earlier in 2026, was a major Summit theme. It unifies security, IT, and business data in one governed lakehouse with petabyte-scale retention and a claimed up to 80% lower TCO vs. legacy SIEM, embedding security agents (via Agent Bricks + Genie Code) for automated triage, rule authoring, and investigation.
• Panther acquisition: Databricks announced intent to acquire Panther, an AI SOC platform with detection-as-code and 100+ security integrations, to accelerate the Security Lakehouse vision. Analysts see this as Databricks moving from a data platform into core security infrastructure.

07Partner ecosystem

• Microsoft / Azure Databricks: emphasis on modernizing Azure data estates and zero-copy federated analytics, combining Azure's enterprise cloud with Databricks' Lakehouse/Lakebase/Genie rather than redundant warehouses.
• AWS: returned as a "Legend Sponsor," highlighting co-built customer stories (e.g., Mastercard) and Bedrock + Databricks agent patterns (Bedrock AgentCore agents querying Unity Catalog, answering via Genie, reading low-latency state from Lakebase/Lakehouse//RT).
• GCP and others: expanded KSA compliance on GCP and a broad open ecosystem around Lakewatch (Okta, Zscaler, Wiz, Panther). OpenSharing and open table formats keep interop viable with third-party engines including Snowflake and Fabric-connected BI tools.

Watch: CustomerLake, the agentic CDP

Among the vertical announcements, Databricks introduced CustomerLake, an agentic Customer Data Platform natively embedded in Databricks.

▶ Introducing CustomerLake: The Agentic CDP, Ali Ghodsi Databricks · YouTube

08Top announcements timeline

Dates use press/blog publication dates; many were also demoed live in the June 15 Day 1 keynote.

09Five trends from the summit

1. From building agents to operating agentic systems. Unity AI Gateway, Genie Ontology, ZeroOps, Lakewatch, and Lakeflow together address runtime concerns, cost, guardrails, observability, and auto-remediation.
2. Collapse of OLTP/OLAP and serving layers into LTAP. One storage copy, multiple engines, open formats, and strict isolation, going beyond "HTAP in a single engine."
3. Governance and semantics as AI infrastructure. Unity Catalog moved from access control to runtime AI governance and semantic context; the catalog is now a decision-maker for AI, not just a registry.
4. Data engineering becoming agentic and unified. Lakeflow consolidates ingestion, streaming, and orchestration, with Genie Code and ZeroOps automating authoring and operations, fewer standalone tools (Airflow, Kafka, proprietary CDC).
5. Security converging with data and AI. Lakewatch + Panther push SIEM workloads onto the lakehouse, with security agents running alongside data and AI workloads.

10Competitive analysis

• vs. Snowflake: LTAP's storage-centric unification (Postgres apps + open Delta/Iceberg) is explicitly contrasted with HTAP and "Zero-ETL" architectures that keep separate engines and hidden CDC. External engine writes, Iceberg v3 GA, and OpenSharing reinforce an open multi-engine story; Lakewatch + Panther position Databricks directly against SIEM players.
• vs. Microsoft Fabric: Azure Databricks content emphasizes modernizing Azure estates with zero-copy federated analytics rather than wholesale replacement. Metrics import from Power BI/Tableau is a clear bridge, and Genie One overlaps conceptually with Copilot for Power BI but stresses governed SQL, multi-app context, and usage-based economics over per-seat licensing.
• vs. point AI/MLOps tools: Genie, App Builder, and Agent Bricks compete with standalone agent frameworks by being tightly integrated into governance, security, and semantics; MLflow Tracing + Mosaic AI remain the evaluation backbone but are framed as parts of end-to-end agent platforms.

11Enterprise implications

Three announcements with the biggest 2026–2027 impact

1. LTAP + Lakebase: if executed, unifying OLTP/OLAP/streaming on one governed storage copy can drastically reduce CDC/ETL complexity and time-to-production for AI apps. A multi-year journey for most, but a clear directional bet away from dual-stack architectures.
2. Genie One + Ontology + Agents: Genie Ontology addresses the "context problem" by making governed data (not embeddings) the ground truth for reasoning, with large potential productivity gains if enterprises standardize on Genie for data-informed work.
3. Unity AI Gateway + UC semantics: contextual policies, budgets with hard caps, and unified tracing respond directly to CFO/CISO concerns about uncontrolled LLM spend and agent risk.

Three announcements most relevant to Power BI / Fabric users

1. Unity Catalog Metrics + Power BI/Tableau import: centralize metrics and reuse them across Databricks SQL, Genie, and external BI while preserving existing definitions.
2. Lakeflow + Power BI performance: Connect, Zerobus, and SDP Real-Time Mode offer a Fabric-like ingestion/transformation experience on the Lakehouse (Panasonic's 50% refresh-time improvement is the showcase).
3. Lakehouse//RT + zero-copy analytics with Azure: low-latency queries on lakehouse data without replication to a separate serving store, aligning with the zero-copy federated analytics patterns Microsoft and Databricks jointly advocate.

Practical takeaways by role

Data engineers: Pilot Lakeflow Designer + SDP on non-critical pipelines; evaluate Zerobus as a Kafka replacement for high-volume producers; start capturing lineage via Lakeflow Connect + UC External Lineage to prepare for ZeroOps and data-aware orchestration. Medium term: rationalize Kafka/Airflow into Lakeflow Jobs and pressure-test Lakebase for greenfield transactional workloads.

Analytics engineers / BI developers: Treat Unity Catalog Metrics and Domains as your new semantic modeling surface; import from Power BI/Tableau where it accelerates migration; explore Genie One/Agents for curated domains, starting with users who already live in Slack/Teams; test Lakehouse//RT for high-concurrency dashboards.

AI/ML engineers: Standardize on Unity AI Gateway for model and tool access so budget and policy enforcement are centralized from day one; use Agent Bricks + Vector Search + MLflow Tracing for RAG observability; integrate Genie Ontology and UC semantics into prompting and tool choices.

Architects / technical leaders: Map roadmaps to LTAP (where OLTP+OLAP unification on open storage is feasible over 2–3 years); build a governance strategy with Unity Catalog at the center (data, AI, security, cost); track Lakewatch + Panther integration timelines if you already centralize security data in Databricks.

12Conclusions and predictions

• Databricks is explicitly betting the lakehouse will be both the data layer and the runtime control plane for agents, apps, and security. LTAP, Unity AI Gateway, and Lakewatch all reinforce that strategy.
• For 2026–2027, the likely near-term wins are in data engineering and AI/BI, not wholesale LTAP adoption. Lakeflow consolidation, Genie One deployment, and UC semantics can deliver value without touching core transactional systems.
• LTAP could be transformative but will land unevenly: cloud-native apps and new AI products are the most natural candidates; SaaS-heavy or vendor-locked workloads will migrate slower.
• Governance will increasingly be the differentiator across platforms, pushing competitors to match runtime policy, spend caps, and semantic openness.
• Security workloads will move closer to the data platform: if Lakewatch + Panther deliver, expect more SOCs to treat the lakehouse as their primary telemetry and detection surface.

For teams already deep in Databricks, Power BI, and cloud data engineering, the immediate opportunity is to treat Databricks less as a warehouse and more as an agentic operating environment, incrementally adopting Lakeflow, Genie, and Unity Catalog semantics while planning if and where LTAP and Lakewatch fit your roadmap.

13What I actually changed after the Summit

A recap is only worth writing if it changes what you do on Monday. Here is what I actually revisited in my own work after DAIS 2026, separated from the announcements above so you can tell reporting from practice.

• [TODO: confirm] Genie / semantic layer. [TODO: describe the specific change you made to how you expose metrics or semantic models after seeing Genie Ontology and the Power BI/Tableau metrics import, e.g. which models you registered in Unity Catalog and what it replaced.]
• [TODO: confirm] Lakeflow adoption. [TODO: state whether you moved any ingestion/orchestration onto Lakeflow, which connector or pipeline, and what it replaced (Airflow job, custom CDC, etc.).]
• [TODO: confirm] Unity Catalog governance. [TODO: note any concrete governance change: metric views, glossary/domains, spend caps via Unity AI Gateway, and the before/after.]
• [TODO: confirm] What I deliberately did not adopt yet. [TODO: name the announcement you decided to wait on (e.g. LTAP/Lakebase, Lakewatch) and the specific reason it is not on your near-term roadmap.]

[TODO: confirm] The one measurable outcome. [TODO: add a single number or before/after result from a change above, e.g. pipeline runtime, cost, or lines of glue code removed. If none yet, say what you will measure and when.]

14Sources (official and reputable coverage)

Prioritize official Databricks sources; confirmed announcements are distinguished from analyst interpretation throughout this post.

01The stack: Next.js, Markdown, and GitHub Pages

I wanted a fast, good-looking blog that costs nothing to host and is trivial to update. A static site on free hosting is the obvious answer. Here is the stack:

• Next.js (App Router) exported to static HTML with output: 'export'.
• CSS Modules with an OKLCH design-token system for theming.
• Markdown files compiled at build time with remark.
• GitHub Pages for hosting, with GitHub Actions for automatic deploys.

02Why use Next.js static export for a blog

GitHub Pages only serves static files, with no servers or databases. Next.js handles this with a single config flag:

// next.config.mjs
const nextConfig = {
  output: 'export',
  basePath: '/blogs',
  images: { unoptimized: true },
};

Because the site lives at a sub-path (/blogs), I set a basePath so every asset and link URL resolves correctly.

03How to write posts in Markdown

Each post is a Markdown file with a small frontmatter block:

---
title: My Post
date: 2026-06-20
excerpt: A short summary.
tags: [example]
author: Brian Castelino
authorUrl: https://www.linkedin.com/in/cas7elino/
---

Your content here.

author and authorUrl are optional. If you omit them, every post defaults to my name linked to my LinkedIn, so the byline stays consistent across the site.

The build reads every file in content/posts, generates a static page for each, and lists them on the home page sorted by date.

04How to deploy a Next.js blog to GitHub Pages

Deployment runs automatically through GitHub Actions on every push to main:

1. Push your new post to the main branch.
2. A GitHub Actions workflow runs next build, which exports the site to the out/ folder.
3. The workflow publishes out/ to GitHub Pages.
4. Wait about a minute, and the post is live.

That is the whole system: simple, fast, and free.

05Related reading

New here? Start with why I built The Brian Journal, or read my first deep dive on Databricks Data + AI Summit 2026.