• Join Reddit Community

• Follow us On X

• Follow us on Instagram

• Follow us on TikTok

🛠️Data Engineering

Ovveride the replace where for a single refresh

You can override a standalone streaming table’s defined REPLACE WHERE predicate for a single refresh by adding a WHERE clause to the REFRESH STREAMING TABLE command. The override applies only to that refresh and doesn’t change the table definition, which is useful for one-time backfills and corrections.

FYI: This is supported only on standalone streaming tables created with a FLOW REPLACE WHERE clause. 📖Documentation

Managed Iceberg Materialized Views are available

Creating iceberg-compatible materialized views bridges the gap between high performance query acceleration and open data architectures 📖Documentation

New Lakeflow Connectors are available

• Veeva Vault

• Mysql integrated CDC Pipeline

• Strac

• Anthropic

• Reddit Ads

• OpenAI

Native Excel support

You can read any sheet from a multi-sheet workbook, target specific cell ranges, automatically infer schema and data types, and work with formula values as their computed results . 📖Documentation

Granular data auditing

The _object_metadata column is a hidden metadata column that exposes cloud object-level properties for each file read by a file-based data source

Unit testing for SDP

You can write and run Python unit tests for your Spark Declarative Pipelines on Lakeflow transformation logic directly in the Lakeflow Pipelines Editor. Validate Python and SQL transformations against mock data using catalog-table redirection, including Auto CDC, streaming tables, expectations, and append flows. 📖Documentation

What’s new in Automatic Upgrades ?

Automatic upgrades begin rolling out row tracking and Checkpoint V2 to existing Unity Catalog managed tables. You can track which features are available for automatic upgrades. 📖Documentation

⚡Data Warehousing

Lakehouse RT

Lakehouse RT is a serverless SQL warehouse type built for low-latency, high-concurrency workloads. By serving sub-second SQL read queries directly against Unity Catalog tables you no longer need to build and maintain reverse-ETL pipelines to copy data into secondary databases. 📖 Documentation

Excel Add-in enhancements

You can use the Add-in to write data from MSFT Excel to a Unity Catalog table by creating a new table or overwriting an existing one without leaving excel.

Agentic code converter

You can use the agentic code converter powered by Genie Code, to convert legacy SQL dialects such as T-SQL, Snowflake, Redshift, Oracle, BigQuery, and Teradata to ANSI SQL in migration projects in your workspace. The converter analyzes each source script, validates syntax and semantic intent and iteratively fixes errors before writing the results to your workspace. 📖 Documentation

Protection against runaway queries

You can set a statement timeout on a single SQL warehouse so that different warehouses can apply different timeouts to their queries. You can set the timeout with the statement_timeout field in the Create or Update warehouse APIs. 📖 Documentation.

🖥️ Platform

Standard performance mode is supported for MVs and STs refresh

Pipelines use the performance-optimized mode when run interactively in the UI, the job's Performance Optimized setting when scheduled with a SQL task and the standard mode when scheduled. You can set the compute mode for pipelines scheduled by a SCHEDULE clause in the definition, edit the schedule in the Catalog Explorer. 📖 Documentation.

Private Link updates

You can use Private Link to privately connect to Databricks account-level resources and to SAP Business Data Cloud (BDC) 📖 Documentation,Documentation

Allow partner platform IPs in context-based ingress control

Context-based ingress control supports Partner Platforms as a network source. You can select a partner platform to allowlist the IPs that third-party apps use to connect to Databricks. The IP lists are managed automatically.

Custom URL for Databricks Account

Git CLI Commands in Git Folders are available

Git folders that you create from the UI automatically get Git CLI access when your workspace is eligible, so you can run standard Git commands such as git stash, interactive rebase, pre-commit hooks, Git submodules, and Large File Storage (LFS) directly from a Databricks terminal.

🤖GenAI & ML

• The web terminal in notebooks connected to serverless GPU compute (AI Runtime) available. You can use it to run shell commands, monitor GPU usage with nvidia-smi, install libraries, and manage files directly on the compute. This requires serverless GPU environment version 5 or above. 📖 Documentation

• For custom model serving endpoints with a tensor-based input signature, you can send the request body as a serialized KServe v2 ModelInferRequest with Content-Type: application/x-protobuf, instead of JSON.

Lakebase

• Lakebase Autoscaling update notification now post 3 days in advanced.

• Lakebase is SOC 2 Type 2 compliant.

• Lakebase adds AI-assisted troubleshooting: database issues and early warnings are surfaced to you with Insights and you can then investigate and fix them conversationally with Genie.

📝Genie Agents

• Databricks Genie App for Teams: 📖Documentation

• Databricks Genie App for Slack: 📖Documentation

• Prompts and responses initiated by Genie Chat are visible in Genie Agents monitoring when the sharing is enabled.

• You can upload local Excel and CSV files to an Agent Mode chat for ad-hoc analysis. 📖 Documentation

• You can retrieve visualization results from Genie Agents using the API. 📖Documentation

• You can save a Genie One thread as a Genie Agent so you can open it in your workspace to fine-tune its context, instructions and data assets.

• You can share individual Genie conversations to collaborate with teammates or allow agent managers to review your results.

  

• Genie Agent can analyze files stored in Unity Catalog volumes such as PDFs, slide decks, and images. 📖Documentation

• Agent mode APIs for Genie Agents are available. 📖Documentation

• You can download visualization results through the Genie Conversation API on Private Link workspaces in supported regions.

📊Genie Code

🚀 Report Import & Migration Tools

• Direct Import: Convert Tableau and Power BI files into new AI/BI dashboards straight from the dashboard list.

• Automatic Relationship Mapping: Preserves source dashboard relationships detected during import. 📖 Documentation

• Unified Metrics: Prioritizes existing Unity Catalog metric views during conversion to reduce redundancy.

🛠️ Dashboard & Filter Customization

• Date Range Tailoring: Request Genie Code to customize quick-select options (e.g., constraining options to day-based ranges).

• Flexible Multi-Select Filters: Set default parameter filter values to empty (“All”) instead of forcing a single default selection.

🛡️ Quality & DevXp

• Automated Semantic Auditing: Every newly created widget is automatically validated across core business logic (grain, units, scale, category values, and populations) for maximum precision.

• Undo/Redo Support: Author confidently with complete history controls.

• Warehouse Picker: Easily select your compute via the warehouse picker in full-page Genie Code. 📖Documentation

📲Genie One

AI Capabilities & Memory

• Genie Code Mentions: Type @Genie Code in a Notebook comment to get help directly within the comment thread.

• Copilot Integration: Connect Genie directly to Microsoft Copilot Cowork. 📖Documentation

• Memory (Beta): Genie One can save specified details (like preferences and workflows) to apply in future conversations. 📖Documentation

• Recall & Reuse Context: Search past conversations and bring that context into your current chat. 📖Documentation

Document Features

• Document Links: Documents supports embedded hyperlinks.📖Documentation

• Edit Timestamps: Displays the exact date and time documents were last edited. 📖Documentation

• Document Citations: Documents display citations linking content directly back to source data.

• Version History: Review earlier versions of documents through built-in history tracking.

• Soft Tabs: Open document links in soft tabs without navigating away from the current document.

Integrations & MCP Apps

• Genie One MCP App (Beta): Adds UI interactions from third-party agents, including interactive visualizations and Genie Ontology citations. 📖Documentation

• GitHub MCP (Beta): GitHub MCP connection for Genie One is available📖 Documentation.

Scheduled Tasks

• Shared Task Overview: Access a dedicated modal and listing page to view scheduled tasks shared with you by others.

• Task Mentions: @mention scheduled tasks directly inside Genie One.

• On-Demand Execution: Select Run now while authoring a scheduled task to run it immediately.

Platform, Data & UI Enhancements

• Ontology Snippets: Support added for ontology snippets (contact your Databricks account team to enroll). 📖Documentation

• Partially Supported Datasets: Ask Genie now queries valid datasets rather than failing completely when a dashboard contains unsupported SQL expressions.

• Front-end Private Link: Configure account-level front-end Private Links by turning on the Custom URLs and Account preview.

• Certified Visualizations: SQL charts and visualizations inserted by Genie One are now read-only certified visuals.

• Inline Conversation Renaming: Edit conversation titles inline by clicking the title in the chat header.

• Mobile Workspace Sign-in: Enter your workspace URL manually in the Genie mobile app if automatic discovery fails. 📖Documentation

• Expanded PDF Limit: The maximum character limit per uploaded PDF increased from 4,000 to 15,000 characters. 📖Documentation

📝Metric views

• Median and percentile expressions in metric views: Low-code metric view building supports two new expression types: median and percentile.📖 Documentation

• Local metric views support cluster-by and partition-by configuration in the UI and YAML and provide a low code option for adding filters to measures. 📖 Documentation

• Dashboard widgets show the display name defined for each field in a local metric view instead of the literal field name. 📖Documentation

📊AIBI Dashboard

• Dashboard variable display names: When a display name is available, dashboard variable drop-down menu options show the column’s display name instead of the field name. 📖 Documentation

• Dynamic annotation lines: Annotation lines can be driven by data or a calculation instead of a fixed value, letting you show running averages, variable thresholds, or anomaly markers that update with your data. 📖Documentation

• Dashboard relationships: Build complex multi-fact data models scoped to an AI/BI dashboard.📖Documentation

• Custom Vega visualization improvements: 📖Documentation

  • Cross-filtering works for Vega-Lite charts. The selection appears in the active filter bar and is preserved in dashboard URLs and share links.

  • Custom Vega-Lite charts can inherit the dashboard’s light or dark mode and custom theme, and authors can reference theme values directly in Vega-Lite expressions.

• Counter widgets support text styling on the comparison date and conditional coloring on the comparison field.

• Table columns adjust their width automatically based on cell content.

• Choropleth maps support English region names from the name_en column as the identifier for admin-0 (country) boundaries. 📖Documentation

• You can select Run now on a dashboard schedule to run it immediately. 📖 Documentation

• Cross-filtering on a faceted chart includes the facet field as part of the filter. 📖 Documentation

• Applied filters in email attachments: Dashboard authors can include the applied filters as a separate tabular attachment in email subscriptions. 📖 Documentation

• Multi-select filter improvements

  • Multi-select filter stays open: Multi-select filter drop-downs stay open between selections and keep your place in the list instead of closing.

  • Select all matching values in a multi-select filter: In a multi-select filter, you can type to narrow the list and select Select all results to select every matching value at once. Selections are additive across searches.

• Image widget: You can add a dedicated image widget to a dashboard.

🛡️Governance and Sharing

Restrict access to AI Functions with Unity Catalog

You can use Unity Catalog permissions to restrict which task-specific AI Functions your organization can access, independent of foundation model access. 📖 Documentation

Secrets in Unity Catalog

You can store, govern and access secrets as securable objects in Unity Catalog. 📖 Documentation

Centralized Governance in Databricks

Governance Hub is a centralized account-level UI in the account console for monitoring governance across your Databricks account. It provides a unified view of data health and coverage, AI usage, spend, and cost drivers,with prioritized insights and recommendations. 📖 Documentation

Role-based access control is Unity Catalog

RBAC lets users assume a role in Databricks using only that role’s permissions for the duration of the session instead of their own accumulated permissions. This enables exclusive access. Users must actively assume a role to reach sensitive data, which prevents them from accessing it as their own identity and from mixing data across use cases.📖Documentation

Governed Tags in System tables

The system.tags.governed_tags system table is in Beta. This table contains a record for each governed tag key in your account, including customer-created and system-generated tags. Use it to programmatically determine which tags in your account are governed. 📖Documentation

⚙️Lakeflow Designer

• Use the new Prepare operator for common data-cleansing tasks, including changing text case and data types, replacing values, and applying formulas. 📖Documentation

• Manually create tabular data in a spreadsheet-style editor. 📖Documentation

• Use the new Select operator to choose which columns to keep, rename columns, and reorder them in the output. 📖Documentation

• Create and view visualizations directly on the canvas. 📖Documentation

• The Output operator supports inserting or appending data in Unity Catalog tables, publishing results as materialized views and writing CSV, Excel, or JSON files to Unity Catalog volumes.

• The Join operator provides three outputs by default: unmatched left rows, matched rows, and unmatched right rows. This makes it easier to branch a workflow based on join results.

• The Join operator now includes a Match case option and matches string keys case-insensitively by default.

• The Filter operator provides separate outputs for matching and non-matching rows, so you can branch data based on whether it meets the filter conditions.

• The AI Function operator now supports ai_query, ai_parse_document, ai_forecast, and ai_prep_search.

• The Source operator can target an entire Unity Catalog volume or folder path instead of an individual file.

• Use Unity Catalog metric views as source nodes in Lakeflow Designer.

• The Note operator supports inline rich-text editing directly on the canvas, including headings, text styling, links, images, and tables.

• Unconfigured operators have a dashed border, and required input ports that aren’t connected are highlighted in yellow.

• Find and navigate to any operator using the new searchable table of contents in the left sidebar. Select an operator to focus it on the canvas, then expand the Generated code section to view its underlying code.

• Command palette: Press Cmd+Shift+P on macOS or Ctrl+Shift+P on Windows to quickly access editor actions.

• Find all of your visual data prep files from a dedicated Visual data prep page in the sidebar.

• When working with visual data prep files, you can choose the serverless environment used for both interactive runs and scheduled jobs.

• Declarative Automation Bundles support: Deploy and run visual data prep files with Declarative Automation Bundles by configuring them as notebook tasks and referencing the file path in notebook_task.notebook_path. 📖Documentation

🛠️Data Engineering

Column popularity in the catalog explorer

You now have more insights about your tables thanks to the column popularity feature available in the catalog explorer. The Genie Ontology is going to rely on these as well as table insights to determine the ranking of the sources based on OntoRank

New Lakeflow Connectors are available

• Slack Access and Integration logs

• Managed RabbitMQ

• Zip

• Workday Reports

• Zoom Logs

• Pendo

• Aha Connector

• Zoho Books

• Monday.com

• Wiz Audit logs

• Salesforce Marketing Cloud

• Netskope Logs

• Square

• Sharepoint Connect (Structured file ingestion and file metadata support)

Table updates triggers on Delta Sharing and System tables

You can trigger a job whenever a provider updates a shared table or view or whenever new operational data lands in a system table.This feature supports Databricks-to-Databricks Delta Sharing only table update.

External access for Streaming Tables and Materialized Views

External Delta and Iceberg clients can read Streaming tables and Materialized Views through Unity Catalog and Iceberg Catalog rest APIs without copying the data. 📖 Documentation

Configure the recovery period for dropped managed tables

You can configure the recovery period for Unity Catalog managed tables and set the recovery period to 0 hours or between 7 to 30 days at the catalog or schema level. 📖 Documentation

Comment on Foreign tables

Lakehouse federation can sync table and column comments from supported external data sources to foreign tables in Unity Catalog. It’s important for Genie Ontology.

Lakeflow Pipelines enhancements

• You can change the owner of standalone pipeline streaming tables and materialized views.

• You can add data quality expectations to materialized views created in Databricks SQL using constraint expectation_name clause.

Parquet v2 for Delta lake tables

Parquet v2 uses advanced encodings, v2 data page headers and INT64 timestamps to improve query performance and reduce table sizes. You can enable it by setting the delta.parquet.format.version table property to 2.12.0 or enable automatic upgrades

Subscribe now

Data Warehousing

Lakehouse RT

Lakehouse RT is a serverless SQL warehouse type built for low-latency, high-concurrency workloads, delivering sub-second SQL read queries against Unity Catalog tables. Use it to serve analytical data to applications, run operational analytics and power dashboards for hundreds to thousands of concurrent users. 📖 Documentation

Excel Add-in enhancements

You can use the Add-in to write data from MSFT Excel to a Unity Catalog table by creating a new table or overwriting an existing one without leaving excel.

🖥️ Platform

Run a subset of job tasks with their dependencies

When you run a subset of a job's tasks, you can expand your selection to include each task's dependencies. Add the + modifier to a task key to also run its upstream tasks (+my_task), its downstream tasks (my_task+), or both (+my_task+). Start a partial run from the UI, the REST API, the Databricks CLI, the SDKs or Declarative Automation Bundles. 📖 Documentation.

Automatic identity management for Okta

You can sync users and groups from Okta in Databricks without configuring SCIM provisioning. 📖 Documentation.

Lakehouse Replay

It protects your production environment by automatically testing a sample set of your read-only serverless workloads against upcoming Databricks Runtime releases. It runs entirely in the background with zero setup or maintenance required. Best of all, your data stays securely within your workspace and Databricks covers 100% of the compute costs for the replay. 📖 Documentation.

Schedule deferred policy enforcement for all purpose compute

When you enforce compute policy compliance on out-of-compliance all-purpose compute, you can schedule the configuration update to apply the next time the compute terminates or restarts, instead of restarting it immediately. 📖 Documentation.

Email notifications for expiring SPT

Databricks automatically sends email notifications to workspace admins when a service principal token is within seven days of expiring

You can use OAuth to authenticate Databricks connections in Power Platform

Network policies can be configured to restrict outbound traffic to specific external destinations

You can block internet destinations from serverless workloads through the network policies REST API. Blocked destinations are enforced regardless of the policy’s network access mode. 📖 Documentation.

🤖GenAI & ML

• Vector Search has been renamed to AI Search. You can create full text search indexes without any vectors or embeddings needed.

• You can enable web search on Supervisor Agent to answer questions that requires information from internet. It returns answers with source citations. Web search is available as a databricks_web_search tool in the supervisor API.

• Agent Bricks Supervisor Agent supports Unity Catalog volumes as sub agent tools

• AI Search supports query-time options for full-text and hybrid queries: 📖 Documentation

• Anthropic Fable 5 and Sonnet 5 are available as a Databricks Hosted Model.

• You can use the air Command line interface to submit and manage distributed training workloads on AI Runtime Serverless GPU compute.

• Request and response payload are delivered to Unity catalog inference table within seconds.

• Omnigent is a coding agent meta-harness that wraps coding agents like Claude Code or Codex. A web UI, persistent and shareable sessions, team collaboration, mobile access, contextual policies, and deployment infrastructure. Swap one line of configuration to switch runtimes while your tools, policies, and prompts stay the same.📖Documentation

• You can bring your own Docker container image to AI Runtime CLI workloads.

• Managed agent memory gives your AI agents long-term memory across conversations. Memory is stored in Unity Catalog memory stores, governed as Unity Catalog securables and accessed through the Unity Catalog REST API. 📖 Documentation

Lakebase

• Starting June 1, 2026, snapshot storage is billable for Lakebase Autoscaling. Snapshot storage size is shown on the Backup & Restore page.

• Lakebase Autoscaling now supports Postgres 18.

📝Genie Agents

• Databricks Genie App for Teams: 📖Documentation

• Databricks Genie App for Slack: 📖Documentation

• Genie Chat prompt monitoring: Prompts and responses initiated by Genie Chat are visible in Genie Agents monitoring. 📖Documentation

• Knowledge store edit confirmation dialog: A confirmation dialog appears before an author removes a table that has knowledge store edits. 📖Documentation

• Delete conversations: Users with CAN MANAGE permission can delete the conversations of other users from the UI. 📖Delete a conversation

• Embed Genie Agent as an iframe:📖Embed a Genie Agent in an external app.

• Save visualizations to a dashboard: You can save Genie Agent visualization outputs to a dashboard. 📖Save a visualization to a dashboard.

• Reference previous visualizations: Select and reference visualizations from previous prompts in follow-up prompts.

• Genie Code for metric view export: Use Genie Code when exporting metric views from a Genie Agent to refine the metric view definition. 📖Documentation

• Deploy Genie Agents as Declarative Automation Bundles resources:📖Documentation

📊AIBI Dashboard

• Dynamic annotation lines: Annotation lines can be driven by data or a calculation instead of a fixed value, letting you show running averages, variable thresholds, or anomaly markers that update with your data. 📖Documentation

• Six new preset themes are available for AI/BI dashboards for more ready-made options without creating a custom theme. 📖 Documentation

• Table header color customization: Authors can set font color, background color, and text formatting for table column headers. 📖Documentation

• Dashboard relationships allow users to create complex multi-fact data models. 📖 Documentation

  

• Local metric view parameters: Local metric views support parameters, allowing Databricks to dynamically change query inputs based on user selections in a UI. 📖Documentation

• Local metric view wildcard expressions: Local metric views support wildcard expressions, which allows users to inherit all fields from upstream tables or metric views.

• Bookmark filter combinations: Dashboard authors can save filter combinations as shared bookmarks, which viewers can quickly apply. 📖Documentation

• Pivot table column improvements: Pivot table columns can expand and collapse. 📖Documentation

• Theme customization enhancements: New theme customization options are available, including the ability to change axis and grid line colors, adjust widget layout and apply a color ramp. 📖Documentation

• New Workspace theme settings are available and automatically apply to new dashboards. 📖Documentation

• Dashboard variables: Dashboard authors can create dashboard variables that allow viewers to swap fields on visualizations. 📖 Documentation

• Customizable table headers: Dashboard authors can customize table column headers. 📖Documentation

• Widget-level color gradient: You can derive a color gradient from the visualization color at the widget level, matching the behavior available in theme settings.

• Gantt charts now available: Gantt charts display tasks as horizontal bars on a continuous axis to visualize project schedules, manufacturing workflows, and other time-ranged events. Each bar requires matching start and end field types (date, datetime, or numeric). 📖Documentation

• Custom visualizations: Build custom charts with the Vega-Lite library for customization beyond the built-in visualization types. 📖Documentation

• Per-category font customization: You can customize fonts per category across all visualization types. 📖Documentation

• Table row cross-filtering and drill-through: Select table rows to cross-filter, or right-click to drill through. 📖Documentation

• Cross-filtering and drill-through: Line, combo, and area charts support cross-filtering and drill-through filters. 📖Documentation

• Dashboard details in draft mode: Draft mode shows dashboard details, making it easier to find dashboard managers and the dashboard’s location.

• New metric view low-code editor syntax : The following syntax is available in the metric view low-code editor:

  • JOIN … RELY

  • WINDOW OFFSET

  • WINDOW INCLUSIVITY

  📖Documentation

📊Genie Code

• Genie Code space authoring skills: Genie Code includes dedicated skills for authoring and maintaining Genie Agents.

• Power BI enhanced report format support: Genie Code supports Power BI files exported with the Power BI enhanced report format (PBIR). 📖Documentation

• Image upload suggestions for Power BI and Tableau imports: Genie Code suggests that authors upload images when importing files from Power BI and Tableau to improve migration quality.

• Genie Code metric view usage: Genie Code relies more heavily on Unity Catalog metric views rather than creating local ones.

• The full page experience is a command center for Genie Code, where the active thread is shown prominently, surfacing assets like notebooks and files alongside it as tabs when needed. You can run multiple threads in parallel, switch between them easily, and easily personalize Genie Code with skills, instructions, and MCP servers.

• Genie Code offers an auto-approve mode that approves tool actions. An AI classifier reviews each action and blocks risky ones. Databricks recommends keeping it off when working with production data or shared resources. 📖 Documentation

Genie One

• Genie has been renamed to Genie One.

• You can connect Genie to Microsoft Copilot Cowork.📖Documentation

• Share chats with account users: You can share Genie One chats with all users in your account as read-only. 📖Documentation

• New external source connections (Beta): Chat in Genie One supports connecting to Gmail and additional Microsoft 365 apps. 📖Documentation

• Scheduled task email prefix: Scheduled task notification emails use the prefix “Genie Task:” instead of “Task update:”.

• Scheduled tasks for consumer entitlement users: Consumer entitlement users can now create scheduled tasks without needing the SQL entitlement. 📖Documentation

• Account-level Genie customization: You can customize Genie at the account level, including colors, a logo, a markdown welcome message and pinned content. 📖Documentation

• Email tool for scheduled tasks: Scheduled tasks use an email tool to send emails, enabling more customized email formats and send conditions. 📖 Documentation

• Workspace-level Genie One customization: Workspace admins can customize Genie One at the workspace level. Under workspace settings, admins can configure colors, a logo, a markdown welcome message, and pinned content. 📖 Documentation

• Certified object prioritization: Chat in Genie One acknowledges and prioritizes certified objects.

• User skills : You can create personal user skills to extend chat in Genie One with custom capabilities. A skill is a repeatable task you teach Genie One once and run whenever you need it, such as building your weekly metrics report. 📖Documentation

• Rich output : Genie One responses for non-SQL answers, such as search results and MCP reads, can now include rich HTML, CSS, and JavaScript content, enabling more varied data presentations beyond plain prose.

• Genie Ontology Genie One automatically builds and maintains a map of your business by extracting and ranking knowledge from sources such as dashboards, notebooks, and pipelines, improving response accuracy and reducing latency. 📖 Documentation

• Compliance security profile iframe embed: Customers with compliance security profile enabled can enable embedding Genie Spaces as an iframe. 📖 Documentation

• You can use the Databricks Genie mobile app to access Genie One from iOS and Android devices.📖Documentation

Metric views

• Median and percentile expressions in metric views: Low-code metric view building supports two new expression types: median and percentile.📖 Documentation

• You can define parameters in metric views to bind values at query time, serving many query variants from a single definition instead of maintaining a separate metric view per variant. 📖Documentation

🛡️Governance and Sharing

Govern the access to Foundation Models with Unity Catalog

Admins can grant permissions on the system.ai to control which models your users can access. The permission are enforced across pay per token, provisioned throughput and batch ai functions workloads.

Delta Sharing is now OpenSharing which is the new open standard for sharing data and AI assets across platforms and organization

ABAC grant policies for models

A GRANT policy can grant the EXECUTE privilege on models, attached at the catalog or schema level. Both customer-registered MLflow models and Databricks-hosted foundation models in system.ai are supported. 📖 Documentation

OIDC supported for sharing to iceberg clients

You can use an Iceberg REST Catalog client such as Snowflake or OSS Spark to read OpenSharing shares secured with OpenID Connect (OIDC) federation.

Share data behind a firewall with SecureConnect

Providers can use Opensharing SecureConnect to share data from Cloud storage behind a firewall or private endpoint without adding each recipient’s network to an allow list. 📖Documentation

Share Genie Agents externally using OpenSharing

Providers can share Genie Agents with users using OpenSharing. Databricks creates a point-in-time snapshot of the space's data assets and instructions. 📖Documentation

Access requests across Unity Catalog metastore

You can enable access requests at the UC metastore level.

Discover page is available

Discover is a browsing experience that help users find the right assets without the need of a deep knowledge. It’s an easy way to organize tables, dashboards, Genie Agents by business area.

Govern LLMs in Unity Catalog with Model Services

A model service is a Unity Catalog securable that represents a governed LLM endpoint, so you can define an endpoint once and share it across workspaces using Unity Catalog privileges, instead of duplicating endpoints per workspace.📖 Documentation.

Subscribe now

The scale of what was accomplished over those few days was historic breaking records across the board:

• 100K Total Attendees: A powerhouse community of data minds, including 32,000 innovators packing the venue in person, with the rest of the global community tuning in remotely from every time zone.

• 174+ Countries Represented: A truly global phenomenon, uniting diverse perspectives and top talent from every corner of the world.

• 240+ Exhibitors: A large ecosystem showcase featuring the industry-leading partners and technologies shaping tomorrow’s data landscape.

The summit may be over but the momentum is just getting started. The connections made, skills learned and breakthroughs shared have officially set the trajectory for the next era of data and AI!

Let’s start with the Unity Catalog

Governance Hub is a centralized command center for data stewards and admins to govern Databricks. You can monitor the data, AI, cost and Performance from a single UI

Attribute Based Access Controls are GA for row filtering and column masking as well as Data Classification and Governed Tags to help you protect you precious assets.

ABAC are getting extra features:

• You can define the attribute once to automatically grant execute permissions across all matching models.

• You are going to be able to build access rules using live user properties synced from the IDP.

• You are going to be able to leverage request context. Whether access originates from an agent or a workspace or application to securely handle application.

• You are going to have the opportunity to carry governed tags from sources tables and columns to downstream tables and views as data is transformed

• RBAC is coming soon: you are going to be able to complements Databricks’ inheritance-based identity and permission model by enabling you to define groups that behave like roles. When you assume a role you act like the role and all the actions including data access are authorized as that role

With the rise of the LLM the CONTEXT is now more important than ever

UC Metrics are very important as they provide agents and data citizens with a single resources to compute KPIs organized by domain and accessible via API, MCP and SQL.

In addition to UC metrics, you can now define a page in Glossary. It lets you define authoritative concepts and taxonomies that helps agent and people understand your business. You can connect many pages to each others, capturing the relationships.

Genie Code is here to help you draft Glossary pages, suggests improvements and more importantly flags definitions that drift from how the data is actually used.

Domains are also as important as Glossary and UC Metrics, you can organize your Data and AI assets into categories that fits your business needs giving agents and people the relevant context needed.

Domains, Glossary and UC Metrics are the foundations that feeds Genie Ontology which is a continuously learned entreprise context layer in Databricks.

The Lineage Experience

Unity Catalog offers an amazing lineage experience. In additional to the automatic inferred lineage you can now extend lineage in UC to assets beyond Databricks by registering upstream source systems and downstream BI reports so a single lineage graph spans your full data flow end to end.

Unity Catalog offers a wide variety of features one of them is table insights where you can see table usage, frequent joined tables, frequent notebooks, dashboards, queries in addition to that you get to see column level popularity from the table’s overview. This also feed Genie Ontology by giving a sharper sense of which columns matter most when reasoning.

CHOICE: choose the open infrastructure that suits your agentic era

Your infrastructure has to be flexible and can run wherever capacity exists that’s why Unity Catalog is now expanding to govern access across your entire footprint spanning accounts, regions and clouds. With the 4 level namespace metastore.catalog.schema.table gives every asset a single address across the entire realm with a unique unified experience. You are going to be able to govern your assets cross-cloud and cross-account. In addition you are getting the Managed disaster recovery feature to support the resilience for the mission critical workloads. It replicate the critical parts of your Databricks deployment to a secondary region and failover to it within minutes of a disaster.

Unity Catalog is the most advanced and open Catalog on the market offering a cross-format and cross-platform interoperability.

• You can read and write from external engines like Apache Spark or Apache Flink to Unity Catalog Managed Delta Tables

• A new file type lets managed tables natively govern unstructured data like images, PDFs, audio, and video.

• You have a geospatial support for Delta lake and Apache Iceberg

A Data and AI platform is only good when it allows you to do open sharing by offering a wide collaboration ecosystem. Databricks pioneered by Open sourcing Delta Sharing few years ago and now Databricks is announcing the next evolution of Delta Sharing: OpenSharing which introduces the first Open vendor neutral protocol for securely sharing AI assets such as Skills, AI Models and unstructured data.

You can use SecureConnect to enable secure connectivity across clouds with Zero-copy data sharing and a Multi-model collaboration by enabling the collaboration on AI assets and applications as well as Genie Sharing.

Building an Open Ecosystem for AI Governance with Unity AI Gateway

Visibility is an important aspect of how the AI is used in addition to being able to control what agents can access to protect entreprises from AI threats.

Unity AI Gateway is Databricks solution for entreprise AI. Built on the foundation of Unity Catalog. It’s extending governance beyond Data and AI assets to the runtime interactions between models, agents, MCP Servers, Skills… Unity AI Gateway is the FEATURE to monitor activities, manage spend and monitor activity across frameworks and providers

• Agent registry: Discover all agents including external agents

• Contextual policies: Admin can allow, deny or require approval for actions like pushing code or writing documents

• Budgets: It gives you full control over your organization’s AI spend across all models from the coding agents to agent serving your customers…

• Smart routing: Cross provider, cloud fallbacks and traffic splitting

• Agent Tracing: Analyze coding logs to spot bad practices and integrate with Lakewatch for alerts and monitoring

Governed Vibe Coding for Entreprise Apps on Databricks

Apps are growing massively as well as the usage of coding agents, but without context and control, “vibe coding” hits a wall in the enterprise.

Databricks Apps active deployment has nearly doubled and weekly users have tripled, fueled by non-technical teams using coding agents to build apps in hours. However, speed isn't enough for corporate environments. For wide impact, enterprise apps must be grounded in real business data (context) and strictly managed for compliance and safety (control).

To solve this, Databricks introduced three key capabilities at the Data + AI Summit 2026 to bring enterprise-grade context and control to Databricks Apps:

• App Spaces: A new governance boundary designed to manage resources, access control, and security policies across groups of applications.

  

• Genie App Builder: A specialized AI authoring tool built with native awareness of an organization’s specific data assets, workspace context, and Unity Catalog semantics.

  

• Serverless Micro Apps: A new class of applications that scale up instantly when needed and scale down to zero when idle, allowing companies to maintain a massive app portfolio without the burden of always-on infrastructure costs.

Databricks is accelerating its shift toward AI-native data and ML workflows, bringing agentic intelligence to every stage of the development lifecycle. At the core of this evolution is Genie Code, an AI partner that helps data and ML teams build, debug and improve systems faster directly within the platform.

What about Genie ?

Databricks Genie products have grown over 10x in the past year and are used by 90% of Databricks customers to build models, debug pipelines, and analyze data faster.

Databricks is expanding Genie Code to support more complex, autonomous and AI-native workflows with three major updates:

• Full-Page Command Center: Moving beyond a small sidebar, teams can now use a dedicated, full-page experience to manage multiple concurrent threads, track progress across different assets (notebooks, SQL, pipelines) and seamlessly run parallel tasks without losing context

  

  .

• Agentic ML Engineering: Built directly into the existing agent, Genie Code now acts as a specialist for production machine learning. It integrates natively with MLflow, Model Serving and AI Runtime compute to write features, optimize GPU utilization, and debug infrastructure. It uses Genie Ontology to learn your team’s specific business metrics and past experiments rather than relying on generic defaults.

• Scheduled Tasks & ZeroOps: Moving from interactive chat to autonomous work, users can now schedule Genie Code to run prompts overnight such as summarizing pipeline runs, reviewing model drift, or analyzing dashboard metric changes so results are ready for review the next morning. This pairs with Genie ZeroOps to extend agentic automation into live production operations and fixes.

Part 2 is about Lakehouse // RT, LTAP, ZeroOps and Omnigent. Coming soon

Don’t forget to check out the NextGenLakehouse Website, Instagram, Tiktok, X and Youtube

Subscribe now

Register now

🛠️Data Engineering

Query-based connectors in Lakeflow Connect are available

Query-based connectors ingest data from databases by querying the source directly using a cursor column, without requiring change data capture configuration or an ingestion gateway.

Lakeflow Designer updates:

• Bidirectional AI-generated descriptions

• N-way Combine

• Custom join conditions

• Improved filter readability

• Multi-modal output types

• Configurable output panel

• Improved drag-to-connect hit target

• Genie Code change summary

• SQL and Python operators surface available parameters with inline sample code showing how to reference them.

• AI operator search

• Configurable sample size

• Git, import, clone, and export:

• User-defined operators

Lakeflow Declarative Pipelines updates

• You can achieve end-to-end latency as a five milliseconds for operational streaming workloads such as real time personalization. RTM uses the new update_flow decorator which writes to sinks using update output mode and support stateful aggregation without requiring a watermark 📖 Documentation

• You can use refresh policies to control when Databricks performs incremental or full refresh to optimize the costs.

• You can use explain create materialized view to check whether a query can be incrementalized before defining the materialized view. 📖 Documentation

• Pipeline parameters let you define key-value pairs at the pipeline level and reference them from SQL source code using named parameters 📖 Documentation

• Standalone pipelines: including materialized views and streaming tables can be created from a notebook running on serverless general compute.

• You can configure an environment version on a pipeline to pin the Python language version and set of preinstalled libraries decoupling pipeline's Python runtime from Databricks Runtime upgrades

New Lakeflow Connector

• Google Drive managed Ingestion Connector 📖 Documentation

• Use a registered connector or create your own with Community Connectors 📖 Documentation

• Smartsheet Connector 📖 Documentation

• Github managed Ingestion Connector 📖 Documentation

• Outlook managed Ingestion Connector 📖Documentation

• HubSpot Ingestion Connector 📖Documentation

Data Warehousing

• You can use the connector to import and query Unity Catalog data. 📖 Documentation

• You can view the run history of scheduled refreshes in the Databricks Connector for Google Sheets. 📖 Documentation

• XPath no longer fetches external DTDs

• NULL struct preservation in INSERT, MERGE and streaming writes with schema evolution

• NullType (VOID) is supported in Delta tables

• SHOW CREATE TABLE supports metric views

• NATURAL JOIN respects case-insensitive column matching

• SQL UDF dependency validation in Unity Catalog

• Optimized writes for partitioned Unity Catalog tables created with CRTAS

• CREATE OR REPLACE support for temporary tables

• Delta table history includes write option flags

• The following geospatial functions are available:

  • st_makepoint function: Returns a point GEOMETRY with the given coordinates.

  • st_makeenvelope function: Returns a GEOMETRY value representing the axis-aligned envelope (bounding box) defined by the given corner coordinates.

🖥️ Platform

Configure additional recipient for workspace operational emails

Workspace admins can add an additional email recipient to receive operational emails about a workspace. This address receives the same operational updates as workspace admins including things like breaking changes, incident communications and feature rollout announcements.

If you want to deliver to multiple people, use a distribution-list alias. 📖 Documentation

Databricks Apps Horizontal scaling

You can collect traces, logs and metrics from your Databricks Apps and persist them to Unity Catalog tables using the OpenTelemetry protocol. 📖 Documentation

Databricks Apps Horizontal scaling

You can run an app across multiple instances behind a single app URL. It provides a higher availability with zero downtime deployments and a session affinity which routes every request from a given user to the same instance so the app can keep per user data on that instance.

If you already have a standard app you can convert it to easily to horizontally scaled else you can enable it during the creation 📖 Documentation

What’s new in the Workspace sidebar navigation and notebooks

You can view profiling statistics for columns in notebook results tables. From a results table, select column headers and click Open selection details to view profiling statistics.

Enforce Execution Timeouts for Serverless Notebooks

Workspace admins can configure the timeout by going to Settings > Compute and, under Serverless interactive, update the Serverless interactive execution timeout setting.

The default timeout is 2.5 hours and users can still override the timeout for an individual notebook by using spark.databricks.execution.timeout

Automatic Identity Management with Entra ID is GA

Automatic identity management lets you sync users, SPs and groups from Microsoft Entra ID to Databricks without SCIM provisioning. When enabled you can search for identities in identity federated workspaces and add them directly to your workspace and group memberships stay in sync with Microsoft Entra ID.

Account access denylist for Automatic Identity Managent

Account admins can add users, groups or SP to the denylist to block their access to the Databricks Account. Denylist membership is transitive, denying a group also denies all of its members.

System tables updates

system.compute.instance_events & system.compute.instance_pools are available. You can use these tables to track state transitions of classic compute instances and the full history of instance pool configurations across your account.

Extra

• You can configure Microsoft Entra ID as the identity provider for (SSO) to Databricks from Power BI using an account-level federation policy. 📖 Documentation

• You can use the OAuth U2M flow to obtain tokens as a service principal's identity instead of your own user account.

• You can use the following features in workspaces encrypted with customer-managed keys:

  • Production monitoring of GenAI apps with MLflow 3 scorers.

  • The Review App for collecting expert feedback on traces.

  • Evaluation datasets. The catalog that stores the dataset must not itself be CMK-encrypted.

• You can use the %uv pip magic command in serverless notebooks on environment version 5 or above as a faster alternative to %pip for Python package management.

• SQL Alert tasks in Lakeflow jobs is available and SQL Alerts are GA

Lakebase

• New Lakebase Autoscaling projects have native Postgres password authentication disabled by default.

• Customer-managed keys for Lakebase is available

• Lakebase autoscaling supports compute sizes up to 128GB

• Lakehouse Sync enables continuous low-latency replication of your Lakebase Postgres tables into Unity Catalog managed Delta tables using Change Data Capture (CDC) 📖 Documentation

• New Lakebase Autoscaling instances scale to zero by default after 24 hours of inactivity, reducing compute costs for idle projects.

• When you delete a Lakebase project, it enters a soft-deleted state for 7 days before permanent deletion. You can recover the project during that window, restoring all branches, databases, endpoints and project settings

🤖GenAI & ML

Document intelligence on Databricks

Next-Gen Hosted Models

You can access Claude Opus 4.8, Alibaba Cloud Qwen3.5 122B A10B directly as Databricks-hosted models through Foundation Model APIs.

You can serve custom LLMs with Custom Model Serving

You can serve custom and fine-tuned LLMs on Databricks with a vLLM engine, including multimodal models and PEFT recipes that Foundation Model APIs does not support. 📖Documentation

Agent Bricks updates

You can add vector search indexes as subagent tools in your Supervisor Agent.📖 Documentation

📝AIBI Genie

• Agent mode thinking traces: Agent mode thinking traces cite the definitions applied to answer questions.

• “Fields” replaces “Dimensions”: Genie Spaces uses “Fields” instead of “Dimensions” in the metric view and SQL expressions interfaces. 📖Documentation

• Benchmarks support agent mode: Benchmarks support agent mode, which uses the same multi-step reasoning as Genie Agent mode. Use the optional evaluation note to guide LLM-based grading. 📖Documentation

• Context window closed by default: In a new Genie Space thread the context window is now closed by default.

• Create API instruction limits raised: The create API enforces the same limits as the UI for instruction creation. The maximum characters for query text has been raised from 20K to 40K

• Answer now in Agent mode: Users can click Answer now to stop Agent mode reasoning and get an answer based on the existing context. 📖Documentation

• Request review on Agent mode conversations: Users can request a review on Agent mode conversations. 📖Documentation

• Export context as metric view: You can export the context from a Genie Space as a metric view to centrally define your business semantics.📖Documentation

• Iframe monitoring (Beta): Messages from iframe-embedded Genie spaces are logged in the Monitoring tab.

• Compliance security profile iframe embed (Beta): Customers with compliance security profile enabled can enable embedding Genie Spaces as an iframe. 📖Documentation

📊AIBI Dashboard

• Banded rows in tables: Tables can display banded rows.

• Custom geographies in maps: Custom geographies are available in choropleth and point maps. 📖Documentation

• AGGREGATE OVER enhancements: The AGGREGATE OVER operator supports an optional INCLUSIVE or EXCLUSIVE keyword for TRAILING and LEADING ranges and an optional OFFSET clause that shifts the range by a specified interval. 📖Documentation

• Point map shape field: Point maps support a shape field to differentiate data points by category.

• Default table sorting: Tables support default sorting, allowing dashboard authors to define a default sort order for table columns.

• Import BI workbooks: Add a Tableau or Power BI file to Genie Code and it builds an AI/BI dashboard that replicates your visualizations. 📖Documentation

• Open-source fonts: Over 20 new open-source fonts are available as defaults.📖Documentation

• Parameter values in text widgets: Dashboard authors can reference parameter values in text widgets. 📖Documentation

• Answer now for Genie on published dashboards: Genie on published dashboards now supports Answer now. Answer now stops Genie from thinking and produces an immediate answer to the user question, at the cost of deeper analysis.

• An API is available for dashboard authors to reset the draft to the last published version.

• Undo-redo on the data page

• Iframe monitoring: Messages from iframe-embedded Genie spaces are now logged in the Monitoring tab.

• Compliance security profile iframe embed: Customers with compliance security profile enabled can enable embedding Genie Spaces as an iframe. 📖Documentation

📊Genie Code

• Genie Code citation links for monitoring: When using Analyze space usage, Genie Code responses include citations. Click a citation to open the relevant conversation directly in the Genie Code thread. 📖Documentation

• Genie Code for dashboard authoring is generally available. 📖Documentation

• Genie Code can render non-table outputs, including charts, text blocks, and other content.

• You can share Genie Code chat threads with other users, groups, or service principals. Recipients receive read-only access to view the shared chat thread. 📖Documentation

Genie

• Genie has a Managed MCP Server: Any agent can ask natural-language data questions across Genie Spaces and Unity Catalog data and receive grounded answers. 📖 Documentation

• Direct link to associated dashboard: Genie surfaces a direct link to the associated dashboard when hovering over entities in the chat pane.

• Ask Genie on the workspace homepage: Ask Genie has been added to the workspace homepage.

• Workspace instructions for Genie chat: Workspace admins can create a /Workspace/.genie_workspace_instructions.md file to add custom instructions that apply to all Genie chat conversations in the workspace.

• Atlassian and Glean MCP connections: You can connect Genie chat to the Atlassian and Glean MCPs from within the chat box.

• Server-sent events support: Externally hosted MCP servers that use server-sent events are now supported.

Metric views

• Median and percentile expressions in metric views: Low-code metric view building supports two new expression types: median and percentile.📖 Documentation

• Metric view SQL editor improvements: Several UX improvements to the metric view SQL editor, including taller SQL and expression editors in the side panel, layout padding improvements and a cropping fix in the join modal table source selector.

• Local metric view join autofill : Local metric views autopopulate join conditions when joining two or more tables.

• Export context as metric view: You can export the context from a Genie Space as a metric view to centrally define your business semantics. 📖 Documentation

🛡️Governance

Cross engine attribute based access controls

External engines can read Unity Catalog tables with ABAC policies enforces. This allows you apply ABAC row filters and column masks that are dynamically enforced even when querying from outside Databricks.

When an external engine queries a table with ABAC policies attached, Databricks uses a specialized serverless compute layer to filter and return sanitized data to the external engine. 📖 Documentation

Unity Catalog Managed Apache Iceberg tables, Foreign Iceberg Tables and Apache V3 features are GA

• Unity Catalog managed Iceberg tables: Create and manage Iceberg tables in Unity Catalog using Databricks Runtime, Databricks SQL or external Iceberg-compatible engines. 📖 Documentation

• Foreign Iceberg tables: Read Iceberg tables managed by external catalogs such as AWS Glue, Hive metastore, or Snowflake Horizon Catalog through Lakehouse Federation.📖 Documentation

• Iceberg v3 features: Use deletion vectors, the VARIANT data type, and row lineage on managed Delta Lake tables with UniForm, managed Iceberg tables, and foreign Iceberg tables. 📖 Documentation

Catalog Commits are GA

Serverless Compute

• CREATE OR REPLACE support for temporary tables: CREATE OR REPLACE TEMP TABLE syntax is supported, allowing you to create or replace temporary tables in a single statement.

• agg() alias for measure() function: agg()is available as an alias for the measure() function.

• pyspark.pipelines.testing is available as a convenience alias for dlt.testing APIs. Import Lakeflow Spark Declarative Pipelines pipeline testing utilities through either namespace.

• Delta table history includes write option flags

• Selectively replace data with replaceOn and replaceUsing DataFrame APIs

• Databricks One has been renamed to Genie

• Lakeflow Designer is available to all Databricks Users

Subscribe now

✨Tips and Tricks

Genie Code is an autonomous AI partner for data work in Databricks, deeply integrated with Unity Catalog so it understands tables, columns, lineage, and governance. It can explore data, train models, build pipelines, and create dashboards across notebooks Lakeflow, SQL, and dashboards. It also helps debug and maintain production systems including pipeline failures and GenAI app observability with support for external workflows via MCP

🛠️Data Engineering

Query-based connectors in Lakeflow Connect are available

Query-based connectors ingest data from databases by querying the source directly using a cursor column, without requiring change data capture (CDC) configuration or an ingestion gateway.

The supported sources include Oracle, Teradata, SQL Server, MySQL, MariaDB, and PostgreSQL, as well as all Lakehouse Federation data sources using foreign catalog ingestion. 📖 Documentation

Create and write to Unity Catalog from External Delta Clients

The Unity REST API supports write and create access to Unity Catalog external and managed Delta tables from external Delta clients such as Apache Spark.

External Delta tables support full read, write, and create operations.

Prerequisite: Creating and writing to Unity Catalog managed Delta tables requires catalog-managed commits to be enabled.

Lakeflow Designer is available to all Databricks Users

Lakeflow Designer is a visual, no-code, AI-native experience for data preparation and analytics. It lets users prepare and explore data through a drag-and-drop canvas and natural language.

Spark declarative Pipelines updates history is retained for 60 days

Lakeflow Spark Declarative Pipelines update history is retained for 60 days, up from 30 days. The extended retention applies to updates shown in the pipeline UI and returned by the Pipelines REST API. 📖 Documentation

Disable a task in Databricks Lakeflow Job

You can disable a task in a Databricks Lakeflow job to skip it at run time without removing it from the job. Disabled tasks keep their configuration and run history, so you can re-enable them later without rebuilding the task. 📖 Documentation

"Useful for temporarily skipping a downstream notification or expensive aggregation without modifying job definitions in source control."

NullType Void support in Delta Tables (DBR 18.2+)

Delta Lake tables support VOID columns. Previously, VOID columns were silently skipped by path-based DataFrame reads

Optimized writes for partitioned Unity Catalog tables created with CRTAS (DBR 18.2+)

Optimized writes are applied to partitioned Unity Catalog tables created with CREATE OR REPLACE TABLE ... AS SELECT (CRTAS).

Pyspark.pipelines.testing namespace alias (DBR 18.2+)

pyspark.pipelines.testing is available as a convenience alias for dlt.testing APIs. Import Lakeflow Spark Declarative Pipelines pipeline testing utilities through either namespace.

Improved auto loader listing perf (DBR 18.2+)

Auto Loader uses a more efficient listing method that improves listing speed for cloud storage sources.

Delta Table history includes write option flags (DBR 18.2+)

Delta table history (DESCRIBE HISTORY) includes write option flags in the operationParameters column for WRITE and REPLACE TABLE operations.

Selectively replace data with replaceOn and replaceUsing Dataframe APIs (DBR 18.2+)

Use these options to replace part of the table with the result of a DataFrame. replaceOn replaces rows that match a user-defined condition. replaceUsing replaces rows where specified columns are equal. These APIs complement the INSERT REPLACE ON and INSERT REPLACE USING SQL statements.

Data Warehousing

Google Sheets connector is GA

You can use the connector to import and query Unity Catalog data. 📖 Documentation

You can bring Governed Lakehouse data to Excel Sheets

The Databricks Excel Add-in connects your Databricks workspace to Microsoft Excel, bringing data directly into your spreadsheets to help you move from data to decisions faster. The add-in is fully supported across Excel for the web and desktop versions for both Windows and macOS. 📖 Documentation

You can configure 5X-Large Cluster for Serverless and SQL Pro

The 5X-Large size provides 512 workers. Workspace admins can control access to this feature from the Previews page.

Create or replace support for temporary tables

CREATE OR REPLACE TEMP TABLE syntax is supported, allowing you to create or replace temporary tables in a single statement. This eliminates the need to explicitly drop and recreate temporary tables.

🖥️ Platform

Git-backed app deployments are GA

You can deploy Apps from Git repos. You only need to configure a git reference 📖 Documentation

You can connect Lovable to Databricks

You can connect Lovable-hosted apps to your Databricks workspace to query lakehouse data using the REST API and OAuth machine-to-machine authentication

"why": vibe-coded UIs on top of governed Lakehouse data without leaving Databricks auth/audit.

New Databricks Apps Interface

The Databricks Apps console has a redesigned interface that provides a more intuitive app management experience. The updated console streamlines workflows for editing, deploying, and monitoring apps.

Language selector for AI generated comments

You can change the default language for AI-generated comments in Catalog Explorer. Your selection is saved in your browser's local storage and applies to all subsequent AI-generated comment requests across all supported entity types. 📖Documentation

Workspace base environments are GA

Workspace base environments let workspace admins create and manage pre-built, cached environments for serverless notebooks.

Lakebase

Lakebase Autoscaling Enhancements

• Lakehouse sync (Beta): Continuously replicate Lakebase Postgres tables to Unity Catalog Delta tables using CDC. 📖Documentation

• You can manage encrypted Lakebase Autoscaling project data using a customer-managed key (CMK) from your own cloud KMS. This gives your organization full control over encryption and helps meet regulatory and compliance requirements. 📖Documentation

• You can export Lakebase Postgres metrics and logs to any OTLP-compatible observability backend. You can configure the endpoint, authentication, and resource attributes from the Lakebase App. 📖Documentation

🤖GenAI & ML

AI_Prep_search() is available

It transforms the structured output of ai_parse_document into search ready chunks optimized for Vector Search and RAG 📖 Documentation

AI_Parse_document() is GA

This function parses structured content from unstructured documents including PDFs, images, Word documents, and PowerPoint files.

For now: documents are limited to a maximum of 500 pages and 100 MB.

Vector Search Retrieval Quality

Mosaic AI Vector Search provides built-in retrieval quality evaluation that measures and compares the relevance of different search strategies on your data. 📖 Documentation

CMK is supported on Mosaic Model Serving

You can use customer-managed keys to encrypt model serving container images and model artifacts stored in Databricks managed registry. Endpoint artifacts created after April 16, 2026 are protected with customer-managed keys

Next-Gen hosted models

You can access Anthropic Claude Opus 4.7, GPT-5.5 and GPT5-5 Pro as a Databricks Hosted models through Foundation Model APIs.

Supervisor API for building agents (Beta)

The Supervisor API lets you build agents in a single API call. Define your model, tools, and instructions, and Databricks handles tool selection, execution, and response synthesis.📖 Documentation

Data Classification is GA

Use Data Classification to automatically classify and tag sensitive data in Unity Catalog using an agentic AI system. 📖 Documentation

CPU_MEDIUM and CPU_LARGE workload type for Model Serving (Beta)

These workload types let you trade concurrency for more memory per worker on the same CPU hardware. Use them when your model needs more memory than the standard CPU workload type provides.

AI Runtime supports the 1xH100 acceleration

It’s designed for workloads that need more GPU memory than 1xA10 provides but do not require multi-GPU distributed training, such as fine-tuning medium-size language models

Manage Supervisor Agent Programmatically with the Databricks SDK (Beta)

You can use the Databricks SDK for Python to create and manage Supervisor Agent and their tools. 📖 Documentation

Data Quality monitoring improvement

The Data Quality Monitoring dashboard includes a Recently Resolved Incidents section. This section shows tables that were previously unhealthy but have since automatically recovered to a healthy status, including the time of resolution. 📖 Documentation

Agent bricks supervisor agent supports custom MCP servers and custom agents on databricks Apps

You can add custom MCP servers and custom agents hosted in Databricks Apps as subagents in your Supervisor Agent. 📖 Documentation

📝AIBI Genie

• Weekly digest and monitoring: The Monitoring tab includes a Weekly digest section showing weekly messages, users, and thumbs up/down feedback. Authors can also use Genie Code to summarize top usage trends and common issues from the monitoring page.

• PDF file uploads: Upload PDF files directly into a Genie Space to analyze their content using natural language. 📖 Documentation

• Databricks Apps search: You can search for Databricks Apps from the Genie search bar.

• Reasoning traces using the API: Users can access Genie’s reasoning traces using the API through the GenieQueryAttachments field in the GenieMessage object. 📖 Documentation

• Message comments using the API : Users can add text comments to messages and list them using the API. 📖 Documentation

• Increased token limit for Agent mode: The maximum token limit for Agent mode conversations increased, improving performance and context retention across long, multi-turn interactions.

• Embed a Genie Space: You can embed a Genie Space as an iframe in a website or application. 📖 Documentation

• Space thumbnails: Genie Space authors can attach thumbnails to a space. Thumbnails appear on the Genie Space’s initial chat page and in Genie.📖 Documentation

• Request Review for Agent mode: Users can request a review for Agent mode messages.

• Table visualizations with conditional formatting: Genie can generate table visualizations with conditional formatting colors for better readability.

• Agent mode as default: Agent mode is the default conversation setting in Genie Spaces.

• Account-level Genie is GA: Account-level Genie includes Databricks Apps, Domains, and Genie. 📖 Documentation

• Updated workspace Genie navigation: The workspace Genie left-hand navigation has been updated and includes Genie Code chat history.

• Column display names: Space authors can set user-friendly display names on table and view columns. Display names appear in query results and in downloaded CSVs.

• Genie Conversation API is GA: Use the API to programmatically start conversations, send questions, and retrieve results from Genie Spaces. 📖 Documentation

• Agent mode workspace themes: Visualizations in Genie Agent mode use workspace-level themes.

• Agent mode prompt caching: Genie Agent mode caches repeated prompts, reducing response latency.

• Navigation between Genie surfaces: Users can navigate between Account Genie and workspace Genie.

📊AIBI Dashboard

• Publish with service principal credentials: Authors can publish dashboards using the data credentials of a service principal. 📖 Documentation

• Service principal ownership: Workspace admins can transfer dashboard ownership to a service principal in the UI. 📖 Documentation

• Choropleth map admin levels: Choropleth maps support US admin levels 3 (regions, multi-state groupings) and 4 (states). 📖 Documentation

• SQL editor line numbers: The SQL query editor displays line numbers to help with legibility and debugging.

• PDF subscription page selection: Dashboard authors can select which pages to include in PDF email subscriptions. 📖 Documentation

• Parameter values in widget titles and descriptions: Dashboard authors can reference parameter values in widget titles and descriptions, so the text updates dynamically as viewers change parameter selections. 📖 Documentation

• Table cross-filtering and drill-through: Tables support cross-filtering and drill-through.

• Counter prefix and suffix: Numbers in counters support custom prefixes and suffixes. 📖 Documentation

• Schema browser default dataset type: Adding a table to a dashboard from the schema browser creates a local metric view by default instead of a SQL dataset.

• Warehouse overload message: Dashboards show a message explaining when rendering is delayed due to the warehouse being overloaded.

• Tabular attachments in email subscriptions: Dashboard email subscriptions include tabular attachments.

• Fullscreen scroll position: Exiting fullscreen mode on a published dashboard returns you to your previous scroll position instead of jumping to the top of the page.

• Local metric views: A new dataset type lets you create metric views directly in a dashboard using a low-code visual interface, without publishing to Unity Catalog first. 📖 Documentation

• Edit hex color values inline: Authors can click directly on a hex color value to edit it in place.

• View SQL for visualization widgets: Authors can view the SQL behind specific visualization widgets while in draft mode.

• Waterfall chart totals: Waterfall charts with categorical X-axis support a total bar.

• Scatter plot shape field: Scatter plots support a shape field to differentiate data points by category.

• Clear applied filters individually: Dashboard viewers can individually clear applied filters from the active selection bar.

• Text box vertical alignment: Text box widgets support vertical alignment (top, center, and bottom).

• Choropleth map boundaries: Choropleth maps support additional boundary types, including ZIP code and NUTS regions.

• “Explain this change” chart types: The “Explain this change” feature is available for pivot table cells, horizontal bar charts, pie charts, and heatmaps, in addition to time series charts. 📖 Documentation

📊Genie Code

• Workspace Admins can create Workspace Skills for Genie Code that can be shared to everyone within the workspace. 📖 Documentation

• Genie Code space configuration: Genie Code includes tools to configure your Genie Space. You can ask it to update instructions, example questions, descriptions, and more.

• Genie Code faceted charts: Genie Code can create faceted charts (also known as trellis or small multiple charts).

• Genie Code starter questions: Genie Code starter questions are scoped to the currently viewed dashboard page.

• Genie Code dataset referencing: Improved the quality of Genie Code for dashboards when referencing datasets while authoring.

Genie

Databricks One, the simplified Databricks interface, has been renamed to Genie. The feature and its capabilities are unchanged. Genie enables users to:

• View and interact with AI/BI dashboards to track key performance indicators (KPIs) and analyze metrics.

• Ask data questions in natural language using Genie.

• Use custom-built Databricks Apps that combine analytics, AI, and workflows.

Chat provides a unified, full-screen interface for asking data questions in natural language. It uses existing Genie spaces, dashboards, and queries to answer questions with your organization's data.

Metric views

• Metric view lineage highlighting: Hovering over metric views in the lineage panel highlights the visualizations built on those metric views.

• Metric views in Excel and Google Sheets: Unity Catalog metric views are supported in Excel and Google Sheets.

• Measures and dimensions creation use Builder mode by default: Builder mode, the UI-based transformation tool, is the default experience for editing measures and dimensions in metric views.

• SHOW CREATE TABLE supports metric views.

• agg() is available as an alias for the measure() function.

🛡️Governance

Attribute-based access control is GA

Governed Tags are GA

Governed tags allow administrators to define a controlled set of tags that can be applied to Unity Catalog objects and workspace objects, helping organizations enforce consistent metadata tagging for access control, data classification, and cost tracking. 📖 Documentation

SAP Semantic metadata syncs automatically into Unity Catalog

When you mount an SAP Business Data Cloud share to a catalog semantic metadata syncs automatically into Unity Catalog. You can use SAP governance tags in ABAC policies to control access to sensitive data. 📖 Documentation

Unity AI Gateway governs MCP

You can use AI Gateway to enforce access control, monitor usage, and audit activity across all MCP interactions in your workspace.

Altering Catalog or Schema Managed storage is GA

The managed storage location is the default location in cloud storage used when new managed tables and volumes are created in Unity Catalog catalogs or schemas

Mlflow trace storage in Unity Catalog tables is available

Traces are stored in OpenTelemetry (OTel) format, access is governed through Unity Catalog schema and table permissions, and you can query traces from Databricks SQL or the MLflow Python SDK. 📖 Documentation

Marketplace and Data sharing

Marketplace listing creation wizard

The provider console features a step by step listing creation wizard that guides you through setting up general information data assets attributes and details on separate pages. 📖 Documentation

Sharing Foreign Iceberg tables with Delta Sharing

You can share Foreign iceberg tables federated from foreign iceberg catalogs suing Delta Sharing. Providers can add foreign iceberg tables to a share and recipients access the shared data in read-only format. 📖 Documentation

Featured articles

• Native OpenTelemetry Observability for Databricks Apps

• Three MCPs, One Answer: Building a Data Quality Monitor on Databricks Apps

• Django and Databricks Apps

March 2026 Release Highlights

• Databricks Asset Bundles has been renamed to Declarative Automation Bundles, reflecting the product’s evolution beyond just assets to full declarative automation workflows.

• Databricks Runtime 18.1 Built on Apache Spark 4.1.0 with new vector functions, schema evolution in SQL INSERT, multi-table transactions, and geospatial improvements.

• Genie Code launches with expanded agentic capabilities for autonomous, multi-step data tasks across data science, engineering, and dashboard authoring.

✨Tips and Tricks

• AI Dev Kit: gives your AI Coding Assistant the trusted sources it needs to build faster and smarter on Databricks.📖 Documentation

• Build Databricks Apps faster with AppKit our brand new node.js+ React SDK

  📖 Documentation

🛠️Data Engineering

Declarative Automation Bundles

Databricks Asset Bundles has been renamed to Declarative Automation Bundles. The new name Declarative Automation Bundles more accurately reflects the usage and capabilities of bundles. In addition, the term assets caused some confusion as it has more than one meaning in Databricks. This name change is non-breaking. The bundle CLI command and all of your existing configuration does not need to be modified.

Auto Loader File Events by Default ( DBR 18.1+)

To reduce costs and listing latency, Auto Loader defaults to file events when loading from external locations where events are enabled. 📖 Documentation

Schema Evolution in SQL INSERT ( DBR 18.1+)

You can use the WITH SCHEMA EVOLUTION clause with SQL INSERT statements to automatically update target Delta tables, supporting additional columns or widened types from the source. 📖 Documentation

Optimized Writes for UC CRTAS ( DBR 18.1+)

Partitioned Unity Catalog tables created via CREATE OR REPLACE TABLE AS SELECT have Optimized Writes enabled by default, resulting in more efficient files.

Automatic Streaming Type Widening ( DBR 18.1+)

Streaming reads on Delta tables automatically handle column type widening without manual intervention.

Unity Catalog Volume Checkpoints ( DBR 18.1+)

DataFrame checkpoints support Unity Catalog volume paths. Configure the checkpoint path using SparkContext.setCheckpointDir on dedicated compute or the spark.checkpoint.dir configuration on standard compute.

Multi-Table Transactions ( DBR 18.1+)

You can group statements across tables using BEGIN ATOMIC ... END; syntax. All changes succeed together or roll back together. 📖 Documentation

SQL Pipelines Features

Materialized views and streaming tables support failure notifications and serverless performance mode configuration. 📖 Documentation

Data Quality Monitoring Alerts

You can create and manage anomaly detection alerts directly within the Data Quality Monitoring interface. 📖 Documentation

New Lakeflow Connectors

Native ingestion is available for Workday HCM (Beta) in Lakeflow Connect. 📖 Documentation

Declarative Feature Engineering APIs (Beta)

Available in AWS us-east-1 and us-west-2, you can declaratively define and compute time-windowed aggregation features from your data sources.

With declarative feature engineering you can:

• Define features in Unity Catalog.

• Materialize batch features to Delta tables or Lakebase.

• Serve Unity Catalog models using online model serving.

Data Warehousing

5X-Large SQL Warehouse (Beta)

A new 5X-Large cluster size is available in beta for serverless and pro SQL warehouses across all supported regions.

SQL Alert Task for Jobs (Beta)

You can add a SQL alert task to your Databricks jobs to evaluate Databricks SQL alerts as part of your workflows. 📖 Documentation

Dynamic Parameters in SQL Editor

The SQL editor supports a dynamic dropdown widget type for named parameter markers. You can select saved queries to automatically populate dropdown values that update as data changes.

Metric Views BI Compatibility Mode

Power BI users can query metric views using standard SQL aggregation functions by enabling BI compatibility mode, which automatically rewrites functions. 📖 Documentation

New usage Dashboard is available

New usage usage dashboard is available for account administrators to import customizable pre-built dashboards monitoring Databricks usage at account and workspace levels.

SQL Scripting and Cursors (DBR 18.1)

A massive win for SQL developers. Compound statements support SQL Cursors for row-by-row processing with DECLARE CURSOR, OPEN, FETCH, and CLOSE statements. 📖 Documentation

Advanced Sketch Functions (DBR 18.1)

New statistical power arrives with Approximate Top-K sketches (approx_top_k_accumulate, approx_top_k_combine, approx_top_k_estimate) and Tuple sketches for high-performance distinct counting. 📖 Documentation

New Vector Functions (DBR 18.1)

New aggregate functions (vector_avg, vector_sum) and scalar functions (vector_cosine_similarity, vector_inner_product, vector_l2_distance, vector_norm, vector_normalize) for vector operations directly in SQL.

parse_timestamp Function (DBR 18.1)

New SQL function (parse_timestamp) that parses timestamp strings using multiple patterns, running on the Photon engine for performance optimization.

Geospatial Enhancements (DBR 18.1)

New functions (st_difference, st_intersection, st_union) runs on Photon support for faster performance.

Connect to Databricks From Microsoft Excel

The new Databricks Excel Add-in allows you to connect Microsoft Excel to your workspace, import data from tables or Metric Views, execute queries and analyze data directly in Excel. Documentation

Schedule data refreshes in Google Sheets

You can schedule autonatic data refreshes in the Databricks Connector for Google Sheets on an hourly, Daily, weekly basis. Documentation

🖥️ Platform

Workspace Browser Tree View

The side panel includes a tree view that provides a flexible, tree-based view of your workspace folder hierarchy with folder expansion and navigation controls. 📖 Documentation

Notebook Tagging

You can apply tags to notebooks to organize and categorize them for easier management, including governance tags for certification and deprecation. 📖 Documentation

Databricks Apps improvements

You can manage Python dependencies using uv with pyproject.toml and uv.lock files. 📖 Documentation

Support for app-to-app communication and table resources that provide governed access to structured data. 📖 Documentation

You can collect traces, logs, and metrics from your apps and persist them to Unity Catalog tables using the OpenTelemetry (OTel) protocol. 📖 Documentation

Databricks Apps Compute sizing Supports all compliance security profile standards with default availability for workspaces with compliance security profile enabled.

Inbound Private Link

Inbound Private link for performance intensive services is Available for services like Zerobus Ingest and Lakebase Autoscaling. 📖 Documentation

Lakebase

Lakebase Autoscaling Enhancements

Added OAuth role management via UI/REST API, budget policies with custom tags, and Lakehouse Sync (Beta) for CDC replication to Unity Catalog.

• OAuth roles: Create OAuth roles using the Lakebase UI or REST API, in addition to SQL. 📖Documentation.

• Budget policies and tags: Associate a serverless budget policy and add custom tags to your project for cost attribution and tracking. 📖Documentation

• Lakehouse sync (Beta): Continuously replicate Lakebase Postgres tables to Unity Catalog Delta tables using CDC. 📖Documentation

Lakebase Autoscaling is available for compliance security profile workspaces with HIPAA, C5, TISAX, or None standards. 📖 Documentation

New Lakebase instances are created as Lakebase Autoscaling projects by default with key capabilities:

💯 Serverless autoscaling and scale to zero

💯 Instant database branching

💯 Point-in-time recovery (instant restore)

💯 Unified governance through Unity Catalog

Automatic failover across availability zones with no connection string changes required. 📖 Documentation

Autoscaling is available in ca-central-1 (Canada), eu-west-2 (London), and sa-east-1 (São Paulo).

You can add Lakebase Autoscaling databases as resources within your Databricks applications. 📖 Documentation

🤖GenAI & ML

Genie Code Launch

Genie Code is your assistant with expanded agentic capabilities for autonomous, multi-step data tasks. Agent mode is GA for data science, engineering, and dashboard authoring. 📖 Documentation

AI Runtime

GPU-enabled serverless compute designed for deep learning workloads, with single-node tasks and distributed training are available. 📖 Documentation

Next-Gen Hosted Models

You can access OpenAI GPT-5.4, GPT-5.4 mini, and GPT-5.4 nano directly as Databricks-hosted models through Foundation Model APIs.

Google Gemini 3.1 Flash Lite is also available for batch inference via AI Functions.

Agent Bricks enhancements

• Enhanced the Supervisor Agent with long-running task mode, which automatically breaks complex tasks into multiple request/response cycles using task continuation to avoid timeouts. 📖 Documentation

• Supervisor Agent is available in additional regions. Knowledge Assistant is available by default for Enhanced Security and Compliance customers with HIPAA controls.

• Classification and Information Extraction are available

Classification: Visual UI for classifying documents into predefined categories. 📖 Documentation

Information Extraction: Visual UI for extracting structured data from documents, PDFS and Images. You can define and refine an extraction schema and preview results. 📖 Documentation

AI_Classify and AI_extract support v2 Signatures

Both functions support new v2 signature with improved quality and performance , greater flexibility and native composability with ai_parse_document

AI_Classify : function classifies text content according to custom labels you provide. You can use simple label names for basic classification, or add label descriptions and instructions to improve accuracy for use cases like customer support routing, document categorization, and content analysis.

AI_extract : function extracts structured data from text and documents according to a schema you provide. You can use simple field names for basic extraction, or define complex schemas with nested objects, arrays, type validation, and field descriptions for business documents like invoices, contracts, and financial filings.

Custom Model Serving Telemetry

You can configure endpoint telemetry to persist logs, traces, and metrics from custom model serving endpoints into Unity Catalog Delta tables using OpenTelemetry. 📖 Documentation

📝AIBI Genie

• Inspect : It automatically improves Genie’s accuracy by reviewing the initially generated SQL, authoring smaller SQL statements to verify specific aspects of the query, and generating improved SQL as needed. 📖 Documentation

• Conversation sharing: You can share Genie conversations with privacy settings: Private, Reviewable by managers, or Account-wide.

• Space management APIs are GA: Create, Update, Get, List, and Trash APIs for Genie spaces are generally available. 📖 Documentation

• Benchmark APIs : The run benchmarks and retrieve benchmark results API, 📖 Documentation

• Workspace-level color palette: Genie spaces integrate with workspace-level color palettes for consistent branding.

• Improved context identification: Genie better identifies context from previous messages for more accurate responses.

• Ask Genie to explain chart changes: Users can right-click on bar, line, and area time series visualizations (including multi-series and stacked charts) and ask Genie to explain changes. Genie enters Agent mode to analyze the change and identify top drivers. 📖 Documentation

• Genie space descriptions on dashboards: Authors can add descriptions to Genie spaces embedded in dashboards. 📖 Documentation

• SQL download settings for full query results: The APIs to download full query results respect workspace-level settings for SQL downloads.

• Share Genie space with all account users: The share modal includes an option to share your Genie space with all account users

📊AIBI Dashboard

• External user embedding GA: External user embedding for dashboards is generally available. 📖 Documentation

• Hide Databricks logo: Option to hide the Databricks logo for embedded dashboards. 📖 Documentation

• Mobile-responsive layouts: Dashboards support mobile-responsive layouts for small screens.

• Custom email subject lines: Custom email subject lines for dashboard subscriptions.

• Custom font selection: Select custom fonts for your dashboards, applied to data point labels.

• Accounting-style number formatting: Parentheses formatting for negative numbers.

• Individual filter removal: Remove individual filters without resetting all filters.

• Waterfall chart faceting: Waterfall charts support faceting.

• Pivot table drill-through: Added drill-through functionality for pivot tables.

• Right-click explanations: Right-click explanations for time series visualizations.

• Dashboard snapshot audit logs: sendDashboardSnapshot events added to audit logs for Slack/Teams subscriptions.

• Date filter keyboard input: Fixed date filter keyboard input for relative date ranges.

• Point map recentering: Point maps auto-recenter when filters are applied.

• Advanced cells in pivot tables: Pivot tables support advanced cells including image, HTML, JSON, and link.

📊Databricks One and Genie Code

• Chat in Databricks One : Full-screen unified interface routing questions to relevant Genie spaces or providing direct agentic responses. 📖 Documentation

• External data source integration for Chat: Connect Google Drive, SharePoint, or custom MCP servers to conversations for document search capabilities. 📖 Documentation

• Account-level Databricks One: Cross-workspace discovery and unified search across your Databricks account.

• Genie Code for dashboard authoring: Automates multi-step workflows for dashboard creation and editing using natural language prompts.

Metric views

• Metric view lineage highlighting: Hovering over metric views in the lineage panel highlights the visualizations built on those metric views.

• Metric views in Excel and Google Sheets: Unity Catalog metric views are supported in Excel and Google Sheets.

🛡️Governance and Data sharing

Catalog Explorer Health Indicators

Tables display health indicators showing a summary of each table’s freshness and completeness status based on anomaly detection. 📖 Documentation

ABAC Policy snake_case Naming

ABAC policy tag functions use snake_case naming (has_tag, has_tag_value, has_column_tag, has_column_tag_value), consistent with other Databricks SQL built-in functions. The camelCase forms (hasTag, hasTagValue, hasColumnTag, hasColumnTagValue) continue to work for backward compatibility, but Databricks plans to deprecate their use when creating new policies in the future.

Databricks recommends updating any new policies to use the snake_case forms

CMK for Unity Catalog

You can protect Unity Catalog data with your own encryption keys using customer-managed keys (CMK). 📖 Documentation

Delta sharing Multi Statement transaction support

Delta sharing tables that use pre-signed URLs Cloud Token Sharing modes support Multi-statement transactions.

Featured articles

• Why was Databricks Asset Bundles renamed to Declarative Automation Bundles?

• Streaming syslog-ng data to your lakehouse: Powered by Zerobus Ingest OTEL

• Your Telemetry, Your Lakehouse

• From 19 Hours to 80 Milliseconds: A Geospatial Cheesecake

Curated hands‑on resources to master Databricks. Check it now

NextGenLakehouse

February 2026 Release Highlights

1. Anthropic Claude 4.6 and Gemini 3.1 Pro are available as Databricks-hosted models.

2. Auto Loader now uses file events by default and JAR tasks can run on Serverless compute.

3. Integrated Python unit testing (pytest) is available directly in the workspace.

✨Tips and Tricks

AI Dev Kit gives your AI Coding Assistant the trusted sources it needs to build faster and smarter on Databricks

🛠️Data Engineering

Apply filters, masks, tags a comments to pipeline created datasets

Using create or alter or the Lakeflow UI to modify ETL and ingestion pipelines you can modify pipelines to apply row filters, column masks, table and column tags, column comments, and (for materialized views only) table comments.

Auto Loader File Events by Default

To reduce costs and listing latency, Auto Loader now defaults to file events when loading from external locations where events are enabled. 📖 Documentation

Optimized Writes for UC CRTAS

Partitioned Unity Catalog tables created via CREATE OR REPLACE TABLE AS SELECT now have Optimized Writes enabled by default, resulting in more efficient files. 📖 Documentation

Schema Evolution in SQL INSERT

You can use the WITH SCHEMA EVOLUTION clause with SQL INSERT statements to automatically update target Delta tables. 📖 Documentation

Automatic Streaming Type Widening

Streaming reads on Delta tables automatically handle column type widening.

Unity Catalog Volume Checkpoints

DataFrame checkpoints support Unity Catalog volume paths, simplifying state management. 📖 Documentation

JAR Tasks on Serverless

You can run JAR-based Spark jobs on serverless compute, eliminating cluster management for JVM workloads.📖 Documentation

New Lakeflow Connectors

Native ingestion is now for TikTok Ads, HubSpot, Google Ads, and Zendesk.

Subscribe now

Serverless

Serverless improvements in version 5

• Serverless JAR jobs are supported

• Arrow optimization is enabled by default for Python UDFs, significantly improving UDF execution performance

Data Warehousing

Set a default SQL Warehouse

Workspace administrators can set a default SQL warehouse that is automatically selected in SQL authoring surfaces, including the SQL editor, AI/BI dashboards, AI/BI Genie, Alerts, and Catalog Explorer.

Individual users can also override the workspace default by setting their own user-level default warehouse.

View Warehouse activity details

You can view detailed annotations on the Running clusters chart in the SQL warehouse monitoring UI to understand why warehouses remain active. The Activity details toggle displays color-coded bars that show query activity, fetching queries, open sessions, and idle states. Hover over bars to see metadata, or click on fetching activity to filter the query history table. 📖 Documentation

SQL Scripting & Cursors

A massive win for SQL developers. Beyond basic scripting, compound statements now support SQL Cursors for row-by-row processing. 📖 Documentation

Query Tags for SQL Warehouse

You can apply custom key-value tags to SQL workloads on Databricks SQL warehouses for grouping, filtering, and cost attribution.

Query tags appear in the system.query.history table and on the Query History page of the Databricks UI, allowing you to attribute warehouse costs by business context and identify sources of long-running queries.

Tags can be set using session configuration parameters, the SET QUERY_TAGS SQL statement, or through connectors including dbt, Power BI, Tableau, Python, Node.js, Go, JDBC, and ODBC. 📖Documentation

.

Advanced Sketch Functions

New statistical power arrives with Approximate Top-K sketches and Tuple sketches for high-performance distinct counting. 📖 Documentation

Behavioral Change - Timestamp Partitions

Partition values use the Spark session timezone instead of the JVM timezone. If you have partitions written prior to Runtime 18.0, run SHOW PARTITIONS to verify metadata before writing new data.

DESCRIBE FLOW

Describe flow is a reserved command. If you have a table named flow, you must now use backticks (`flow`) to describe it.

Smart SQL Performance

SQL commands (like SHOW TABLES) no longer re-run when calling .cache() on the result DataFrame.

🤖GenAI & ML

Next-Gen Hosted Models

You can access OpenAI’s GPT-5.3 Codex, Qwen3 and Google Gemini 3.1 Pro directly as Databricks-hosted models and the latest reasoning models from Anthropic, Claude 4.6 Sonnet and Opus are available for pay-per-token serving.

Agent Bricks - Supervisor Agent

Orchestrating multi-agent systems is simpler and more reliable.📖 Documentation

AI Gateway

Manage and govern access to external LLM providers with the new AI Gateway, providing centralized rate-limiting and security. 📖 Documentation

🛡️Governance & Security

Delta Sharing Multi-Statement Transactions

Shared tables using pre-signed URLs now support multi-statement transactions, pinning the table version for consistent reads across a session. 📖 Documentation

Global Compliance Expansion

Databricks supports ISMAP compliance (Japan) and HITRUST compliance controls

Discover page and domains

The Discover page provides a centralized interface for searching, browsing, and previewing data assets governed by Unity Catalog. This Beta release introduces:

• Domains: Business-aligned organization layer that groups data assets by functional area (for example, Marketing, or Finance) to improve discovery and stewardship

• Custom curation: Curators can customize and highlight specific assets in their organization or domain

• AI-powered recommendations: Surface popular and valuable datasets to enable curation at scale

• Unified discovery: Access tables, dashboards, Genie spaces, and more in one place

Domains are built on governed tags, allowing you to organize data assets in a way that’s best suited for your business needs.

Resilient PAT Management

Revoking a user’s permissions now preserves their Personal Access Tokens in an inactive state, allowing for easy recovery. 📖 Documentation

🖥️Platform

Python Unit Testing in the Workspace

Professional engineering comes to the notebook. You can discover and run pytest-based tests using a dedicated sidebar and inline UI. 📖 Documentation

Deploy Apps from Git

Streamline your app lifecycle by deploying Databricks Apps directly from Git repositories. 📖 Documentation

Assistant + MCP Servers

You can connect Databricks Assistant to Model Context Protocol (MCP) servers, allowing it to interact with local tools securely. 📖 Documentation

ADBC Default for Power BI

For faster data transfer, new Power BI connections now use the ADBC driver by default. 📖 Documentation

Lakehouse federation updates

Support added for the DATETIMEOFFSET data type in Synapse, and BigQuery table descriptions are exposed as table comments.

Google Sheets Pivot Tables

Business users can create pivot tables in Google Sheets directly from Databricks tables. 📖 Documentation

📊AIBI Dashboard

• Unity Catalog metric view lineage: Dashboard info shows Unity Catalog metric views in the lineage section.

• Dashboard refresh cancellation: Cancelling a dashboard refresh shows the last available data instead of an error.

• DD-MM-YYYY date format: The DD-MM-YYYY date format is available for date fields.

• Date range parameter “All” selection: Date range parameters support All. When All is selected, the parameter value is null and all rows are returned. 📖 Documentation

• Genie agentic experience (Public Preview): Genie for dashboard insights offers an agentic experience.

• Pivot table hierarchies: You can expand and collapse hierarchies in pivot tables.

• Stacked bar chart labels: Stacked bar charts support showing min/max label values and total labels.

• Area and line chart sorting: Area and line charts support sorting axes by measures.

• Dashboard content search: Global search in Lakehouse and Databricks One now matches on dashboard content, not just titles.

• Workspace admin theme permissions: Fixed a bug where workspace admins couldn’t edit workspace-level themes.

• Level of detail calculations: Level of detail calculations now support EXCLUDE syntax on multiple fields. 📖 Documentation

• Color legend sorting: You can sort color legends by another field in visualizations.

• Pivot table cell styles: Pivot tables support HTML, JSON, and image cell styles.

• Pivot table cross filtering: Pivot tables support cross filtering.

• Table resizing: You can resize table and pivot table headers and columns.

• Histogram and funnel chart labels: Histogram and funnel charts can show “total” labels.

• Assistant for dashboard authoring improvements: Assistant for dashboard authoring supports creating labels and setting color encoding values.

• Multi-select parameter enhancement: Multi-select parameters support selecting all values. 📖 Documentation

• Dashboard grid expansion: Dashboard layouts in the canvas support 12 columns across instead of 6, providing more flexibility for widget placement.

• Parameterized dashboard tasks: Dashboard tasks in Jobs can be parameterized, allowing job parameters to dynamically set dashboard filter values at runtime.

• Workspace-level dashboard themes: Dashboard themes can be configured at the workspace level. 📖 Documentation

• Top-aligned text widgets: Text widgets default to top-aligned text.

• Refresh-only schedules for individual data permissions: Authors and viewers of dashboards published with individual data permissions can create refresh-only schedules that trigger cache refreshes without sending notifications, improving dashboard load times. 📖 Documentation

• Tag search: Tag search is supported for dashboards, allowing users to find dashboards by their assigned tags.

• Assistant for dashboard authoring: Authors can automate multi-step dashboard workflows using natural language prompts to create datasets, visualizations, layouts, and filters. 📖 Documentation

📝AIBI Genie

• Metric view display names: Genie uses metric view display names and display formats for query result tables.

• Automatic JOIN relationships: Creating and updating Genie spaces using the API automatically creates JOIN relationships based on the primary and foreign keys defined in source tables.

• Context tab for space consumers: A new Context tab is available for Genie space consumers with CAN VIEW or CAN RUN permissions. Consumers can view the space description and common questions at any point during their Genie interactions. 📖 Documentation

• Agent mode thinking traces: Agent mode streams thinking traces in-line for improved readability.

• Agent mode parallel queries: Agent mode runs queries in parallel for faster results.

• Research Agent renamed to Agent mode (Beta): The rename better aligns with the feature’s function. 📖 Documentation

• Improved Agent mode (Beta) responses: Agent mode produces more concise and faster responses for “what” type questions through improved system prompting.

• Heatmap and sankey visualizations: You can generate heatmap and sankey visualizations in Genie.

• Query tagging for benchmarks: In query history, Genie attaches query tags to queries generated from benchmarks, helping you differentiate workloads from end users versus benchmarks.

• Auto-generated space titles: Space titles automatically generate based on the context of selected tables.

• Persistent knowledge store suggestions: Knowledge store suggestions persist on messages so you can navigate to other tabs within Genie before returning and accepting suggestions.

• Knowledge store suggestions on download: Genie suggests adding query results to the knowledge store when you download them.

• Inspect mode: It automatically improves standard Genie’s accuracy by reviewing the initially generated SQL, authoring smaller SQL statements to verify specific aspects of the query, and generating improved SQL as needed. 📖 Documentation

• Increased table limit: The table limit for Genie spaces has been increased from 25 to 30 tables.

• Embedded compute credentials: Compute credentials are embedded for all users of a Genie space. These credentials are embedded when a Genie space author changes the attached warehouse in the Genie space.

• Tag search: Tag search is supported for Genie spaces, allowing users to find spaces by their assigned tags.

Featured articles

• Introducing Permissions Analysis for Databricks resources with the Security Analysis Tool (SAT)

• Databricks Unity Catalog Tags: Governance Best Practices & Patterns

• Fine grained access Control at scale with ABAC

1. Databricks Lakebase (Generally Available) Offers serverless autoscaling, instant database branching, and unified governance in a single, enterprise-ready platform that simplifies data management and testing.

2. Managed connectors support row filtering and incremental formula field ingestion to cut costs and duplication.

3. Store and query Mlflow traces in Unity Catalog

Subscribe now

✨Tips and Tricks

• DQX: simplified Data Quality checking at scale for Pyspark Workloads on Streaming and standard dataframes.

• AI Dev Kit gives your AI Coding Assistant the trusted sources it needs to build faster and smarter on Databricks

🛠️Data Engineering

Row filtering is available for managed ingestion connectors

Lakeflow connector does allow row filtering for managed connectors to minimize data duplication. You can use the where clause to ingest only the data you need

📖 Documentation

Ingest Salesforce formula fields incrementally

You can enable incremental formula field ingestion which improve the performance and reduce costs. 📖 Documentation

Google Drive Connector

The Connector allows you to ingest Google Drive files into Databricks. You can use read_files, spark.read, COPY INTO, and Auto Loader to create Spark DataFrames, materialized views, and streaming tables, enabling you to build custom pipelines for common file ingestion use cases. 📖 Documentation

Metric views enhancements (DBR 18+)

You can use SQL window functions in metric views to calculate running totals, rankings, and other window-based calculations. 📖 Documentation

You can use the FILTER clause with measure aggregate functions in metric views to define per-aggregate filters when referencing metric view measures.

Structured Streaming enhancements (DBR 18+)

You can change the number of shuffle partitions in stateless streaming queries without restarting the query. 📖 Documentation

Adaptive Query Execution (AQE) and auto-optimized shuffle (AOS) are supported in stateless streaming queries. 📖 Documentation

SQL enhancements (DBR 18+)

The ability to coalesce sequential string literals such as 'Hello' ' World' into 'Hello World' has been expanded from expressions to any place string literals are allowed.

You can use named (:param) and unnamed (?) parameter markers virtually anywhere a literal value of the appropriate type can be used. This includes parameter markers in DDL statements such as CREATE VIEW v AS SELECT ? AS c1, column types DECIMAL(:p, :s), or COMMENT ON t IS :comment.

The reach of the IDENTIFIER clause, which casts strings to SQL object names, has been expanded to nearly everywhere an identifier is permitted. Along with the improvements to Literal string coalescing a parameter markers you can now parameterize anything from column aliases (AS IDENTIFIER(:name)) to column definitions (IDENTIFIER(:pk) BIGINT NOT NULL).

🖥️ Platform

Automatic email notification for expiring Personal Access Tokens

By default the maximum lifetime of new tokens is 730 days. Set a shorter maximum token lifetime in your workspace using the Databricks CLI or the Workspace configuration API.📖 Documentation

Custom base environments for Serverless Jobs

Serverless jobs support custom base environments defined with YAML files for Python, Python wheel, and notebook tasks. For notebook tasks, you can either select a custom base environment in the job’s environment configuration or use the notebook’s own environment settings, which support both workspace environments and custom base environments.

Serverless Egress IPs available via public json endpoint

You can retrieve serverless egress IP addresses from a public JSON endpoint.

📖 Documentation

Front-end PrivateLink for performance-intensive services

You can use AWS front-end PrivateLink for private connectivity to performance-intensive services like Zerobus Ingest and Lakebase Autoscaling. 📖 Documentation

Serverless Workspaces are Generally Available

Serverless workspaces come pre-configured with serverless compute and default storage, providing a fully-managed, enterprise-ready SaaS experience. 📖 Documentation

Dev Experience

• You can paste images from your local file system and past them into markdown cells using ⌘ + V (mac) or Ctrl + V (Windows).

• You can restore your recent workspace tab sessions

Trigger on update for pipelines

You can use trigger on update in Databricks SQL when creating a pipeline schedule to refresh the pipeline when a source table changes.

Lakebase and Apps

Databricks Lakebase is Generally Available

Key capabilities available today include:
💯 Serverless autoscaling and scale to zero: Compute resources dynamically adjust to match traffic spikes and shut off completely when idle to eliminate wasted costs.
💯 Instant database branching: Create isolated, zero-copy clones of production data in seconds for risk-free testing and development.
💯 Point-in-time recovery (PITR): Protect against accidental deletions or bugs with millisecond-level restoration.
💯 Unified governance: Manage access control and auditing through Unity Catalog for a single security model across your entire data estate.
💯 Sync tables: Keep your operational data and historical lakehouse context in sync without maintaining fragile pipelines.

Unified Lakebase interface

Lakebase Provisioned instances are accessible through the Lakebase App. The new unified interface consolidates Lakebase Provisioned and Lakebase Autoscaling management in one location, replacing the previous workflow of navigating to the Compute tab in the Lakehouse UI

Compute sizing for Databricks is generally available

You can choose between Medium (2 vCPUs, 6 GB) and Large (4 vCPUs, 12 GB) compute sizes to match your workload requirements. More features are integrations are coming soon. Stay tuned

🤖GenAI & ML

Agent Bricks is Generally Available

Agent Bricks provides a streamlined approach to operationalize data into production-grade AI agents. Use Agent Bricks: Knowledge Assistant to create a chatbot that can answer questions about your documents and provide high-quality responses with citations.

Store and query Mlflow traces in Unity Catalog

You can store MLflow traces in Unity Catalog tables using OpenTelemetry format and query them using Databricks SQL. This provides several benefits:

• Store unlimited traces in Delta tables for long-term retention and analysis

• Query trace data directly using SQL through a Databricks SQL warehouse

• Manage access control through Unity Catalog schema and table permissions

• Ensure compatibility with other OpenTelemetry clients and tools

📖 Documentation

Managed MCP Servers

Managed MCP servers allow your AI agents to securely connect to Databricks resources and external APIs. 📖 Documentation

📝AIBI Genie

• Catalog permission warnings: Users see an informative banner when they lack USE CATALOG permissions on attached tables, rather than encountering errors at SQL execution time.

• Unity Catalog lineage for Genie spaces: Genie spaces are discoverable in Unity Catalog lineage.

• Bulk actions for entity matching: Users can make bulk actions for entity matching (add, remove, and refresh).

• The value sampling features in Genie have been renamed to better describe their function:

  • Prompt matching => previously value sampling

  • Format assistance => previously example values

  • Entity matching => previously value dictionaries

• Sample data tab: View sample data to understand the context behind a Genie space. 📖 Documentation

• Text answer summaries in API: Text answer summaries are returned in the API.

• Research Agent starter questions : Curated starter questions are available for Research Agent mode.

• Landing page redesign: The landing page has been redesigned to improve starter question readability.

• Benchmark execution: Benchmarks with AI functions execute correctly.

• Space manager feedback management: Space managers can update thumbs up/down feedback from other users and re-run query results from other users’ messages using their own data credentials to debug feedback.

• Example SQL title character limit: The maximum character length for example SQL titles has been increased to 1024.

• PDF export for Research Agent (Beta) reports: Export Genie Research Agent reports as PDF files. 📖 Documentation

• SQL example citations: Genie’s analysis steps cite the SQL examples used.

• Improved question type handling: Genie categorizes common question types (top K, percentage, and distribution) more accurately so the right context is applied, improving results. 📖 Documentation

📊AIBI Dashboard

• Full-width pivot tables: Widgets render pivot tables at full width for better readability.

• Tooltip encodings on charts: Combo and dual-axis charts support tooltip encodings.

• Image rendering in pivot tables: Cells can display images directly in pivot tables.

• Dark mode scrollbar visibility: Scrollbars are easier to see when using dark mode.

• Increased pivot data download limits: Download limits now support up to 10,000 rows by 1,000 columns for pivoted data.

• Account-level user subscriptions: Dashboard authors can now subscribe account-level users to scheduled dashboard updates. 📖 Documentation

• Custom calculation references: Custom calculations can reference custom calculations on dashboard datasets.

• Cumulative totals on stacked bar charts: Stacked bar charts can display the cumulative total of values.

• Improved tooltips on combo and dual axis charts: Tooltips are now easier to access when hovering over combo and dual axis charts.

• Dashboard page navigation scroll bar: The horizontal scroll bar to navigate between dashboard pages is thicker.

• Dashboard title character limit: Dashboard titles have more horizontal space before truncating.

• Pivot table header height: Pivot table header height can be configured to enable wrapping for multi-row heights.

• Text widget markdown tables: Text widgets support markdown table syntax.

• Parameterized default catalog and schemas: Parameterize the default catalog and schemas when deploying dashboards using Databricks Asset Bundles. 📖 Documentation

• Multi-select filter paste enhancement: Paste comma-separated and newline-separated values into multi-select filters.

• Pivot table total rows: Pivot tables can display total rows at the top.

• Dashboard widget drag handles: Dashboard widgets have drag handles for resizing and repositioning.

• Pivoted table Excel downloads: Download dashboard data to Excel in pivoted format rather than flattened.📖 Documentation

• Timezone display fixes: Fixed a number of timezone-related issues, ensuring that displayed timezones match underlying data timezones instead of displaying UTC.

• Selective data point labels: Labels can be shown only for first/last or min/max data points.

📊Databricks One

• Databricks Apps filtering: Databricks One consumers can filter to include or exclude inactive Databricks apps when browsing. Inactive apps are those with a status other than Running, such as Stopped, Deploying, or Crashed. 📖 Documentation

• Dashboard widget title search: Search supports dashboard widget titles.

• Enhanced search rankings: Certified and favorite content is boosted in search rankings in Databricks One.

• Recents and favorites in discovery: Recents and favorites are surfaced in the For you page.

• Mark assets as favorites: Click the star icon next to asset names in Databricks One to mark them as favorites.

• Favorite and certified content filtering: Filter content by favorite and certified status in Databricks One. 📖 Documentation

🛡️Governance and data sharing

Shared isolation execution environment for Unity Catalog Python UDFs

Unity Catalog Python UDFs with the same owner can share an isolation environment by default. This can improve performance and reduce memory usage by reducing the number of separate environments that need to be launched.

To ensure a UDF always runs in a fully isolated environment, add the STRICT ISOLATION characteristic clause. 📖 Documentation

Featured articles

• 10 Things you no longer need to think about

• Zerobus ingest on Databricks

• Building a research paper curator for knowledge assistants

• From AIBI dashboards to metrics views

We strongly recommend you to subscribe to the AIBI Newsletter

Subscribe now

• New Lakeflow connectors (MySQL, Meta Ads, Confluence, PostgreSQL, SharePoint, NetSuite) are available

• Lakebase Autoscaling on AWS brings autoscaling compute, scale‑to‑zero, branching, instant restore, readable secondaries, ACL‑based project control, richer metrics, and new Databricks Apps resources like MLflow experiments, vector search indexes, UDFs, and UC connections.

• New hosted LLMs (OpenAI GPT‑5.2, Claude Haiku 4.5, Gemini 3 Flash) are available across APIs, reasoning/vision endpoints, and batch jobs, plus Vector Search Reranker is GA to boost retrieval relevance.

Happy New Year 2026

✨Tips and Tricks

• The databricks-labs-pytester package is a valuable utility for orchestrating integration tests with Databricks, enhancing pytest by providing native Databricks capabilities for resource management, Spark session handling and test isolation.

• You can use Kasal which is an interactive, low-code way to build and deploy AI Agents on the Databricks platform.

🛠️Data Engineering

Lakeflow updates

• Databricks released many connectors this month: MySQL connector, Meta Ads connector, Confluence connector, PostgreSQL connector, Customizable SharePoint connector, NetSuite connector

• You can change owner for Materialized Views and Streaming Tables defined in Databricks SQL.📖 Documentation

• ForEachBatch for Lakeflow Spark Declarative Pipelines is available. It helps processing streams as a series of micro-batches in Python. 📖 Documentation

• New columns are available in lakeflow system tables:

Autoloader improvement

Autoloader can discover files with the efficiency of notification while retaining the simple setup of directory listing.

Excel format support on Databricks

Databricks provides built in support for reading Excel files. 📖 Documentation

Streaming improvements (DBR18+)

Adaptive Query Execution (AQE) and auto-optimized shuffle (AOS) are now supported in stateless streaming queries. 📖Documentation.

You can change the number of shuffle partitions in stateless streaming queries without restarting the query.

Parameters markers everywhere (DBR18+)

You can use named (:param) and unnamed (?) parameter markers virtually anywhere a literal value of the appropriate type can be used. This includes parameter markers in DDL statements

🖥️Platform

The Databricks Assistant Agent Mode preview is enabled by default for most customers.

You can disable DBFS root and mounts

Once your file-based workflows migration to Unity Catalog volumes is completed you can prevent users from uploading modifying or accessing data in DBFS root and DBFS mount. 📖 Documentation

Flexible nodes type is a game changer

Flexible node types allow your compute resource to fall back to alternative, compatible instance types when your specified instance type is unavailable. 📖 Documentation

Context based ingress control is available

This feature allows account admins to define allow and deny rules that consider who is making a request, where the request originates, and what resources it targets within Databricks. With context-based ingress control, only trusted combinations of identity, request type, and network source can access your workspace. A single policy can protect multiple workspaces, ensuring consistent and centralized access enforcement across your organization.

Single use refresh tokens for OAuth applications

You can configure single-use refresh tokens for OAuth applications integrated with Databricks. This security feature requires token rotation after each use, enhancing protection for user-to-machine authentication flows. 📖Documentation

Lakebase and Apps

Lakebase (Autoscaling) is available on AWS

This new version of Lakebase introduces autoscaling compute, scale-to-zero, database branching, instant restore, and a redesigned project-based interface📖 Documentation

Run Read-Only Queries on Lakebase Readable Secondaries in Databricks SQL Editor

You can connect to Lakebase readable secondaries and run read-only queries from the Databricks SQL editor

ACL support for Lakebase AutoScaling

Lakebase Autoscaling supports Access Control Lists (ACLs). Grant CAN CREATE or CAN MANAGE permissions to control who can access and manage project resources. Manage permissions from project settings in the Lakebase App 📖 Documentation

Connect to Lakebase Autoscaling from the SQL Editor with r/w access

Lakebase Autoscaling metrics dashboard

New resources types for Databricks Apps

You can add mlflow experiments, Vector Search Index, UDF, and UC Connections as resources in Databricks Apps. 📖 Documentation

🤖GenAI & ML

New available Models on Databricks

OpenAI GPT-5.2, Claude Haiku 4.5, Gemini 3 flash are available as a hosted models allowing you to access them through:

• Foundation Model APIs in pay‑per‑token mode

• Query reasoning model endpoints

• Query vision model endpoints

• Batch inference workloads using AI Functions

Vector Search Reranker is Generally Available

Reranking is a technique that improves retrieval quality by evaluating the retrieved documents to identify the ones that are semantically most relevant. 📖 Documentation

📝AIBI Genie

• Numeric column formatting: Numeric columns in tables are now formatted more effectively, including IDs, ZIP codes, and currency.

• Value dictionary indexing improvements: Improved value dictionary indexing for better coverage and recall on user prompts.

• Service Principal monitoring display: The user column in the Monitoring page now shows the Service Principal ID when a service principal is used

• Genie Research Agent (Beta) architecture update: Research Agent has a single reasoning agent that generates hypotheses and SQL.

• Improved clarifying questions: Genie’s ability to ask clarifying questions in cases of semantic ambiguity in the user’s prompt has been improved.

• Streamlined Research Agent: The Research Agent experience has been streamlined by simplifying the thinking trace and moving hypothesis evaluations into a sidebar for easier review.

• Benchmark error explanations: Benchmarks include an LLM explanation of what was incorrect in the Genie generated answer.

• Better instruction following: Genie effectively follows text instructions on how to format answer summaries to common questions. 📖 Documentation

Subscribe now

📊AIBI Dashboard

Databricks AIBI Caching explained

For more information read the medium article by Andrea Picasso.

• Sort by hidden measure fields: Users can sort charts by a different measure field that does not visually appear on the bar chart. 📖 Documentation

• Map legend position control: Authors can change map legend positions.

• Counter visualization text: Removed automatic “vs” text from counter visualizations.

• Range filter spacing: Added spacing between minimum and maximum values in range filters when the slider bar is hidden.

• Dashboard subscriptions to Microsoft Teams channels: AI/BI dashboards support sending scheduled snapshots to Microsoft Teams channels. Teams can receive dashboard updates directly in their Teams workspace, including a PNG image snapshot, a direct link to open the dashboard in Databricks, and a PDF attachment.📖 Documentation

• Individual series labels: You can enable labels for individual series, instead of all series on the dashboard at once.

🛡️Governance and data sharing

Token expiration policy for Delta Sharing

All new Delta Sharing open sharing recipient tokens are issued with a maximum expiration of one year from the date of creation. Tokens with an expiration period longer than one year or no expiration date can no longer be created.

Delta Sharing to external Iceberg Clients

You can share tables, materialized views, and streaming tables to external Iceberg clients such as Snowflake, Trino, Flink, and Spark. 📖 Documentation

Update request parameters for Delta Sharing recipient audit log events

For Delta Sharing recipients, deltaSharingProxy* audit log events now also include the catalog_name request parameter, in addition to share_name. 📖 Documentation

Featured articles

• Beyond ADLS Limits: File Arrival Triggers for File Updates

• AI Newsletter Series (Part 1): Parsing Enterprise PDFs Into AI-Ready Data

• Fine-Grained Access Control in Delta Sharing: Dynamic Views vs Partition Filtering

• Integration Testing for Lakeflow Jobs with Pytest and Databricks Connect

• You can run Scala or Java JAR tasks on serverless compute as long as you build a Databricks‑compatible JAR using the right JDK/Scala versions and Databricks Connect.​

• The new MCP Servers tab centralizes management of managed and external MCP servers and lets you install curated servers from Databricks Marketplace.​

• Genie’s Research Agent adds multi‑step reasoning over Genie spaces to answer complex business questions.

• Attribute‑based access control (ABAC) in Unity Catalog is in public preview and ABAC‑secured assets can be shared like standard tables.​

• The Databricks SQL Alerts UI has been revamped to streamline how you define queries, conditions, schedules, and notifications.

The TRUTH About Product Management & AI’s Future With David Meyer Databricks SVP

David Meyer is the Senior Vice President (SVP) of Products at Databricks, where he leads product strategy and execution for the company. In this role, Meyer oversees the development and direction of Databricks’ core offerings, including its Data Intelligence Platform, the Lakehouse architecture, and AI-driven features.

✨Tips and Tricks

• Did you know you can use the inline Databricks Assistant with Cmd+I to iterate on just the code below your cursor (or a highlighted block) without rewriting the whole cell? This speeds up micro-edits and refactors.

• For better results, @-mention tables and keep Unity Catalog descriptions and column comments up to date; the Assistant uses metadata context to generate more accurate SQL and code examples.

• The Assistant has a “Find tables related …” action that puts it in table search mode; selecting “Query in natural language” explicitly binds the next query to your chosen table for precise answers

🛠️Data Engineering

You can run JAR Tasks on Serverless Compute

Use JAR task to deploy Scala or Java code compiled into JAR. Make sure to create a Databricks compiled JAR. 📖 Documentation

Lakeflow updates

• Lakeflow Declarative Pipelines has been renamed to Lakeflow Spark Declarative Pipelines.

• SFTP Connector extends autoloader features to ingest files from SFTP servers.

  📖 Documentation

• You can schedule when full refresh snapshots occur and enable automatic full refresh to recover from unsupported schema changes. This applies to managed database connectors like SQL Server in Lakeflow Connect. 📖 Documentation

Convert Foreign tables to managed tables

You can convert foreign tables from external catalogs (such as Hive Metastore or AWS Glue) to Unity Catalog managed while retaining table history and configurations. Use ALTER TABLE SET MANAGED to convert to managed tables with an optional DRY RUN validation. 📖 Documentation

🖥️Platform

You can disable DBFS root and mounts

Once your file-based workflows migration to Unity Catalog volumes is completed you can prevent users from uploading modifying or accessing data in DBFS root and DBFS mount. 📖 Documentation

Audit Logs for request-for access events are emitted

Improved experience with tag search

You can search by tag and value => tag:<tag_key>:<tag_value>

JDBC Unity Catalog connection

You can use a JDBC Unity Catalog connection to read and write to a data source with the Spark Data Source API or Databricks Remote Query SQL API. 📖 Documentation

Why should I use it ?

• Governed access to data sources using Unity Catalog connections

• Create the connection one time and reuse it across any Unity Catalog compute type: serverless, standard clusters, dedicated clusters, and Databricks SQL

• Stable for Spark and compute upgrades

• Connection credentials are hidden from the querying user

New jobs triggers available

Table update triggers for jobs support views as a source. 📖 Documentation

Automatic email reminders for expiring personal access tokens

Databricks sends automatic email notifications to workspace users approximately 7 days before their personal access tokens expire. 📖 Documentation

You have more flexibility to size Databricks Apps

You can configure the compute size to control CPU, memory and cost based on your workloads requirements. 📖 Documentation

Git CLI Commands in Git folders

You can run standard Git commands directly from the Databricks web terminal in Git folders. 📖 Documentation

Improved collaboration in notebook cells, files and SQL Editor

Real-time collaboration is available in notebook cells, files, and the new SQL editor. Multiple users can edit the same cell simultaneously and can view each other’s edits. 📖 Documentation

🤖GenAI & ML

Function calling and structured output is available on Qwen Next instruct Model

Improvements on the MCP Server

The new MCP Servers tab in the workspace shows your managed and external MCP servers, and a curated list of MCP servers available to install from Databricks Marketplace. 📖 Documentation

The Marketplace includes listing MCP Servers

New available Models on Databricks

OpenAI GPT-5.1 and Google Gemini Pro 3 are available as a hosted models allowing you to access them through:

• Foundation Model APIs in pay‑per‑token mode

• Query reasoning model endpoints

• Query vision model endpoints

• Batch inference workloads using AI Functions

📝AIBI Genie

• You can connect Genie Spaces to Microsoft Copilot. 📖 Documentation

• Increased SQL snippets and JOIN relationship limit: The limit for SQL snippets and JOIN relationships in a Genie space has been increased to 200.

• Research Agent starter questions: Research Agent displays starter questions to help users begin their data exploration.

• Auto-suggested benchmarks: Genie automatically suggests benchmarks based on the space context to help authors build evaluation sets.

• Genie Create and Update APIs: Genie Create and Update APIs have been released. Get API has been updated to allow users to retrieve a serialized definition of the Genie space. 📖 Documentation

• Research Agent in Beta: Research Agent extends Genie’s capabilities to help you uncover deeper insights and tackle complex business questions using multi-step reasoning and hypothesis testing. 📖 Documentation

• Improved column selection: Genie optimizes column selection to provide more columns with higher relevance while efficiently managing context consumption.

• Text explanations for empty results: Genie includes text explanations of query results when query results are empty.

• Improved auto-generated charts: Visualizations use query result metadata for higher-quality auto-generated charts.

• Improved date formatting: The process for formatting date results in visualizations has been improved.

📊AIBI Dashboard

Databricks AIBI Caching explained

For more information read the medium article by Andrea Picasso.

• Sort by hidden measure fields: Users can now sort charts by a different measure field that does not visually appear on the bar chart. 📖 Documentation

• Counter visualization improvements: You can use counters to track and compare metrics over time and show sparklines. To create a counter with a sparkline, select a main value and a date column. 📖 Documentation

• Configure filter application behavior: Dashboards support applying multiple filters at the same time alongside the default instant application. 📖 Documentation

• Draft dashboards no longer displayed: Only published dashboards are visible in Databricks One.

• Enhanced search functionality: Users can search over dashboard names, page names, widget titles and descriptions, dataset names, and their queries.

• Ask Genie button for embedded dashboards: When viewers access dashboards configured with basic embedding, they can access the Ask Genie button for dashboards published with an associated Genie space.

• Per-widget filter enhancements: Per-widget filters support date ranges, relative date ranges, and number filters.

• Metric view metadata support: AI/BI dashboards consume Unity Catalog metadata from metric views, including formatting and display name.

• Horizontal chart scrollbars on hover: Scrollbars for horizontal charts now show on hover.

🛡️Governance and data sharing

Attribute-based access control is now public preview

SAP BDC Connector for Databricks supports Cross cloud sharing

You can share data between different cloud environments using the SAP Business Data Cloud (BDC) Connector for Databricks.

ABAC secured assets can be shared

You can share a table or schema secured by ABAC policies like standard tables.

Delta Sharing views permitted in Serverless network policies

Delta Sharing views are permitted when using serverless network policies, regardless of storage location alignment between shared tables and view dependencies. 📖 Documentation

🔍 Data Warehousing

The DBSQL Alerts experience has been revamped. 📖 Documentation

Featured articles

• How to perform Semantic Search in Databricks Lakebase

• Revolutionizing GenAI Application Management: MLflow 3 Prompt Registry in Databricks

• How to Deploy Metric Views with DABs

• Operationalize your lakehouse: Lakebase for low-latency apps and api

• Mosaic AI Model Serving adds support for GPT‑5, Claude Sonnet 4.5, Qwen3 Next, Gemini 2.5 Pro, and multimodal inputs, with autoscaling improvements and prompt caching for Claude models.

• Databricks introduces Context‑based ingress control and AI‑powered data classification to enhance security and compliance governance.

• AIBI Genie and Dashboards receive major updates including SQL result summaries, benchmark enhancements, top/bottom N chart categories, and waterfall visualizations.

• Serverless OCR is now available through ai_parse_document!

• Zerobus now available (stream record by record to your UC Tables!)

• UC Compatibility Mode now enables external tools like Snowflake, Athena, and Microsoft Fabric to read Unity Catalog managed tables and materialized views efficiently.

Building the future of AI: Classic ML to GenAI with Patrick Wendell Databricks Co-Founder

Want to know more about the AI market, Databricks MosaicAI and how to build AI products like Agent Bricks in a massive organization like Databricks? Listen to Patrick Wendell, Databricks CoFounder share his AI insights!

🛠️Data Engineering

Convert to Unity Catalog managed table for external table

Check out this quick video to discover how to migrate External Table to UC Managed table!

Smaller table, faster queries! Zstd is now the default compression for new Delta Tables

All newly created Delta tables in Databricks Runtime 16.0 and above use Zstandard compression by default instead of Snappy.

Existing tables continue to use their current compression codec. To change the compression codec for an existing table, set the delta.parquet.compression.codec table property

Access UC Managed table with other engine through Compatibility Mode

UC Compatibility Mode generates a read-only version of a Unity Catalog managed table, streaming table, or materialized view that is automatically synced with the original table. This enables external Delta Lake and Iceberg clients, such as Amazon Athena, Snowflake, and Microsoft Fabric to read your tables and views without sacrificing performance on Databricks. Save the data in UC, read it from other engines! 📖 Documentation

OCR made simple with ai_parse_document

The ai_parse_document enables users to extract structured content and contextual layout metadata from unstructured documents, such as PDFs and images (JPEG, PNG), using state-of-the-art generative AI models provided through Mosaic AI Model Serving APIs. 📖 Documentation

Zerobus Ingest connector in Lakeflow Connect is available

One of the most Exciting release! Having kafka queue or similar system to ingest data into your UC tables? Zerobus Ingest provides a fully manage experience through API!

The Zerobus Ingest connector enables record-by-record data ingestion directly into Delta tables through a gRPC API (more APIs to come!). This serverless connector operates at any scale and streamlines ingestion workflows by eliminating the need for message bus infrastructure and Delta-specific dependencies. 📖 Documentation

input_file_name is no longer supported

Prefer the new _metadata.file_name!

Auto Loader incremental listing default changed

cloudFiles.useIncrementalListing option has changed from auto to false

Support MV/ST refresh information in Describe extended as Json

🪄Serverless

Version 17.3

• LIMIT ALL support for recursive CTEs: You can now use the LIMIT ALL clause with recursive common table expressions (rCTEs) to explicitly specify that no row limit should be applied to the query results.📖 Documentation

• Appending to files in Unity Catalog volumes returns correct error: Attempting to append to existing files in Unity Catalog volumes now returns a more descriptive error message to help you understand and resolve the issue.

• st_dump function support: You can now use the st_dump function to decompose a geometry object into its constituent parts, returning a set of simpler geometries.

• Polygon interior ring functions are supported:

  • st_numinteriorrings: Get the number of inner boundaries (rings) of a polygon.

  • st_interiorringn: Extract the n-th inner boundary of a polygon and return it as a linestring.

• EXECUTE IMMEDIATE using constant expressions: The EXECUTE IMMEDIATE statement now supports using constant expressions in the query string, allowing for more flexible dynamic SQL execution. 📖 Documentation

• Allow spark.sql.files.maxPartitionBytes in serverless compute: You can now configure the spark.sql.files.maxPartitionBytes Spark configuration parameter on serverless compute to control the maximum number of bytes to pack into a single partition when reading files. 📖 Documentation

🖥️Platform

New permissions for Databricks Github App

Granting this permission will let Databricks retrieve and save your primary GitHub account email to your Linked Git credential in Databricks

Git email identity configuration for Git folders

You can specify a Git provider email address, separate from your username, when creating Git credentials for Databricks Git folders. This email is used as the Git author and committer identity for all commits made through Git folders, ensuring proper attribution in your Git provider and better integration with your Git account.

Notebook improvements

• The cell execution minimap now appears in the right margin of notebooks. Use the minimap to get a visual overview of your notebook’s run status and quickly navigate between cells. 📖 Documentation

• Use Databricks Assistant to help diagnose and fix environment errors, including library installation errors. 📖 Documentation

• When reconnecting to serverless notebooks, sessions are automatically restored with the notebook’s Python variables and Spark state. 📖 Documentation

• Databricks supports importing and exporting IPYNB notebooks up to 100 MB. Revision snapshot autosaving, manual saving, and cloning are supported for all notebooks up to 100 MB.

• When cloning and exporting notebooks, you can now choose whether to include cell outputs or not.

Billable usage track the performance mode

Billing logs record the performance mode of Serverless jobs and pipelines. The workload’s performance mode is logged in the product_features.performance_target column and can include values of PERFORMANCE_OPTIMIZED, STANDARD or NULL

Context based ingress control

Context-based ingress enables account admins to set allow and deny rules that combine who is calling, from where they are calling, and what they can reach in Databricks. Context-based ingress control ensures that only trusted combinations of identity, request type, and network source can reach your workspace. A single policy can govern multiple workspaces, ensuring consistent enforcement across your organization. 📖 Documentation

Lakeflow Jobs Updates

• Create backfill job runs to backfill data from the past. This is useful for loading older data, or repairing data when there are failures in processing

• Jobs can be triggered on source table update

• Unified runs list

  

Databricks Connector for Google Sheets offers additional features

Configure compute to AWS Capacity Blocks

You can use Databricks data to build canvas apps in Power Apps, flows in Power Automate and agents in Copilot Studio by creating a Databricks connection in Power Platform. 📖 Documentation

🤖GenAI & ML

Mosaic Models Serving updates

• Mosaic AI Model Serving supports Anthropic Claude Sonnet 4 for batch inference workflows

• Mosaic AI Model serving support OpenAI GPT 5 Models

• Alibaba Cloud Qwen3_Next instruct, Anthropic Claude Sonnet 4.5, Gemini 2.5 Pro and Flash Models are available on Pay-Per-Token

Prompt Caching is supported for Claude Models

Prompt caching is supported for Databricks-hosted Claude models. You can specify the cache_control parameter in your query requests to cache the following:

• Thinking messages content in the messages.content array.

• Images content blocks in the messages.content array.

• Tool use, results and definitions in the tools array.

Model Serving improvements

• Mosaic AI Model Serving supports multimodal inputs for Databricks hosted foundation models. 📖 Documentation

• Autoscaling in Mosaic AI Model Serving has been tuned to ignore extremely brief traffic surges and instead respond only to sustained increases in load. This change prevents unnecessary provisioned concurrency scaling during momentary bursts and reduces serving costs without impacting performance or reliability.

Data Classification powered by AI

Databricks offers advanced data classification capabilities powered by AI, making it easy to automatically identify, classify, and tag sensitive data in any table or dataset within Unity Catalog. AI-driven classification streamlines compliance, simplifies governance, and helps teams apply access control policies through features like attribute-based access control (ABAC)

SQL MCP Server is available

Databricks provides a SQL managed MCP server that allows AI agents to execute SQL queries directly against Unity Catalog tables using SQL warehouses. 📖 Documentation

Multi-Agent Supervisor supports Unity Catalog functions and external MCP Servers

Multi-Agent Supervisor to create a supervisor system that coordinates Genie Spaces, agent endpoints and tools to work together to complete complex tasks across different, specialized domains. You can provide the supervisor system tools like Unity Catalog functions and external MCP servers.

📝AIBI Genie

• Add tables and metric views to spaces: Users can add tables and metric views to a space and analyze them together.

• Benchmarks scoring improvement: Benchmarks treat extra rows or columns in Genie’s output as incorrect. 📖 Documentation

• Thinking steps in responses: Each Genie response now includes thinking steps that show how the prompt was interpreted, along with the tables and example SQL statements that were used. 📖 Documentation

• Easier benchmark generation:

  • Space editors can save representative messages or SQL answers from chat conversations to benchmark questions.📖 Documentation

  • Space editors can generate benchmark SQL answers after entering a benchmark text question. 📖 Documentation

  • Accurate Genie answers from one benchmark result can now be saved to update future benchmark sets. 📖 Documentation

• Improved performance and latency: Genie’s internal reasoning process has been simplified, slightly improving overall latency and streamlining context for SQL generation.

• JOIN relationship discovery improvements: You can sort JOIN relationships and SQL snippet definitions for easier management.

• API endpoints moving to Public Preview:list conversation messages, delete conversation message, and send thumbs up/down feedback.

• Benchmark result explanations: Benchmark results include an explanation for why a result was rated incorrect.

• SQL execution result summaries: Genie automatically generates summaries of SQL execution results to make responses more natural and easier to interpret.

• SQL measures in knowledge stores: Users can add SQL measures to a space’s knowledge store, along with filters and dimensions, to guide Genie in generating specific SQL expressions. 📖 Documentation

• Fixed permissions messaging: Messaging for users with CAN RUN or CAN VIEW permissions has been corrected to accurately reflect their capabilities in the monitoring and data tabs.

• Increased benchmark limit: The maximum number of benchmarks you can add to a Genie space has been raised from 100 to 500.

• Run benchmarks selectively: You can select a subset of benchmark questions to start a run from either the benchmark list page or from a previous benchmark result. 📖 Documentation

• Unity Catalog SQL functions as benchmark answers: You can add Unity Catalog SQL functions as gold standard answers to benchmarks for more comprehensive testing.

📊AIBI Dashboard

• Access for account-level users: Users without a workspace can access dashboards shared with them that were published without embedded credentials. 📖 Documentation

• Preview fields and measures for metric views: Authors can preview available fields and measures for metric views in the Add data dialog.

• Query insights in the dashboard dataset editor: Query insights are available in the dashboard dataset editor to help analyze query performance.

• Copy and paste widgets and pages: Dashboard widgets and pages can be copied and pasted between dashboards.

• Top or bottom N categories in bar charts: Bar charts can now display the top or bottom N categories. To edit, open the kebab menu for the categorical axis dimension and set the number under Default number of categories.

• Adjustable line thickness in line charts: You can control the thickness of lines in line charts. If no series are specified in the size area, the slider changes all line thickness uniformly. If series are specified, you can set different thickness values for each series, similar to different size circles in scatter charts.

• Copy link URLs from tables: You can copy link URLs directly from table visualizations.

• Additional predicate support for custom calculations: Custom calculations now support IN, BETWEEN, and pattern matching predicates (LIKE, ILIKE, RLIKE) for more flexible filtering and comparison operations. 📖 Documentation

• Dynamic text functions in pivot tables: Pivot tables support contains(), startswith(), and endswith() functions for conditional formatting.

  📖 Documentation

• Gridline toggle: Charts support toggling gridlines on and off.📖 Documentation

• Dashboard tags and certification: You can add tags to dashboards and Genie spaces to improve organization across your workspace.

• Drill-through without target filters: Drill-through filters any visualization based on the same dataset as the source selection without an explicit target filter. If a target filter exists, it is still applied. 📖 Documentation

• New functions for custom calculations: Custom calculations support over 170 functions. 📖 Documentation

• Drag-and-drop field reordering in visualizations: Authors can drag to reorder fields when creating some visualizations such as Pivot and Line charts.

• Waterfall visualization type: Waterfall charts are available for dashboards. 📖 Documentation

🛡️Governance

Certification status system tag

Delta Sharing recipients can apply row filters and column masks

Delta Sharing recipients can apply their own row filters and columns masks on shared tables and shared foreign tables.Delta Sharing providers still cannot share data assets that have row-level security or column masks.

🔍 Data Warehousing

You can define semantic metadata in a metric view. Semantic metadata helps AI tools such as Genie spaces and AI/BI dashboards interpret and use your data more effectively.

To use semantic metadata, your metric view must use YAML specification version 1.1 or higher and run on DBR 17.2 or above. 📖 Documentation

That’s it for all the latest news!

Don’t miss any update, don’t forget to subscribe!

• 5 new system tables are available

• SQL Server connector now supports SCD type 2

• Databricks Online Features stores powered by lakebase is available

🛠️Data Engineering

SQL Server Connector supports SCD type 2

History Tracking or SCD determines how to handle changes in the data over time. 📖 Documentation

Lakeflow Declarative pipelines improvements

Lakeflow Declarative Pipelines supports querying the event log for metrics about the progress of a stream

You can use Python custom data sources and sinks in Spark Declarative Pipelines.

The Perks of using Lakeflow Pipelines editor

• All-in-one experience: Build edit and debug ETL pipelines in one place. No more window switching

• Code-first workflows: Work in Python or SQL, across multiple files. Organize code just how you like it.

• Real-time previews: Get instant previews of your data and tables directly inside the editor.

• Easy debugging: Spot issues fast with execution insights, an issues panel, and direct navigation to errors in your code.

• Pipeline graphs: See your data pipeline visually

• Version control & scheduling: Integrate your pipeline with Git and run it on a schedule for peace of mind.

• Flexible runs: Test a single file, a single table, or the whole pipeline iterate quickly, deploy confidently.

• Default folder structure: Start organized, stay organized with pre-built folders for transformations, explorations, and utilities.

• Notifications: Get alerts when things go wrong (or right), right out of the box.

• Preview and monitor: New, improved monitoring for complete visibility into pipeline runs.

  

Bridging SAP BDC and Databricks: New connector delivers Seamless, secure data sharing

The SAP BDC Connector enables secure zero-copy data sharing between SAP BDC and a Unity Catalog-enabled Databricks workspace. You can access and analyze the data on Databricks and share Databricks data assets back.

Databricks Runtime 17.2

• The LIKE operator supports the UTF8_LCASE collation, allowing for case-insensitive matching.

• The EXPLAIN command and Spark UI for RemoteSparkConnectScan on dedicated compute now display predicates, aggregations, group by clauses, limits, and samples that are pushed down into the remote scan.

• You can declare multiple sessions or local variables of the same type and default value in a single DECLARE statement

• You can use the TEMPORARY keyword when creating a metric view.

• Fine-grained access control on dedicated compute supports the FSCK REPAIR TABLE and DESCRIBE DETAIL commands.

• INSERT REPLACE USING replaces rows when the USING columns compare equal under equality. INSERT REPLACE ON replaces rows when they match a user-defined condition.

• ANSI_MODE is enabled by default for Pandas API.

• Auto Loader infers partition columns from file paths when ingesting data as a semi-structured variant type using the singleVariantColumn option.

• Databricks includes user-defined environment settings in the DESCRIBE CONNECTION output for JDBC connections that support custom drivers and run in isolation. Other connection types remain unchanged.

• You can truncate uniform history when migrating tables with Uniform/Iceberg enabled using ALTER TABLE...SET MANAGED. This simplifies migrations and reduces downtime compared to disabling and re-enabling Uniform manually.

• You can specify Delta read options for SQL-based streaming queries using the WITH clause.

• SELECT * FROM STREAM tbl WITH (SKIPCHANGECOMMITS=true, STARTINGVERSION=X);

• Databricks shares the execution environment for Python user-defined table functions (UDTFs) from the same owner and Spark session. An optional STRICT ISOLATION clause is available to disable sharing for UDTFs with side effects

• Databricks enables row tracking by default for all newly created Unity Catalog managed tables. Existing tables are not affected and maintain their current row tracking settings.

🖥️Platform

Serverless jobs and pipelines configured in the UI are now performance optimized by default.

Tables backed by default storage can be shared to any recipient.

Databricks Connector in Microsoft Power Platform

You can use Databricks data to build canvas apps in Power Apps, flows in Power Automate and agents in Copilot Studio by creating a Databricks connection in Power Platform. 📖 Documentation

Databricks Apps support for Genie resources

Databricks Apps supports adding an AI/BI Genie space as an app resource to enable natural language querying over curated datasets. 📖 Documentation

Mount Delta Shares to an existing Shared catalog

Delta Sharing recipients can mount shares received from their Delta Sharing provider to an existing shared catalog. 📖 Documentation

Delta Sharing on Lakehouse federation is available

You can use Delta Sharing to share foreign schemas and tables created with query federation in Databricks-to-Databricks sharing and open sharing.

Databricks One is available for everyone

Explore table data using an LLM

You can ask natural language questions about sample data using Catalog Explorer. The Assistant uses metadata context and table usage patterns to generate a SQL query. You can then validate the query and run it against the underlying table.

Requirements for the Salesforce ingestion

In early September 2025, Salesforce began restricting the use of uninstalled connected apps. This restriction does not break existing Unity Catalog connections to Salesforce but it prevents you from creating a connection to a new Salesforce instance without the Databricks connected app installed.

System tables updates

system.lakeflow.pipeline_update_timeline provides complete historical tracking of pipeline updates and supports fine-grained analysis of update activity, triggers, results and compute usage.

system.data_classification.results: stores detections for sensitive data classes at the column level across enabled catalogs in your metastore.

system.mlflow. experiments_latest: records experiment names and soft-deletion events. This data is similar to the experiments page in the MLflow UI.

system.mflow.run_metrics_history: records the name, value, timestamp, and step of all metrics logged on runs, which can be used to plot detailed timeseries from runs

system.mlflow.runs_latest: records run-lifecycle information, the params and tags associated with each run, and aggregated stats of min, max, and latest values of all metrics.

🤖GenAI & ML

AI Agents: Authorize on behalf of user

AI agents deployed to Model Serving endpoints can use on-behalf-of-user authorization. This lets an agent act as the Databricks user who runs the query for added security and fine-grained access to sensitive data.

Databricks Assistant Agent Mode: Data Science Agent

Mosaic AI Agent framework supports automatic authentication passthrough for Lakebase resources. 📖 Documentation

Databricks Online Features stores is available

Feature Stores powered by Lakebase provide highly-scalable low-latency access to feature data while maintaining consistency with your offline feature tables. 📖 Documentation

Serverless GPU compute now supports H100 accelerators

Route Optimized endpoints require route optimized url path for querying

All newly created route-optimized endpoints must be queried using the route-optimized URL.

Claude Opus 4.1 is available as a Databricks hosted foundation model

📝AIBI Genie

• Benchmarks run in the background: Benchmarks run as background tasks. Users can start a run without blocking their workflow.

• Improved benchmarks accessibility: Benchmarks are available from the top-line toolbar for easier access.

• Notifications for benchmarks: A notification encourages users to run benchmarks after making changes to instructions.

• Upgraded large language models: The LLMs used by Genie to author SQL have been upgraded for improved accuracy and performance.

• SQL autocomplete: Autocomplete is now available when editing Genie SQL to help users write queries more efficiently.

• Feedback API endpoints: Thumbs up and thumbs down feedback API endpoints are now available for collecting user feedback on Genie responses.

• Suggested follow-up questions API: Suggested follow-up questions through the API are now available.

• Consumer entitlement API access: Users with the workspace consumer entitlement can now use the API suite.

• Warehouse ID in API responses: GET API requests for Genie spaces include the warehouse_id attached to the space.

• Increased value dictionary limit: Spaces can include up to 120 value dictionaries. Newly created spaces automatically select 120 columns.

• Improved accuracy for percentage questions: Genie returns more accurate results for percentage-type queries.

• Space ID field in settings: A new field in the settings page lets users directly copy the Genie space ID for use in building Genie space agents.

• New API endpoints: List conversation messages,Delete conversation message,List all conversations

• Unified permission APIs: Genie uses the Databricks Permission APIs. 📖 Documentation

• Improved Benchmarks evaluation: The evaluation function now classifies results more flexibly.📖 Documentation

📊AIBI Dashboard

• Global filters panel expanded by default: The global filters panel opens by default when users first visit a dashboard to improve discoverability.

• Genie credential handling for embedded dashboards: Genie integrated with dashboards published with embedded credentials now runs generated queries using the publisher’s data credentials. Dashboards published without embedded credentials continue running using the viewer’s data credentials.

• Simplified publish dialog: The dashboard publish dialog has been redesigned and simplified.

• Improved text widget editor: The dashboard text widget has an enhanced UI for formatting text, adding links, and inserting images.

• Active filter bar: A new filter bar shows all active filters on a dashboard. Any filter not set to All, as well as any cross-filtering, appears in this bar.

• Minimum and maximum aggregations for date fields: Date fields support minimum and maximum aggregations.

• Improved histogram labels: Histograms display both the start and end of each bin.

🛡️Governance

Access requests in Unity Catalog

🔍 Data Warehousing

You can use LIKE with columns that have one of the following collations enabled: UTF8_Binary, UTF8_Binary_RTRIM, UTF8_LCASE, UTF8_LCASE_RTRIM 📖Documentation

You can now use the ST_ExteriorRing function to extract the outer boundary of a polygon and return it as a linestring. 📖Documentation.

You can declare multiple session or local variables of the same type and default value in a single DECLARE statement.

You can use the TEMPORARY keyword when creating a metric view.

You can specify Delta read options for SQL-based streaming queries using the WITH clause.

SELECT * FROM STREAM tbl WITH (SKIPCHANGECOMMITS=true, STARTINGVERSION=X);

Subscribe now

🛠️Data Engineering

Migrate Lakeflow Declarative Pipelines from legacy publishing mode is GA

Lakeflow Declarative Pipelines has a legacy publishing mode that only allows publishing to a single catalog and schema. The default publishing mode enables publishing to multiple catalogs and schemas. 📖 Documentation

Selectively and atomically replace data with Insert replace using and insert replace on is GA

INSERT REPLACE USING replaces rows when the USING columns compare equal under equality. INSERT REPLACE ON replaces rows when they match a user-defined condition.

Microsoft SQL Server connector and ServiceNow are GA

Salesforce Data Cloud File Sharing connector is GA

New table property for Delta lake compression

You can explicitly set the compression codec for a Delta table using the delta.parquet.compression.codec table property. 📖 Documentation

Create external Delta tables from third party clients

You can now create Unity Catalog external tables backed by Delta Lake from external clients and systems, such as Apache Spark.📖 Documentation

Lakeflow Declarative Pipeline improvements

You can change the identity that a pipeline uses to run updates and the owner of tables published by the pipeline. This feature allows you to set a service principal as the run-as identity, which is safer and more reliable than using user accounts for automated workloads. 📖 Documentation

You can now easily create an ETL pipeline in a bundle in the workspace using the new Lakeflow Declarative Pipelines template project.

You can use automatic liquid clustering with CLUSTER BY AUTO and Databricks intelligently chooses clustering keys to optimize query performance.

⚡Lakebase

OLTP Database tab renamed to Lakebase Postgres

Lakebase synced tables supports syncing Apache Iceberg and Foreign tables

You can create synced tables in Snapshot sync mode from Iceberg tables or foreign tables. 📖 Documentation

Data type mapping

For new synced tables: TIMESTAMP types in source tables are mapped to TIMESTAMP WITH TIMEZONE in synced tables.

Budget policy is supported by Lakebase

You can tag a database instance and a synced table with a budget policy to attribute billing usage to specific policies. Additionally, custom tags can be added to a database instance for more granular attribution of compute usage to teams, projects, or cost centers.

Lakebase is enabled by default

The Lakebase: Managed Postgres OLTP Database preview is enabled by default

🪄Serverless

Serverless compute for notebooks, workflows, and Lakeflow Declarative Pipelines is available in the Asia Pacific (Jakarta) region (ap-southeast-3)

Base environments are custom environment specifications for serverless notebooks that define a serverless environment version and a set of dependencies. 📖 Documentation

Serverless GPU supports hyperparameters sweeps and Multinode Workloads and schedule jobs. 📖 Documentation

New features are available on Serverless Compute:

• Databricks connect upgraded to 17.0

• Scalar Python UDFS support service credentials

• PySpark and Spark Connect support the df.mergeInto

🖥️Platform

Notebooks Improvements

Edit Mode in Assistant does multi-cell code refactoring and more. 📖Documentation.

Use the cell execution minimap to track your notebook’s progress at a glance. The minimap appears in the right margin and shows each cell’s execution state. Hover to see cell details, or click to jump directly to a cell.

Notebook autocomplete supports enhanced suggestions for complex data types including structs, maps, and arrays in SQL cells.

Lakeflow job improvements

Jobs that are set to run in continuous mode have the option to retry individual tasks on task failure. 📖Documentation

Power BI Databricks connector supports M2M OAuth

You can authenticate into Power BI Desktop using M2M OAuth. Databricks recommends switching to the new client credentials authentication option. 📖Documentation

Account SCIM 2.0 updates

Databricks has updated the Account SCIM API for identity management as follows:

• Calling GET with filter params filter=displayName eq value_without_quotes results in a syntax error. To prevent this error, use quotation marks to wrap the value

• Calling GET /api/2.0/accounts/{account_id}/scim/v2/Groups no longer returns members. Instead, iterate through get group details to get membership information

• Calling PATCH /api/2.0/accounts/{account_id}/scim/v2/Groups/{id} returns a 204 response instead of a 200 response.

OAuth token federation is GA

Databricks assistant improvements

You can chat with Databricks Assistant on some compute pages. Use the Assistant chat panel to help you create a new compute resource, pool, and policy.

You can tailor how Databricks Assistant responds by adding custom user instructions. Guide the Assistant with preferences, coding conventions, and response guidelines.

Disable legacy features for new workspaces

A new account console setting allows account admins to disable certain legacy features on new workspaces created in their account. 📖Documentation

Serverless Workspaces are available

🤖GenAI & ML

AI Playground is GA

OpenAI GPT OSS models are available on Mosaic AI Model Serving

Mosaic AI Model Serving supports OpenAI's GPT OSS 120B and GPT OSS 20B as Databricks-hosted foundation models. 📖Documentation

Batch inference with GPT OSS models

OpenAI GPT OSS 120B and GPT OSS 20B are optimized for AI Functions, which means you can perform batch inference using these models and AI Functions like ai_query()

External MCP servers are in Beta

You can connect Databricks to external MCP servers. 📖 Documentation

Mosaic AI Vector Search reranker is available

Mosaic AI Vector Search offers reranking to help improve retrieval quality. 📖 Documentation

Token based rate limits are available on AI Gateway

You can configure token-based rate limits on your model serving endpoints.. 📖 Documentation

OpenAI GPT OSS models releases

The Databricks-hosted foundation models, OpenAI GPT OSS 120B and GPT OSS 20B support function and tool calling and provisioned throughput. 📖 Documentation

📝AIBI Genie

• SQL expression validation for join relationships

• File uploads are accessible only to the users who uploaded them

• You can define join relationships locally within a Genie space's knowledge store. This is useful when authors lack permissions to define primary and foreign keys on upstream tables or when the join relationship has specific requirements, such as one-to-many or complex joins

• Value dictionaries select the most frequent 1024 values from the first 100k rows, instead of the first 1024 values encountered.

📊AIBI Dashboard

• Dashboard external embedding Public Preview and new custom calculations functions

• You can define up to 200 custom calculations per dashboard.

🛡️Governance

Governed tags are public preview

You can create governed tags to enforce consistent tagging across data assets such as catalogs, schemas, and tables. Admins define the allowed keys and values and control which users and groups can assign them to objects. This helps standardize metadata for data classification, cost tracking, access control and automation. 📖Documentation

Single node compute on standard access mode is GA

This configuration allows multiple users to share a single-node compute resource with full user isolation. Single-node compute is useful for small jobs or non-distributed workloads.

Column masks retained when replacing a table

If a column in the new table matches a column name from the original table, its existing column mask is retained, even if no mask is specified. This change prevents accidental removal of column-level security policies during table replacement.

Access requests in Unity Catalog

You can enable self-service access requests in Unity Catalog by configuring access request destinations on securable objects.

Users can request access to Unity Catalog objects that they discover. These requests are sent to configured destinations, such as emails, Slack or Microsoft Teams channels, or they can be redirected to an internal access management system.

📖Documentation

Path credential vending

You can use path credential vending to grant short-lived credentials to external locations in your Unity Catalog metastore. Documentation

🔍 Data Warehousing

Default warehouse setting is available in Beta

Support for timestamp without time zone syntax

Support for schema and catalog level default collation

Expanded spatial SQL expressions and GEOMETRY and GEOGRAPHY data types

Subscribe now

July 2025 Release Highlights

🛠️Data Engineering

Moving tables between the Lakeflow declarative pipeline is GA

Teams can seamlessly reorganize streaming tables and materialized views as needs change—split a pipeline, consolidate workloads, or migrate to improved refresh schedules—without disruption or rebuilding from scratch. This increases agility in managing and scaling data platforms, helping maintain velocity as business requirements evolve. 📖 Documentation

Dynamic Partition overwrite with INSERT REPLACE USING

You can now surgically replace rows in a table based on match conditions, improving performance and simplifying updates out of the box. 📖 Documentation

Real-time mode is now available in Structured Streaming

You can now use Real Time mode, a trigger type for Structured Streaming that enables sub-second latency data processing. It’s designed for operational workloads that require immediate response to streaming data (typically from kafka to kafka). 📖 Documentation

🪄Serverless

New features are now available on Serverless Compute:

• SQL procedure support

• Set a default collation for SQL Functions

• Recursive common table expressions (rCTE) support

• PySpark and Spark Connect now support the DataFrames df.mergeInto API

• Support ALL CATALOGS in SHOW SCHEMAS

• Liquid clustering now compacts deletion vectors more efficiently

• Allow non-deterministic expressions in UPDATE/INSERT column values for MERGE operations

• Change Delta MERGE Python APIs to return DataFrame instead of Unit

Subscribe now

⚡Lakebase

Synced tables are now automatically metered and billed

Synced tables are automatically tracked for usage and costs. This brings full cost transparency and control—critical for data teams scaling to many pipelines or sharing resources across business units. No more manual tracking when optimizing spend, forecasting budgets, or allocating costs between departments—usage is visible and billable out of the box. All usage is tracked and reported in the system.billing.usage

Databricks Apps support for Lakebase resources

Teams can now integrate Lakebase databases directly as resources for Databricks Apps, streamlining the workflow from analytics and operational data to application development. This enables building smarter, data-driven applications faster—without managing separate infrastructure or extra integrations—accelerating time to insight and innovation. 📖Documentation

Run job and for each have a separate limit

Complex workflows that rely on “Run job” or “For each” tasks can now operate in parallel without hitting the overall task concurrency ceiling. This means increased throughput for orchestrating batch workloads or parallel pipelines, less bottlenecking, and more efficient use of compute—especially in large-scale environments where maximizing concurrency is critical to meeting SLAs and deadlines.

🔗Data sharing

Delta Sharing supports sharing tables and schemas secured by ABAC

Delta Sharing providers can add tables and schemas secured by attribute-based access control to a Delta share. The policy does not govern the recipient's access, so recipients have full access to the shared asset. Recipients can apply their own ABAC policies.

The name of the organization is required to enable Delta Sharing on the metastore

An organization name must be specified if you are sharing data with a Databricks recipient not in your account when enabling Delta Sharing on your metastore. 📖Documentation

Git support updates

• Databricks Git folder can be used to track and manage changes to alerts. 📖Documentation

• You can use the UI to add and manage multiple Git credentials in the workspace from one or multiple git providers. 📖Documentation

Databricks Connector for Power BI supports the ADBC Driver

You can set the Databricks Connector for Power BI to use the Arrow Database Connectivity instead of the ODBC driver. 📖Documentation

🖥️Platform

New compute policy form

Jobs and Pipelines list now includes DBSQL Pipelines

The Jobs & Pipelines list now includes pipelines for materialized views and streaming tables that were created with Databricks SQL.

System tables improvements

The usage_metadata.job_name value in the system.billing.usage table now contains the run names for runs triggered through the one-time run API.

New columns are now available in the query history system table, providing additional query insights:

• cache_origin_statement_id: For query results fetched from cache, this field contains the statement ID of the query that originally inserted the result into the cache.

• query_parameters: A struct containing named and positional parameters used in parameterized queries.

• written_rows: The number of rows of persistent data written to cloud object storage.

• written_files: Number of files of persistent data written to cloud object storage.

Disable DBFS root and mounts is now available

You can disable access to DBFS and Mounts in existing Databricks workspaces. 📖Documentation

Notebooks improvements

• Add a split view to edit notebooks side by side. 📖Documentation.

• Pressing Cmd + F (Mac) or Ctrl + F (Windows) in a notebook now opens the native Databricks find-and-replace tool. This allows you to quickly search and replace text throughout your entire notebook, including content outside the current viewport. 📖Documentation

Data exploration using an LLM

You can ask natural language questions about the sample data using Catalog Explorer. The Assistant generates the SQL based on metadata context and table usage patterns. After the query is generated, you can validate the query and then run it against the underlying table.

Restore Python Variables after idle termination in Serverless Notebooks

Databricks snapshots your notebook’s Python variables before terminating idle serverless compute. When you reconnect your notebook is automatically restored from its snapshot letting you continue your work seamlessly.

🤖GenAI & ML

Expanding region availability for ai_parse_document

ai_parse_document is now available in the following regions (AWS):

us-west-2/ us-east-1/ us-east-2 /ap-northeast-1 /ap-northeast-2 /ap-south-1 /ap-southeast-1

ap-southeast-2/ca-central-1/eu-central-1 /eu-west-1 /eu-west-2 /sa-east-1

Agent Bricks: Multi-Agent Supervisor is in Beta

Agent Bricks: Multi-Agent Supervisor supports designing a multi-agent AI system that combines Genie spaces and Knowledge Assistant agent endpoints to work together on complex tasks that require different specialized skills. Documentation

Google Gemma 3 12B is available on Mosaic AI Model Serving

Gemma 3 12B supports text inputs for foundation models APIs pay-per-token and provisioned throughput and AI functions.

📝AIBI Genie

• Caching enhancements speed up question performance when starting a new Genie session.

• Genie automatically cancels query executions that take longer than 15 minutes to execute.

• Genie allows editable parameters on date and numeric fields, expanding beyond previous support for string fields.

• You can edit Genie space descriptions directly in place.

📊AIBI Dashboard

• Authors can now undo and redo actions on the canvas using keyboard shortcuts or the toolbar.

• Within the Add data dialog, authors can now filter to show only metric views.

• Custom calculations now support AGGREGATE OVER to enable window function-like behavior, such as computing moving averages and running totals. 📖Documentation

• The maximum number of pages per dashboard has increased from ten to fifteen.

• Support for parameterized schedules has been restored. 📖Documentation

🛡️Governance

Unity Catalog Python User-defined Table functions

You can now register Python UDTFs in Unity Catalog for centralized governance and reusable logic across SQL queries. 📖Documentation

Scalar Python UDFs support service credentials

Scalar Python UDFs can use Unity Catalog service credentials to securely access external cloud services. This is useful for integrating operations such as cloud-based tokenization, encryption, or secret management directly into your data transformations.

🔍 Data Warehousing

IDENTIFIER support is now available in Databricks SQL for catalog operations.

CAN VIEW permission on SQL Warehouse is GA

The permission allows users to view SQL Warehouse, query history, and query profiles.

Expanded Spatial SQL expressions and Geometry and Geography Data types

Support for advanced spatial analytics and collation at the schema/catalog level simplifies complex location-based analytics and ensures consistency across all database objects, reducing friction in data warehousing projects. More than 80 new spatial SQL expressions are available. 📖Documentation

Support for Schema and Catalog level default collation

You can set a default collation for schemas and catalogs in Databricks Runtime 17.1. This allows you to define a collation that applies to all objects created within the schema or catalog, ensuring consistent collation behavior across your data.

Share

GenAI with Mosaic AI

Databricks has just unveiled a wave of innovation with Mosaic AI, redefining how organizations build, deploy, and govern production-grade AI and BI at scale. Let’s break down the highlights—matching your enterprise-forward, solution-focused style.

Agent Bricks: Auto-Optimized Agents for Production

Agent Bricks is a game-changer for enterprise agent development. Instead of painstakingly scripting logic or wrangling pipelines, just describe the agent’s purpose and connect your datasets in a few clicks!

Applications range from information extraction, custom knowledge assistants to multi-agent orchestration and robust enterprise copilots. Agent Bricks is already making it possible for companies to go from prototype to production in days—not weeks.

MLflow 3.0: Unified AI Observability and Governance

MLflow 3.0 reimagines agent and AI application lifecycle management for the GenAI era:

• Universal tracing: Monitor, debug, and benchmark any model or agent—on Databricks or across clouds and on-prem.

• End-to-end assessments: Built-in and customizable LLM judges, plus human-in-the-loop feedback, keep your agents compliant and continually improving.

• Prompt registry: Version, test, and deploy prompt chains side-by-side with model artifacts.

• MLOps for Agents: Complete integration with Unity Catalog for data lineage, governance, and audit trails. MLflow 3.0 transforms GenAI evaluation from guesswork into measurable business quality.

AI Functions in SQL: Lightning-Fast, Multi-Modal ai_parse_document

Mosaic AI’s SQL-native AI Functions now deliver:

• Up to 3x faster and 4x lower cost data transformation at scale.

• Multi-modal capabilities (text, images, documents) with functions like ai_parse_document, unlocking value from complex enterprise content instantly—not just structured tables.

Storage-Optimized Vector Search

Vector Search infrastructure now separates compute and storage for massive scalability and economics:

• Easily scale up to billions of vectors for RAG, semantic retrieval, and more.

• Achieve up to 7x lower cost versus previous architectures, removing the bottleneck for advanced search and copilot scenarios.

Serverless GPU Compute and High-Scale Model Serving

• GPU-powered AI is now available in serverless: launch jobs or notebooks on A10G and soon H100s, with zero infrastructure burden.

• Model Serving: Ultra-low latency and throughput exceeding 250,000 QPS, powered by an in-house LLM inference engine that outpaces open-source alternatives. Production LLM API on demand, globally.

Agent Ecosystem: MCP Support and AI Gateway

• Model Context Protocol (MCP) is now native: Build and host MCP servers within Databricks Apps, and directly connect agents to Genie, Vector Search, and more—all tested in the Mosaic Playground.

• AI Gateway: The one-stop entry for cross-service governance, safety, logging, and seamless fallback between providers.

AI/BI, Genie and Databricks One

Genie

Next-Generation Knowledge Store for Genie

Authors can now enrich data assets with semantic context—like column synonyms, joins, and certified metrics—directly in Unity Catalog. Genie also suggests new knowledge based on lineage and query history, helping teams build a living semantic model.

Automatic Knowledge Extraction

Genie extracts insights from every chat, proposing reusable knowledge snippets for user approval and storage.

Deep Research Mode

For complex “why” and “how” questions, Genie now creates research plans, tests multiple hypotheses in parallel, and summarizes results—enabling deeper, automated analysis.

Databricks One

One of the most exciting announcements! A new experience for non-technical users to access dashboards & Genie as well as Databricks Apps, without having to navigate the technical Databricks workspace.

• More Value Dictionaries: Genie now supports up to 60 value dictionaries, improving accuracy through value sampling.

• Consumer Entitlement GA: Admins can grant consumer access to users, service principals, and groups—enabling SQL warehouse queries, dashboard views, and limited workspace UI access.

• Editable Filters: Users can now refine responses by editing filter values. For string columns with dictionaries, Genie intelligently offers editable filters when they improve precision.

• New Genie API Endpoints (Public Preview): Programmatically list or delete conversations and spaces using the new Genie API.

AI/BI Dashboard

• File upload to Unity Catalog from dashboards: Authors can now upload files to Unity Catalog directly from the dashboard data tab. See Create and manage dashboard datasets.

• Customizable dashboard theme: Authors can now customize the dashboard theme by setting custom font, background, and widget colors. See Dashboard settings.

• Improved diff review for dashboard files: When committing dashboard changes to Git, queries and text box content now appear across multiple lines in the diff, making changes easier to review.

• Multi-statement queries in Genie on dashboards: Genie on dashboards now supports multistatement SQL queries.

Data Sharing

Databricks is making it easier to share assets securely and work across platforms.

Delta Sharing now gets full iceberg interoperability.

• Cross-platform sharing to any iceberg-compatible engine, allowing data providers to reach their customers across platforms like Snowflake.

• Data providers can now use Delta Sharing to share Iceberg Tables just like any other UC asset, both within Databricks and beyond

  

The new Delta Sharing network gateway removes the burden. Providers can share with recipients with minimal manual network configuration.

Delta Sharing’s Attribute-Based Access Control support allows providers to securely apply fine-grained data access policies when sharing data, ensuring sensitive information is protected while enabling broad data collaboration

Data providers can securely share data with non-Databricks users on any computing platform who prefer to authenticate using a custom Identity Provider (IdP), such as Azure Entra ID or Okta with OIDC Token Federation for Enhanced Delta Sharing Security

Cleanrooms

Databricks introduced an advancement in identity resolution. It does empower partners and markets to confidently and securely match and enrich datasets using common identifiers

Cleanrooms now support multiple collaborators in a single room

Apps

No more struggling with the “last mile” to turn AI and analytics into real-world business value. Databricks Apps are now GA, letting teams build, deploy, and scale secure, interactive, data-powered apps—all directly inside the Databricks Data Intelligence Platform.

• Instant, Secure Production: Apps run in a fully managed, serverless environment, eliminating manual setup for authentication, containers, and governance. Security, access, and observability are built in—so you focus on building, not babysitting infrastructure.

• Faster Innovation: Deploy LLM copilots, business dashboards, data quality monitors, or custom ops tools within the workspace, with governance and identity handled automatically. No more data duplication or risky exports.

• Zero Platform Drag: Start building right now in your language of choice (Python, JS, and more). Push live apps to business teams in days, not months. Over 20,000 apps have already launched across 2,500+ organizations—proof that users want to bring their AI and analytics to life, without infrastructure headaches.

• Unified with Your Data: Your apps run alongside all your governed, secure data—enabling richer, safer user experiences and blazing-fast analytics.

Marketplace: Your Open Platform for Data, Analytics, and AI

The Marketplace now goes beyond data—offering ML models, dashboards, notebooks, and applications, all instantly usable across Databricks or any Delta Sharing-compatible platform.

• Built on Open Standards: Powered by Delta Sharing, the Marketplace enables secure, cross-cloud collaboration with no vendor lock-in or ETL overhead.

• Faster Adoption with Prebuilt Solutions: Prebuilt notebooks and sample data help users quickly evaluate and implement data products with confidence.

• Expanded Sharing Capabilities: Now with full Iceberg support and a new Delta Sharing network gateway, providers can securely share assets across any analytics engine. Attribute-Based Access Control ensures secure, fine-grained sharing—even outside Databricks.

That’s a wrap! Thanks for reading all the latest Data+AI announcements!

Open Source + Community

Databricks Free Edition is available for all!

Want to try Databricks ? With Databricks Free Edition, you can learn and explore the latest data and AI technologies for FREE.

Want to get started? Sign up for Free Edition today and read the supporting documentation to begin - you’ll get a workspace out of the box, ready to use! Learn more from Tarek Madkour, Director of Product Management at Databricks

Doubling down on Open Source with Declarative Pipelines!

Previously known as DLT, Databricks shared with the Apache Spark™ ecosystem Declarative Pipelines! It adds Declarative capabilities to Spark: tell the engine your transformation in SQL or Python, and it’ll do the hard work for you.

This will land in the next Apache Spark. In the meantime, you can review the Jira and community discussion for the proposal, or hear all the details from Michael Armburst!

Unity Catalog

No more format lock-in, silos, or fragmented governance. Unity Catalog is now the only unified catalog that seamlessly connects all your data—Delta Lake or Apache Iceberg™—across any engine or cloud.

• Break Format Barriers:
  Unity Catalog fully supports the Iceberg REST Catalog API for reading and writing.

  Store your data in open Iceberg tables managed by Unity Catalog, and read/write from DBSQL, Spark, Trino, Snowflake, Amazon EMR and more.
  Iceberg catalog federation lets you govern and query Iceberg tables in AWS Glue, Hive Metastore, and Snowflake Horizon without copying or extra pipelines. This allows you to bring full Databricks governance and security to external catalogs, breaking down legacy data boundaries.

• Share Data Without Limits:
  Delta Sharing for Iceberg enables you to share Unity Catalog and Delta tables with any client via the Iceberg REST Catalog API. Collaborate without friction,

• One Semantic Layer for All

  Unity Catalog now provides a unified business-aware experience for every data stakeholder:

  • Unity Catalog Metrics
    Make business metrics a first-class citizen. Define KPIs once at the lakehouse layer. Use them everywhere in dashboards, notebooks, Genie, Lakeflow jobs, and soon in top BI and observability tools.

  • Curated Discovery and Intelligent Insights
    The new business domain marketplace and discover experience turns Unity Catalog into an internal data hub:

    • Certified data products, metrics, dashboards, AI agents, and Genie spaces—all organized by business unit (Sales, Marketing, Finance, etc.).

    • AI-powered recommendations, intelligent signals (quality, usage, relationships), and curation surface the highest-value assets, so users always find what’s trustworthy and relevant.

    • Certifications and deprecation tags signal data trust, lifecycle, and business relevance directly in your workflow.

• Attribute-Based Access Control (ABAC)
  Define dynamic tag-driven access policies at any level catalog, schema, or table. ABAC policies now inherit across the hierarchy and can leverage tags for region, business domain, sensitivity, and beyond. Combined with new data classification, sensitive and regulated data is automatically detected and protected, simplifying policy management and audits at any scale.

• Tag Policies
  Account-level policies ensure that tags used for classification, cost attribution, and compliance remain consistent.

• Request for Access
  Users can now instantly request access to any asset, streamlining delivery and reducing friction between data consumers and governance teams.

DBSQL

With the launch of Predictive Query Execution and Photon Vectorized Shuffle, queries can be up to 25% faster, on top of the 5x gains we saw over the last years. These new engine improvements roll out automatically across all DBSQL Serverless warehouses at zero additional cost.

A bunch of functions and features has been added lately. This list is not exhaustive - SQL team is shipping very fast to provide state of the art Datawarehousing capabilities

Lakebridge: Free, Open Data Migration to DBSQL using AI powered tooling

• Migrating from legacy, siloed data warehouses is key to unlocking faster insights, reducing costs, and unifying analytics & AI on Databricks. But complex code and hidden dependencies make it risky.

  Lakebridge is a free tool that automates up to 80% of the migration, making the process faster and safer.

  It includes three core components:

  • Analyzer – Assesses your legacy environment in detail.

  • Converter – Translates legacy SQL and ETL (including stored procedures) into Databricks SQL or Spark SQL.

  • Validator – Verifies data accuracy with built-in reconciliation tools.

Data ingestion with Lakeflow

Databricks Lakeflow is now GA. It does include three core components:

Lakeflow Connect

Ingest from enterprise apps, databases, file systems and real-time streams with just a few clicks

Zerobus was announced. It is an innovative approach for pushing event data directly to your lakehouse, bringing you closer to the data source and eliminating data hops and operational burden to provide high-throughput direct writes with low latency. Zerobus is part of Lakeflow Connect.

Lakeflow Declarative Pipelines (previously DLT)

Scalable ETL pipelines built on the open standard of Spark Declarative Pipelines, integrated with governance and observability, and providing a streamlined development experience through a modern IDE for data engineering

Lakeflow Jobs

Native orchestration for the Data Intelligence Platform supporting advanced control flow, data triggers, and comprehensive monitoring.

Lakeflow Designer: probably the most exciting addition!

Databricks announced an AI-Powered No Code ETL called Lakeflow Designer that is fully integrated with the Databricks Data Intelligence Platform. With a visual canvas and built-in natural language interface, Designer lets business analysts build scalable production pipelines and perform data analysis without writing a single line of code–all in a single. Every pipeline built in Designer creates a Lakeflow Declarative Pipeline under the hood.

Lakebase

Lakebase is Databricks new, fully managed Postgres-compatible OLTP database, purpose-built to power modern operational and AI-driven applications within the Databricks Lakehouse Platform.

• Separation of Compute and Storage: Store data in open formats on low-cost object storage, scaling compute and storage independently.

• Serverless and Autoscaling: Instantly scale compute resources to meet workload demands—or scale to zero when idle, so you only pay for storage when not in use.

• Database Branching: Instantly create isolated database branches for testing, feature development, or experiment rollback using copy-on-write, just as you would in a Git repo—ideal for development, auditing, and managing AI agents.

• Unity Catalog Integration: Register Lakebase as a catalog, enabling seamless management, governance, and access controls unified with your Lakehouse data.

• Managed Data Synchronization: Sync operational data between Lakebase and Delta tables, reducing custom ETL and keeping data consistent for analytics and operational use cases.

• Performance & Availability: Multi-zone replication, point-in-time recovery, and enterprise-grade security with automated backups. Monitor performance metrics in real time.

• AI-Ready: Designed for high-concurrency, low-latency workloads required for modern AI applications and agents—enables native development of GenAI and real-time analytics apps.

Why Lakebase?

Lakebase accelerates the development of intelligent operational applications by bringing transactional and analytical capabilities together. Its instant scalability, powerful branching, and serverless operation help teams deliver faster, run more efficient workloads, and reduce operational overhead and cost. This marks a significant step toward simplifying enterprise data architectures—replacing costly, legacy OLTP stacks with a single, cloud-native platform for all data-driven applications.

It’s a game-changer for all your Databricks applications!!

Stay tuned for the Part 2 - coming in a few days!

May 2025 Release Highlights

• Databricks Apps is GA

• Metrics views are in Public preview

Data Engineering

File events on external location

You can now enable file events on external locations that are defined in Unity Catalog. This makes file arrival triggers in jobs and file notifications in Auto Loader more scalable and efficient. Learn more

Views in ETL pipelines

The CREATE VIEW SQL command is now available in ETL pipelines. You can create a dynamic view of your data. Learn more

Pipelines and workflows enhancements

Tables created by DLT in Unity Catalog ETL pipelines can now be moved from one pipeline to another.

In the past repaired tasks were unblocked once their direct dependencies completed. Now, repaired tasks wait for all transitive dependencies. For example, in a graph A → B → C, repairing A and C will block C until A finishes.

Databricks Asset bundles in the workspace

Collaborating on Databricks Asset Bundles with other users in your organization is now easier with bundles in the workspace, which allows workspace users to edit, commit, test, and deploy bundle updates through the UI. Learn more

Other updates

• Type widening support added for streaming for Delta tables.

• You can use the IDENTIFIER clause when performing : Create Catalog, Drop Catalog, Comment on Catalog, Alter Catalog

Governance and data sharing

System tables update

You can now use the system.access.workspaces_latest table to monitor the latest state of all active workspaces in your account.

The system.lakeflow schema, which contains system tables related to jobs, is now enabled by default in all Unity Catalog workspaces.

The system.lakeflow.pipelines table is a slowly changing dimension table (SCD2) that tracks all pipelines created in your Databricks account.

The lineage system tables (system.access.column_lineage and system.access.table_lineage) have been updated to better log entity information.

• The entity_metadata column replaces entity_type, entity_run_id, and entity_id, which have been deprecated.

• The record_id column is a new primary key for the lineage record.

• The event_id column logs an identifier for lineage event, which can be shared by multiple rows if they were generated by the same event.

• The statement_id column logs the query statement ID of the query that generated the lineage event. It is a foreign key that can be joined with the system.query.history table.

Cleanroom enhancements

Databricks Clean Rooms now supports:

• Up to 10 collaborators for more complex, multi-party data projects.

• New notebook approval workflow that enhances security and compliance, allowing designated runners, requiring explicit approval before execution

• Auto-approval options for trusted partners.

• Difference views for easy review and auditing.

Platform

Predictive Optimization is now enabled for all Databricks accounts

Query Snippets are now available everywhere

Query snippets are segments of queries that you can share and trigger using autocomplete. You can now create query snippets through the View menu in the new SQL editor, and also in the notebook and file editors. Learn more

Jobs and pipelines share a single unified view

Automatically provision user with JIT

You can now enable just-in-time (JIT) provisioning to automatically creates new user accounts during first-time authentication. When a user logs in to Databricks for the first time using single sign-on (SSO), Databricks checks if the user already has an account. If not, Databricks instantly provisions a new user account using details from the identity provider. Learn more

Support of workflow scope in Databricks App to support Authoring Github actions

Databricks made a change where you may get an email request for the Read and write access to Workflows scope for the Databricks GitHub app. This change makes the scope of the Databricks GitHub app consistent with the required scope of other supported authentication methods and allows users to commit GitHub Actions from Databricks Git folders using the Databricks GitHub app for authorization.

If you are the owner of a Databricks account where the Databricks GitHub app is installed and configured to support OAuth, you may receive the following notification in an email titled "Databricks is requesting updated permissions" from GitHub. Accept the new permissions in order to enable committing GitHub Actions from Databricks Git folders with the Databricks GitHub app.

SQL Authoring improvements

• Filters applied to result tables now also affect visualizations, enabling interactive exploration without modifying the underlying query or dataset.Learn more

• In a SQL notebook, you can now create a new query from a filtered results table or visualization.Learn more

• You can hover over * in a SELECT * query to expand the columns in the queried table.

• Custom SQL formatting settings are now available in the new SQL editor and the notebook editor. Click View > Developer Settings > SQL Format. Learn more

Improved UI for managing notebook dashboards

Databricks MFA is GA

Account admins can now enable multi-factor authentication (MFA) directly in Databricks, allowing them to either recommend or require MFA for users at login. Learn more

Dashboards, alerts and queries are supported as workspace files

Dashboards, alerts, and queries are now supported as workspace files, which means you can programmatically interact with these Databricks objects like any other file, from anywhere the workspace filesystem is available.

Databricks Apps is now GA

GenAI & ML

Llama 4 Maverick is available in EU regions

Llama 4 Maverick can now be used in EU regions where Foundation Model API pay-per-token is supported.
Learn more
Why it matters: This expands access to cutting-edge language models for European customers, supporting data residency and compliance needs while enabling advanced generative AI applications.

Llama 4 Maverick supports provisioned throughput workloads (Public Preview)

Llama 4 Maverick is now supported on Foundation Model APIs provisioned throughput workloads, available in Public Preview.
See Provisioned throughput Foundation Model APIs
See Preview limitations
Why it matters: Provisioned throughput enables consistent, high-performance access to Llama 4 Maverick for production workloads, removing latency variability typical in on-demand usage.

Mosaic AI Model Serving expands to new regions

Mosaic AI Model Serving is now available in ap-northeast-2, eu-west-2, and sa-east-1.
See Model serving feature availability
Why it matters: More regional availability helps global organizations reduce inference latency and comply with local data regulations.

Provisioned throughput workloads support compliance security standards

Provisioned throughput Foundation Model APIs now support HIPAA, PCI-DSS, and UK Cyber Essentials Plus compliance standards.
See compliance availability
Why it matters: Teams working with sensitive or regulated data can now use high-performance AI workloads while meeting critical compliance requirements.

Model Serving CPU and GPU workloads support compliance standards

Model Serving workloads (CPU and GPU) now support HIPAA, PCI-DSS, and UK Cyber Essentials Plus compliance via the compliance security profile.
See compliance availability
Why it matters: Ensures secure and compliant deployment of AI services across regulated industries, enabling safer AI adoption in healthcare, finance, and government sectors.

Databricks Asset Bundles now available in the workspace (Public Preview)

Collaborate more easily on Databricks Asset Bundles through the UI by editing, committing, testing, and deploying bundle updates directly from the workspace.
See Collaborate on bundles in the workspace
Why it matters: Makes it easier for cross-functional teams to manage code and configuration changes in production workflows without relying on local CLI setups.

Claude Sonnet 4 and Claude Opus now on Mosaic AI Model Serving

The Anthropic Claude Sonnet 4 and Claude Opus 4 models are now available as Databricks-hosted foundation models via Mosaic AI Model Serving in US regions (pay-per-token).
See Foundation Model APIs
Why it matters: These models offer strong reasoning and summarization capabilities, giving users more choices for deploying conversational AI and enterprise LLM solutions.

Agent Bricks: Knowledge Assistant now in Beta

Agent Bricks offers a no-code tool for creating domain-specific Q&A chatbots. The Knowledge Assistant supports fine-tuning your assistant based on feedback from subject matter experts.
See how to use Knowledge Assistant
Why it matters: Lowers the barrier to building specialized AI assistants, enabling business teams to deploy effective AI solutions without needing deep ML expertise.

Here are polished entries for each video, with a title, what it covers, and why you should watch it:

📺 Automate Cost Reporting with Genie + MCP in Databricks

🔗 Watch the video

What it covers:
A hands-on demo of using Genie and the Anthropic’s (MCP) protocol to automate the generation of Databricks cost reports, complete with executive summaries and usage breakdowns via a conversational interface.

📺 Building a Multi-Genie AI Agent Architecture on Databricks

🔗 Watch the video

What it covers:
This video follows Aarni's journey as he builds a multi-genie AI architecture on Databricks. It features a “Genie of Genies” system that routes user queries to domain-specific LLM agents using orchestration, permissions, logging, and session memory—all deployed using Databricks Apps and service principals. Includes a live demo and open-source code walkthrough.

📺 KPMG + Databricks: GenAI for Scalable Financial Audits

🔗 Watch the video

What it covers:
KPMG and Databricks showcase how they use GenAI and Unity Catalog to automate audit workflows, including intelligent document parsing, entity extraction, and secure access controls across financial datasets.

AIBI

AI/BI dashboards

• Unity Catalog metric views are now in Public Preview: Metric views provide a centralized way to define and manage consistent, reusable, and governed core business metrics. They abstract complex business logic into a centralized definition, enabling organizations to define key performance indicators once and use them consistently across reporting tools like dashboards, Genie spaces, and alerts.

• Global filters panel: Dashboard authors can now add filters to a global filters panel, which applies across all dashboard pages.

• Relative date and time preview improvement: Previews for relative date and time controls are now hidden to avoid confusion caused by time zone differences.

• Dashboard PDF snapshots improvement: Dashboard PDF snapshots now wrap the content of the dashboard and no longer include white space.

• Genie in dashboards query update: Genie in dashboards will now only query from dashboard datasets, instead of the underlying tables within dashboard datasets.

• Custom calculation support: Added support for custom calculations using isnull, isnotnull functions, and the null operator.

• AI/BI dashboard schedule support: AI/BI dashboard schedules now support dashboards without embedded credentials.

• Number formatting support: Number formatting in table and pivot table cells from notebooks, SQL editor, and legacy dashboards can now be carried over to AI/BI dashboards.

• Select and add multiple tables and views: Users can now select and add multiple tables and views at once from the Add data dialog.

• AI-assisted point maps in dashboards: You can now use natural-language prompts to generate point map charts in dashboards.

• Deploy dashboard tasks using Databricks Asset Bundles: You can now use dashboard tasks within a job resource when deploying bundles.

AI/BI Genie

• Popular query suggestions in Genie: Genie will now suggest popular queries as example SQL for newly added tables to the space. Only popular queries that the author has access to are suggested. See Create a Genie space.

• Increased accuracy with example values: Genie now collects example values when you add tables to a space, helping it better understand the data in your tables and generate more accurate responses. See Edit column metadata.

• Edit column descriptions in Genie: You can now edit table and column descriptions at the Genie-space level. This allows you to fine-tune metadata descriptions within a specific the Genie space. See Edit column metadata.

• Column synonyms support: You can now add column synonyms to help Genie interpret your data, making column descriptions easier to understand. See Edit column metadata.

• Hide unnecessary columns: You can now hide unnecessary columns from Genie to simplify the data model, without needing to create new UC assets.

• SQL function de-emphasis: The option to add SQL functions has been de-emphasized, making adding example SQL the default action.

• Increased value dictionary support: Each value dictionary in Genie now supports 1024 values per column, up from 255. Click Refresh value dictionary on your existing value dictionaries to increase cardinality support in those columns. See Use value dictionaries to improve Genie's accuracy.

• Point maps support in Genie: Point maps are now supported in Genie.

Data Warehousing

IDENTIFIER support now available in Databricks SQL for catalog operations.

You can now use the IDENTIFIER clause when performing the following catalog operations:

• CREATE CATALOG

• DROP CATALOG

• COMMENT ON CATALOG

• ALTER CATALOG

This new syntax allows you to dynamically specify catalog names using parameters defined for these operations, enabling more flexible and reusable SQL workflows. As an example of the syntax, consider CREATE CATALOG IDENTIFIER(:param) where param is a parameter provided to specify a catalog name.

New alert version in Beta

A new version of alerts is now in Beta. This version simplifies creating and managing alerts by consolidating query setup, conditions, schedules, and notification destinations into a single interface. You can still use legacy alerts alongside the new version.

Metric Views are in Public Preview

Unity Catalog metric views provide a centralized way to define and manage consistent, reusable, and governed core business metrics. They abstract complex business logic into a centralized definition, enabling organizations to define key performance indicators once and use them consistently across reporting tools like dashboards, Genie spaces, and alerts. Use a SQL warehouse running on the Preview channel (2025.16) or other compute resource running Databricks Runtime 16.4 or above to work with metric views.

SQL Editor UI Updates

• New queries in Drafts folder: New queries are now created by default in the Drafts folder. When saved or renamed, they automatically move out of Drafts.

• Query snippets support: You can now create and reuse query snippets—predefined segments of SQL such as JOIN or CASE expressions, with support for autocomplete and dynamic insertion points. Create snippets by choosing View > Query Snippets.

• Audit log events: Audit log events are now emitted for actions taken in the new SQL editor.

• Filters impact on visualizations: Filters applied to result tables now also affect visualizations, enabling interactive exploration without modifying the SQL query.

• Salesforce and Workday connectors are GA

• Google Analytics raw data connector is now public preview

Data Engineering

Salesforce ingestion and Workday reports connectors are GA

Google Analytics raw data connector is now public preview

Access UDF context information using TaskContext

The TaskContext Pyspark API allows to retrieve context information like identity and cluster tags while running BATCH UNITY CATALOG PYTHON UDF or PYSPARK UDF. Documentation

Batch Unity Catalog Python UDFs

Unity Catalog batch python UDFs extend the capabilities of Unity Catalog UDFs by allowing you to write python code to operate on batches of data. Batch python UDFs support service credentials to access external cloud services. Documentation

Run a subset of tasks within a job

You can run a subset of tasks when manually triggering the job.

Multi file editor for ELT pipeline development in DLT

You can now use the new multi-file editor in DLT to develop and debug ETL pipelines. The multi-file editor shows a pipeline as a set of files in the pipeline assets browser. You can edit the files and control the configuration of the pipeline and which files to include in one location. Documentation

Delta Live Tables enhancements

All streaming tables and materialized views are moving to use predictive optimization for maintenance tasks.

New streaming tables and materialized views will follow the workspace settings for deletion vectors

Governance and data sharing

Share STs and MVs to all recipients using Delta sharing

You can now use Delta Sharing to share streaming tables and materialized views to all recipients.

Enforcement of RLS/CLS in Delta Sharing

Delta Sharing now consistently enforces row-level security and column masking policies applied to tables a shared data asset is dependent on, of whether those policies were applied before or after the data asset was shared. Recipients may experience differences in query behavior when accessing shared data that is depending on tables with row-level security or column masking policies. This ensures that data access aligns with the provider's intended security controls at all times.

Browse privilege is GA

The BROWSE privilege in Unity Catalog is crucial for enhancing user data discovery capabilities. This privilege enables users to view metadata without having direct access to the underlying data, thus facilitating the exploration of data lineage and object relationships without compromising data security.

Compute System tables are GA

Compute system tables: Clusters, node_types and node_timeline are GA.

Platform

Databricks MFA is now available

Account Admins can enable MFA in Databricks. This feature is only available for accounts where SSO is not enabled. Documentation

Databricks assistant improvements

In answer to prompts and queries, assistant now provides suggestions for next steps and citations from documentation. The assistant can also run any code

Reference SQL output in downstream tasks of a job

You can use dynamic values to reference the output of a SQL task in downstream tasks in the same job. For each tasks can iterate over the rows of data in the output. Documentation

GenAI & ML

• Use vector search indexes to prototype tool-calling generative AI agents in AI Playground

  🔗Watch the Demo video​

  Rather than creating custom Unity Catalog functions with a vector search index, you can now add the vector search index to your gen AI agents directly in AI Playground. The agent will be able to use the vector search index as a tool to help it respond to queries. See Prototype tool-calling agents in AI Playground.

• AI Gateway now supports custom model serving endpoints​

  You can now enable the following governance and monitoring features on your custom model serving endpoints:

  • Permission and rate limiting to control who has access and how much access.

  • Payload logging to monitor and audit data being sent to model APIs using inference tables.

  • Usage tracking to monitor operational usage on endpoints and associated costs using system tables.

• Llama 4 Maverick is now available as a Databricks-hosted foundation model.

  You can use it for batch inference endpoints optimized for AI Functions.

• AI Builder: Model specialization for custom gen AI text agents is in Beta

  🔗Watch the ​Demo video

  AI Builder provides a simple, no-code approach to build and optimize domain-specific, high-quality AI agent systems for common AI use cases. In Beta, AI Builder supports model specialization for custom text-based tasks, such as summarization, classification, text transformation, and content generation.

  See Use AI Builder: Model specialization to create a gen AI agent for text.

• Build GenAI Focused Front End Applications on Databricks with Ivan Trusov

  🔗 Watch the video

  In this session, we cover the fundamentals of Databricks Apps, deployment best practices and supported frameworks. The highlight is a live demo of a real-world Retrieval-Augmented Generation (RAG) chatbot with advanced PDF highlighting capabilities.

AIBI

AI/BI dashboards​

End of support for legacy dashboards

Legacy dashboards can no longer be created or cloned via the UI or API. Databricks will continue to address critical security issues and outages, but the recommendation will be to use AI/BI dashboards for new development. Documentation.

Source control for dashboards

Dashboards can now be integrated into Git folders. Documentation.

Streamlined publish flow

Publishing a dashboard now embeds the user’s warehouse permissions, removing the previous requirement of end users requiring individual access to the selected warehouse. Credentials to the data will still need to be specified.

Sankey Visualisations are now available on AI/BI dashboards. Documentation.

Genie Conversations on AI/BI dashboards now resumes the last conversation when opened until a new one is started.

Custom Dimensions

Authors can now create custom dimensions which allows a new set of functions including CASE WHEN statements, CONCAT, COALESCE to be supported. Documentation.

AI/BI Genie​

The Genie Conversations API is now in available in public preview. Documentation.

Example and follow up questions

Genie now uses table metadata to offer example and follow up questions

Improved data access transparency

Users can now view and remove missing tables in a Genie space. The Data tab now shows visual indicators to mark tables that cannot be accessed. You can use the UI to remove those tables.

Revamped feedback flow

Genie feedback workflow is now more interactive and responsive. When a response is incorrect, you can explain why it's wrong by choosing from a selection of common issues or entering your own explanation. Documentation.

SQL

Write procedural SQL scripts based on ANSI SQL/PSM (Public Preview):

You can now use scripting capabilities based on ANSI SQL/PSM to write procedural logic with SQL, including conditional statements, loops, local variables, and exception handling. Documentation.

New H3 functions have been added:

• h3_try_coverash3

• h3_try_coverash3string

• h3_try_tessellateaswkb.

• Automatic Liquid clustering is now public preview

• Serverless Compute can now use instance profiles for data access

• Write procedural SQL scripts based on ANSI SQL/PSM

Data Engineering

ServiceNow Lakeflow Connector is now available

You can now ingest from ServiceNow using Databricks Lakeflow Connect

Write procedural SQL scripts based on ANSI SQL/PSM​

You can now use scripting capabilities based on ANSI SQL/PSM to write procedural logic with SQL, including control flow statements, local variables, and exception handling. Documentation.

Table and view level default collation​

You can now specify a default collation for tables and views. This simplifies the creation of table and views where all or most columns share the same collation. Documentation

Support for transformWithStateInPandas on standard compute​

You can now use transformWithStateInPandas on compute configured with standard access mode.

Support for appending data to fine-grained access controlled tables on dedicated compute​

In DBR16.3+ dedicated compute supports appending to Unity Catalog objects that use fine-grained access control.

Enhanced support for self-joins with fine-grained access controlled objects on dedicated compute​

In DBR 16.3+ the data filtering functionality for fine-grained access control on dedicated compute now automatically synchronizes snapshots between dedicated and serverless compute resources with the exception of materialized views and any views, materialized views, and streaming tables shared using Delta Sharing.

Governance and data sharing

Install custom dependencies in Unity Catalog Python UDFs

You can extend the functionality of Unity Catalog Python UDFs beyond the Databricks Runtime environment by defining custom dependencies for external libraries.

Delta Sharing support for query optimizer statistics is GA

Data recipients accessing shared tables can benefit now fro Optimizer statistics to improve the query performance. The provider should have enabled Predictive Optimization or run Analyze command.

Hive metastore federation is GA

Hive metastore federation enables you to use Unity Catalog to access and govern data that is registered in a Hive metastore. This includes externally-managed Hive metastores, AWS Glue, and legacy internal Databricks Hive Hive metastores.

Bind storage credentials, service credentials and external location to a specific workspace

The ability to bind storage credentials, service credentials, and external locations to specific workspaces is now generally available. Workspace binding prevents access to those objects from other workspaces. This feature is especially helpful if you use workspaces to isolate user data access.

OIDC federation for Databricks to open Delta Sharing

OIDC federation is an alternative to using long-lived Databricks-issued bearer tokens to connect non-Databricks recipients to providers. It enables fine-grained access control, supports MFA, and reduces security risks by eliminating the need for recipients to manage and secure shared credentials.

Databricks-to-open sharing with OIDC federation supports user-to-machine (U2M) authentication for recipients on Power BI and Tableau, and machine-to-machine (M2M) OAuth Client Credentials authentication from Python client apps registered in a recipient's IdP. Documentation

Delta sharing supports renaming tables

Delta Sharing now supports renaming shared tables. If a provider renames a shared table, the table on the recipient side automatically reconciles without erroring

Platform

Databricks Apps: Control access to resources based on user permissions

Databricks Apps can now use on-behalf-of-user authorization to ensure apps interact with resources using the permissions assigned to the app user. You can use on-behalf-of-user authorization when access to resources, such as UC-managed tables or model serving endpoints, should be restricted by the permissions assigned to the user

Budget policies renamed to Serverless budget policies

Monitor and revoke PAT in your account

Token report is now in Public Preview. Token report enables account admins to monitor and revoke personal access tokens (PATs) in the account console. Databricks recommends you use OAuth access tokens instead of PATs for greater security and convenience. Documentation

Download as Excel is supported in Notebooks

For notebooks connected to serverless compute, you can now download tabular cell results as an Excel file. Download as Excel is also supported for notebooks connected to SQL warehouses

Compute logs can be delivered to Unity Catalog Volumes

You can now deliver a compute resource's driver, worker, and event logs to a Unity Catalog volume. Documentation

GenAI & ML

• AI Gateway now supports fallbacks for external models:

  • Demo video

  • You can now enable fallbacks on your model serving endpoints using AI Gateway. If your request to an external model fails, you can specify that the request be redirected to other external models served on that endpoint. See Configure AI Gateway using the UI.

• Mosaic AI Agent Framework is generally available:

  • Mosaic AI Agent Framework comprises a set of tools on Databricks designed to help developers build, deploy, and evaluate production-quality agents like Retrieval Augmented Generation (RAG) applications. It is compatible with third-party frameworks like LangChain and LlamaIndex, allowing you to develop with your preferred framework and while leveraging Databricks’ managed Unity Catalog, Agent Evaluation Framework, and other platform benefits.

  • Quickly iterate on agent development using the following features:

    • Create AI agents using any agent authoring library and MLflow. Parameterize your agents to experiment and iterate on agent development quickly.

    • Create agent tools to extend the LLMs capabilities beyond language generation.

    • Agent tracing lets you log, analyze, and compare traces across your agent code to debug and understand how your agent responds to requests.

    • Agent Evaluation helps evaluate the quality, cost, and latency of agents to identify quality issues and determine the root cause of those issues.

    • Improve agent quality using DSPy. DSPy can automate prompt engineering and fine-tuning to improve the quality of your gen AI agents.

    • Deploy agents to production with native support for token streaming and request/response logging, plus a built-in review app to get user feedback for your agent.

  • Agent Evaluation Framework: Demo video

  • Custom Metrics for Evaluating AI Agents on Databricks | MLflow Trace & AI Performance: Demo video

• Agent Framework monitoring is in Beta

  • Demo video

  • Agent Framework monitoring is in Beta. You can now set up monitoring on your agents deployed using Mosaic AI agent framework.

  • AI Agent Monitoring Dashboard: Demo video

• Claude Sonnet 3.7 now supported on Mosaic AI Model Serving:

  • Demo video

  • Anthropic's Claude Sonnet 3.7 is now supported on Mosaic AI Model Serving. Claude Sonnet 3.7 is available using:

    • Foundation Model APIs pay-per-token

    • General purpose AI Function: ai_query

    • Function calling

• AI Builder: Information extraction is in Beta

  • Demo video

  • AI Builder provides a simple, no-code approach to build and optimize domain-specific, high-quality AI agent systems for common AI use cases. In Beta, AI Builder supports information extraction and simplifies the process of transforming a large volume of unlabeled text documents into a structured table with extracted information for each document.

• AI Gateway now supports pay-per-token endpoints:

  • Mosaic AI Gateway now supports Foundation Model APIs pay-per-token endpoints.

• Perform batch inference using AI Functions (Public Preview):

  • Mosaic AI now supports batch inference using AI Functions. This functionality is in Public Preview.

  • Demo video on how to run LLM Batch Inference with ai_query() on Databricks

  • Example demo video on how to create forecasts with SQL and AI functions on Databricks

• External models now supports OpenAI-compatible LLMs:

  • External models in Mosaic AI Model Serving now supports a custom provider option. You can use this option to specify alternate providers that are not explicitly listed as a model provider or models behind custom proxies that are OpenAI API compatible.

AIBI

AI/BI dashboards​

• Chart migration fix: Dual-axis charts are now converted as expected when migrating from legacy dashboards to AI/BI dashboards.

• Customize dashboard visualization settings by locale: Use the editing panel on your draft dashboard to select a locale and customize settings across all filter and visualization widgets.

• Improved URL stability: Importing a new version of a dashboard in draft and publishing no longer breaks filter values saved in existing URLs

• Query history update: When viewing the query history, scheduled dashboard update queries now display the dashboard publisher as the user instead of System Service Principal.

• Increased custom sorting limits: You can now custom sort up to 500 items in a visualization.

• New audit log emitted for dashboard downloads: Downloading dashboards as PDF now triggers the audit log event triggerDashboardSnapshot.

• Dashboards are supported as a task type in workflows: Create a job task with the Dashboard task type to refresh dashboard results and optionally send subscription emails.

• Calculated measures from dashboards better supported: Genie spaces created from dashboards are now better able to use any calculated measures defined on the dashboard.

• Fixed null value styling: Pivot table cells with null values now display with the correct styling.

• Sorting with filters now supported: Single-value and multi-value filters now support sorting.

AI/BI Genie​

The Genie Conversations API is now in available.. You can now ask questions, retrieve generated SQL and query results, or get the details associated with a Genie space using the Databricks REST API. Documentation.

• Improved creation flow: When you create a new Genie space, you need only to add tables to the space to start testing and iterating.

• Conversations API update: Messages sent through the Conversations API now appear in the monitoring tab.

• New Genie space UI: Genie now has a redesigned UI for authoring and chats. The new layout has more space for conversations and a reorganized interface for adding and refining instructions.

• Privileged users can help refine instructions: Users with at least CAN EDIT permissions can now view the source SQL used to generate answers, allowing them to help refine Genie instructions.

• Restricted table selection: To return results, users must have at least SELECT privileges on the Unity Catalog objects in the space. Authors are now restricted from adding tables where they have insufficient permissions.

• Improved SQL matching: Genie can now better utilize example SQL statements when user prompts closely match the example SQL.

• Easier to understand audit events: The updateConversationMessageFeedback audit event now includes a feedback_rating field to quickly see whether a rating is positive or negative.

• Genie data sampling is now available in Public Preview: This feature improves Genie’s ability to translate user prompts into the right column and row values. To test this feature, contact your Databricks account team.

• Genie now self-reflects: As Genie generates SQL, it self-reflects to fix issues and return higher-quality answers. This improves Genie’s ability to author filter conditions and fewer SQL errors.

• Improved keyword contextualization: Genie has improved its ability to provide relevant context in its responses based on keywords in the user question.

• Improved benchmark evaluation logic: Genie now supports column and row reordering and permits extraneous column selection in SQL results.

Genie leverages generative AI tailored to an organization’s data, usage patterns, and business concepts and continuously learns from user feedback.

What are the requirements to get started ?

The data for the genie space must be registered to Unity Catalog and require Pro or Serverless SQL Compute.

You would need the Can use access to the default SQL Warehouse and at least the select privilege on the data used by Genie.

How to setup an AIBI Genie Space ?

• Title and Description: What is the scope and domain of questions they're expected to know how to answer? This is visible to anyone browsing and searching across all of the Genie spaces they can access

• Default warehouse: What compute will this Genie use to run SQL queries for you?

• Tables: What tables are in scope for this Genie to use to answer questions? Genie will read up on all of the comments in each table and in each of its columns and use these to inform its answers.

How to I improve the accuracy of Genie ?

Metadata: add comments

You can add comments on the different columns using the AI Generate or type them manually When using AI Generate make sure to review the comments before the validation.

Metadata: add PK and FK

You can add PK and FK to help Genie running the joins.

Data sampling

Data sampling greatly enhances response quality because it teaches Genie about the actual values in your data, so its able to generate more precise and relevant queries

General instructions

It’s a plain text guidelines that define how genie should behave and process data. It allows data teams to specify the guidance by defining the KPI, internal concepts, unique jargon of the company..

History Tab

you can monitor the space to proactively to understand the queries raised by the users and updates the instructions accordingly based on the responses. The users can rate each response with a thumbs-up or thumbs-down or ask for a response to be reviewed.

If you want a second opinion you can use Request Review feature to complete the review cycle directly in Genie.

By accessing the history of all the answers you can spot the errors and correct the SQL queries then save them as instructions.

Trusted Assets

Trusted assets are predefined functions and example queries meant to provide verified answers to questions that you anticipate from users. When a user submits a question that invokes a trusted asset, it’s indicated in the response, adding an extra layer of assurance to the accuracy of the results.

Trusted assets can include:

• Parameterized example SQL queries: When a parameterized example SQL query is used to generate a response, the reponse is labeled a trusted asset.

• User-defined table functions: You can define functions and register them with Unity Catalog then you can add those functions as trusted assets when you’re setting up instructions.

  

Benchmarks

Benchmarks allow you to create a set of test questions covering the most frequently asked ones in order to assess the overall response accuracy.

How to expose AIBI Genie in a WebApp ?

You can start using the Genie Conversation API programmatically by submitting natural language prompts and receiving data insights just as you would in the Genie user interface. The API is stateful, allowing Genie to retain context as you ask follow-up questions within a conversation thread. Documentation

With the REST API you can leverage Genie capabilities from Teams, Databricks App, Web App..