What's new in Databricks - August 2024
August 2024 Release Highlights
Databricks Runtime 15.4 LTS is GA
Meta Llama 3.1 405B models are now supported in Mosaic AI Model Training
GenAI/ML
No time to read? Check out this recap video of all the announcements listed below for August 2024 👇
→ Subscribe to our YouTube channel for regular updates
Lakehouse Monitoring (GA)
Lakehouse Monitoring allows you to profile, diagnose, and enforce quality, with automated profiling for any Delta Table in Unity Catalog out-of-the-box. It creates two metric tables in your account — one for profile metrics and another for drift metrics. For inference tables representing model inputs and outputs, you also get model performance and drift metrics. Leveraging these computed metrics, Lakehouse Monitoring automatically generates a dashboard plotting trends and anomalies over time. You can then edit this dashboard to tailor it to your use cases, with custom metrics, alerts, and more. → Blog Post and Demo below
Fine-grained access control is available on single-user compute (Public Preview)
You can take advantage of Unity Catalog's fine-grained access control features (views, row filters, column masks) as well as materialized views and streaming tables when using single user clusters. This capability is automatically enabled in Single User clusters running runtime versions 15.4 LTS+ in workspaces that have Serverless compute for Notebooks and Workflows enabled. → Documentation
Platform releases
Mosaic AI Vector Search is now HIPAA-compliant in all regions except us-west-2 → Documentation
Meta Llama 3.1 405B models are now supported in Mosaic AI Model Training. → Documentation
You can now specify a subset of columns in a table to use in a vector search index. The primary key column and the embedding column are always synced. → Documentation
Foundation Model APIs pay-per-token is now GA: Pay-per-tokens models are accessible in your Databricks workspace → Documentation
Blog posts:
A Framework for Multi-Model Forecasting on Databricks: A recent blog introduced the Many Model Forecasting (MMF) framework for the comparative evaluation of forecasting models on Databricks. MMF enables users to train and predict using multiple forecasting models at scale on hundreds of thousands to many millions of time series at their finest granularity. → Blog Post + Demo below
Long Context RAG Performance of LLMs: With the availability of LLMs with longer context lengths, LLM app developers are able to feed more documents into their RAG applications. The Mosaic AI Research team explored the impact of increased context length on the quality of RAG applications. → Blog Post + Deep dive
Beyond the Leaderboard: Unpacking Function Calling Evaluation: This blog post deep dives into function calling, what it is and how to evaluate it. It outlines strategies for improving an LLM's function calling and tool use abilities.→ Blog Post + Deep Dive
Data Engineering
Databricks Workflows
For each task is GA: The For each task is now generally available, allowing users to run another task in a loop, passing a different set of parameters to each iteration of the task. This feature enables users to automate repetitive tasks and workflows. Learn more
Serverless SQL warehouses with the compliance security profile is now GA: Support for serverless SQL warehouses with the compliance security profile is now GA, enabling secure and compliant data analysis. This feature provides a secure and scalable way to analyze data in a serverless environment. Learn more
Databricks Runtime
Databricks Runtime 15.4 LTS is GA: Databricks Runtime 15.4 LTS and Databricks Runtime 15.4 LTS ML are now generally available, providing improved performance and stability. This release includes several new features, including improved support for Apache Spark 3.3 and enhanced security features. Learn more
Governance & Security
Row filters and column masks are now GA: The ability to apply row filters and column masks to tables is now generally available, preventing access to sensitive data by specified users. This feature enables users to control access to sensitive data and ensure compliance with data governance policies. Learn more
Canadian Centre for Cybersecurity (CCCS) Medium (Protected B) compliance controls: CCCS Medium controls provide enhancements that help with compliance for your workspace. This feature provides a set of security controls that are designed to meet the requirements of the Canadian Centre for Cybersecurity (CCCS) Medium (Protected B) compliance standard. Learn more
Create a new workspace with enhanced security and compliance features: Users can now enable the compliance security profile, add compliance standards, and enable enhanced security monitoring during workspace creation. This feature enables users to create a secure and compliant workspace that meets the requirements of various compliance standards. Learn more
Compute & Data Engineering
Clusters API now supports partial configuration updates: A new API call allows users to partially update a cluster configuration, requiring only the attributes to be updated. This feature enables users to update cluster configurations more efficiently and reduce downtime. Learn more
Lakehouse Federation
Lakehouse Federation is GA: Lakehouse Federation connectors across various database types are now generally available, enabling seamless data integration and analysis. This feature enables users to integrate data from multiple sources and analyze it in a single platform. Learn more
Databricks Clean Rooms
Databricks Clean Rooms (Public Preview): Databricks Clean Rooms uses Delta Sharing and serverless compute to provide a secure and privacy-protecting environment for collaborating on data projects. This feature enables users to collaborate on data projects while maintaining data privacy and security. Learn more
Other Updates
Databricks JDBC driver 2.6.40: Databricks JDBC Driver version 2.6.40 is now available, with enhancements and new features such as OIDC discovery endpoint support and updated Arrow support. This release includes several bug fixes and performance improvements. Learn more
Wrap lines in notebook cells: Users can now enable or disable line wrapping in notebook cells, allowing text to either wrap onto multiple lines or remain on a single line with horizontal scrolling. This feature enables users to customize the appearance of their notebooks. Learn more
Databricks SQL
Serverless SQL warehouse with Compliance Security Profile (GA): Customers who use compliance security profile on Databricks to process data that is regulated under compliance standards such as HIPAA, PCI-DSS, FedRAMP etc can now use Serverless SQL. Please note that support is dependent on region Learn more.