Skip to content

Kedro Databricks

uv Ruff License: MIT codecov Python Version Download/Month PyPI Version Read the Docs

Kedro plugin to develop Kedro pipelines for Databricks. This plugin strives to provide an excellent developer experience when using Kedro on Databricks.

Key Features

  1. Initialization: Transform your local Kedro project into a Databricks Asset Bundle.
  2. Generation: Generate Asset Bundle resource definitions from your Kedro pipelines.
  3. Deployment: Deploy your Kedro pipelines to Databricks as Jobs.
  4. Execution: Run your Kedro pipelines on Databricks straight from the command line.
  5. Cleanup: Remove all Databricks resources created by the plugin.

Advanced Capabilities

  • Resource generation modes (node and pipeline) with support for custom generators.
  • Flexible override model with defaults, named overrides, and regex-based overrides.
  • Support for non-job resources (for example volumes) in conf/<env>/databricks.yml.
  • Databricks CLI passthrough (-- ...) for advanced target/profile control.
  • Automatic local data upload during deploy when _file_path is configured.

For practical examples and deep-dive configuration patterns, see the User Guide.