User Guide¶

This is the user guide for Kedro-Databricks. It contains detailed instructions on how to use the plugin, including how to customize it for your specific needs.

Configuration¶

You can change how kedro-databricks works by configuring it as a tool in your pyproject.toml. For example:

# pyproject.toml
[tool.kedro-databricks]
init_schema = "default"
init_catalog = "workspace"
default_env = "dev"

The available configuration options are:

Resource Generation¶

You can choose how resources are generated using -g/--resource-generator:

• node (default): creates a job task for each Kedro node with dependencies.
• pipeline: creates a single task that runs the entire pipeline.

You can also provide a fully-qualified dotted path to a custom generator class that subclasses kedro_databricks.resource_generator.AbstractResourceGenerator.

Examples:

# Generate per-node tasks (default behavior)
kedro databricks bundle -g node

# Generate a single-task job for the whole pipeline
kedro databricks bundle -g pipeline

# Bundle only one pipeline by name
kedro databricks bundle -g pipeline -p my_pipeline

# Pass runtime parameters to tasks
kedro databricks bundle -g node -r "param1=val1,param2=val2"

Tip: The same -g/--resource-generator, -p/--pipeline, and -r/--params options are also available when using kedro databricks deploy --bundle.

Creating a custom resource generator¶

You can implement your own generator by subclassing kedro_databricks.resource_generator.AbstractResourceGenerator and returning a Databricks job payload in _create_job_dict. For example, the snippet below creates per-node tasks and attaches a custom cluster:

# my_project/generators/custom.py
from __future__ import annotations
from typing import Any

from kedro.pipeline import Pipeline
from kedro_databricks.resource_generator.abstract_resource_generator import (
    AbstractResourceGenerator,
)


class CustomGenerator(AbstractResourceGenerator):
    """Example generator with a predefined job cluster per task."""

    def _create_job_dict(
        self, name: str, pipeline: Pipeline, pipeline_name: str
    ) -> dict[str, Any]:
        # Your custom logic
        return {"name": name, "tasks": []}

Use your custom generator by passing its dotted path with -g/--resource-generator:

kedro databricks bundle -g "my_project.generators.custom.CustomGenerator"

Override Behavior¶

Override files are read from conf/<env>/databricks.yml and merged into generated resources.

• resources.jobs.<job-name> applies to a specific generated job.
• resources.jobs.<default-key> applies to all jobs (default key is default, configurable through --default-key or project config).
• Regex keys are supported using the configured regex_prefix (default re:).

Example with job-level regex override:

resources:
    jobs:
        default:
            max_concurrent_runs: 1
        re:^my_project_.*:
            tags:
                owner: data-eng

Task-level regex overrides are also supported under resources.jobs.<job>.tasks:

resources:
    jobs:
        my_project_default:
            tasks:
            - task_key: default
                timeout_seconds: 7200
            - task_key: re:^train_.*
                max_retries: 2

Resource Type Support¶

You can define any Databricks resource type under resources.

• jobs use specialized merge logic (for tasks, environments, clusters, notifications, health rules, and parameters).
• Other resource types use shallow key-based merging and are written as-is into generated bundle files.

This makes it possible to combine generated jobs with manually maintained resources such as volumes, while still using advanced merge behavior for jobs.

Capability Matrix¶

The table below maps common capabilities to where they are documented and where you can find concrete examples.