airflow_dbt_python.hooks

Core dbt hooks

Provides a hook to interact with a dbt project.

class airflow_dbt_python.hooks.dbt.DbtHook(*args, dbt_conn_id=None, project_conn_id=None, profiles_conn_id=None, **kwargs)[source]

A hook to interact with dbt.

Allows for running dbt tasks and provides required configurations for each task.

Parameters:
  • dbt_conn_id (Optional[str])

  • project_conn_id (Optional[str])

  • profiles_conn_id (Optional[str])

dbt_directory(config, upload_dbt_project=False, delete_before_upload=False, replace_on_upload=False, env_vars=None)[source]

Provides a temporary directory to execute dbt.

Creates a temporary directory for dbt to run in and prepares the dbt files if they need to be pulled from S3. If a S3 backend is being used, and self.upload_dbt_project is True, before leaving the temporary directory, we push back the project to S3. Pushing back a project enables commands like deps or docs generate.

Yields:

The temporary directory’s name.

Parameters:
  • upload_dbt_project (bool)

  • delete_before_upload (bool)

  • replace_on_upload (bool)

  • env_vars (Dict[str, Any] | None)

Return type:

Iterator[str]

download_dbt_profiles(profiles_dir, destination)[source]

Pull a dbt profiles.yml file from a given profiles_dir.

This operation is delegated to a DbtFSHook. An optional connection id is supported for remotes that require it.

Parameters:
  • profiles_dir (URLLike)

  • destination (URLLike)

Return type:

Path

download_dbt_project(project_dir, destination)[source]

Pull a dbt project from a given project_dir.

This operation is delegated to a DbtFSHook. An optional connection id is supported for remotes that require it.

Parameters:
  • project_dir (URLLike)

  • destination (URLLike)

Return type:

Path

static get_dbt_target_hook(conn_id)[source]

Get a hook to get a dbt profile based on the Airflow connection.

Parameters:

conn_id (str)

Return type:

DbtConnectionHook

get_dbt_task_config(command, **config_kwargs)[source]

Initialize a configuration for given dbt command with given kwargs.

Parameters:

command (str)

Return type:

BaseConfig

static get_fs_hook(scheme, conn_id)[source]

Get a fs_hook to interact with dbt files.

FSHooks are defined by the scheme we are looking for and an optional connection id if we are looking to interface with any Airflow hook that uses a connection.

Parameters:
  • scheme (str)

  • conn_id (Optional[str])

Return type:

DbtFSHook

prepare_directory(tmp_dir, project_dir, profiles_dir=None)[source]

Prepares a dbt directory for execution of a dbt task.

Preparation involves downloading the required dbt project files and profiles.yml.

Parameters:
  • tmp_dir (str)

  • project_dir (URLLike)

  • profiles_dir (Optional[URLLike])

Return type:

tuple[str, Optional[str]]

run_dbt_task(command, upload_dbt_project=False, delete_before_upload=False, replace_on_upload=False, artifacts=None, env_vars=None, write_perf_info=False, **kwargs)[source]

Run a dbt task with a given configuration and return the results.

The configuration used determines the task that will be ran.

Returns:

A tuple containing a boolean indicating success and optionally the results

of running the dbt command.

Parameters:
  • command (str)

  • upload_dbt_project (bool)

  • delete_before_upload (bool)

  • replace_on_upload (bool)

  • artifacts (Iterable[str] | None)

  • env_vars (Dict[str, Any] | None)

  • write_perf_info (bool)

Return type:

DbtTaskResult

setup_dbt_logging(debug)[source]

Setup dbt logging.

Starting with dbt v1, dbt initializes two loggers: default_file and default_stdout. As these are initialized by the CLI app, we need to initialize them here.

Parameters:

debug (bool | None)

upload_dbt_project(project_dir, destination, replace=False, delete_before=False)[source]

Push a dbt project from a given project_dir.

This operation is delegated to a DbtFSHook. An optional connection id is supported for remotes that require it.

Parameters:
  • project_dir (URLLike)

  • destination (URLLike)

  • replace (bool)

  • delete_before (bool)

Return type:

None

class airflow_dbt_python.hooks.dbt.DbtTaskResult(success, run_results, artifacts)[source]

A tuple returned after a dbt task executes.

Parameters:
  • success (bool)

  • run_results (Optional[RunResult])

  • artifacts (dict[str, Any])

success

Whether the task succeeded or not.

Type:

bool

run_results

Results from the dbt task, if available.

Type:

Optional[RunResult]

artifacts

A dictionary of saved dbt artifacts. It may be empty.

Type:

dict[str, Any]

artifacts: dict[str, Any]

Alias for field number 2

run_results: RunResult | None

Alias for field number 1

success: bool

Alias for field number 0

class airflow_dbt_python.hooks.dbt.DbtTemporaryDirectory(suffix=None, prefix=None, dir=None, ignore_cleanup_errors=True)[source]

A wrapper on TemporaryDirectory for older versions of Python.

Support for ignore_cleanup_errors was added in Python 3.10. There is a very obscure error that can happen when cleaning up a directory, even though everything should be cleaned. We would like to use ignore_cleanup_errors to provide clean up on a best-effort basis. For the time being, we are addressing this only for Python>=3.10.

Supported dbt remote hooks

The DbtFSHook interface

dbt git remote

dbt localfs remote

dbt S3 remote

dbt GCS remote