airflow_dbt_python.hooks
Core dbt hooks
Provides a hook to interact with a dbt project.
- class airflow_dbt_python.hooks.dbt.DbtHook(*args, dbt_conn_id=None, project_conn_id=None, profiles_conn_id=None, **kwargs)[source]
A hook to interact with dbt.
Allows for running dbt tasks and provides required configurations for each task.
- Parameters:
dbt_conn_id (Optional[str])
project_conn_id (Optional[str])
profiles_conn_id (Optional[str])
- dbt_directory(config, upload_dbt_project=False, delete_before_upload=False, replace_on_upload=False, env_vars=None)[source]
Provides a temporary directory to execute dbt.
Creates a temporary directory for dbt to run in and prepares the dbt files if they need to be pulled from S3. If a S3 backend is being used, and self.upload_dbt_project is True, before leaving the temporary directory, we push back the project to S3. Pushing back a project enables commands like deps or docs generate.
- Yields:
The temporary directory’s name.
- Parameters:
upload_dbt_project (bool)
delete_before_upload (bool)
replace_on_upload (bool)
env_vars (Dict[str, Any] | None)
- Return type:
Iterator[str]
- download_dbt_profiles(profiles_dir, destination)[source]
Pull a dbt profiles.yml file from a given profiles_dir.
This operation is delegated to a DbtFSHook. An optional connection id is supported for remotes that require it.
- Parameters:
profiles_dir (URLLike)
destination (URLLike)
- Return type:
Path
- download_dbt_project(project_dir, destination)[source]
Pull a dbt project from a given project_dir.
This operation is delegated to a DbtFSHook. An optional connection id is supported for remotes that require it.
- Parameters:
project_dir (URLLike)
destination (URLLike)
- Return type:
Path
- static get_dbt_target_hook(conn_id)[source]
Get a hook to get a dbt profile based on the Airflow connection.
- Parameters:
conn_id (str)
- Return type:
DbtConnectionHook
- get_dbt_task_config(command, **config_kwargs)[source]
Initialize a configuration for given dbt command with given kwargs.
- Parameters:
command (str)
- Return type:
BaseConfig
- static get_fs_hook(scheme, conn_id)[source]
Get a fs_hook to interact with dbt files.
FSHooks are defined by the scheme we are looking for and an optional connection id if we are looking to interface with any Airflow hook that uses a connection.
- Parameters:
scheme (str)
conn_id (Optional[str])
- Return type:
DbtFSHook
- prepare_directory(tmp_dir, project_dir, profiles_dir=None)[source]
Prepares a dbt directory for execution of a dbt task.
Preparation involves downloading the required dbt project files and profiles.yml.
- Parameters:
tmp_dir (str)
project_dir (URLLike)
profiles_dir (Optional[URLLike])
- Return type:
tuple[str, Optional[str]]
- run_dbt_task(command, upload_dbt_project=False, delete_before_upload=False, replace_on_upload=False, artifacts=None, env_vars=None, write_perf_info=False, **kwargs)[source]
Run a dbt task with a given configuration and return the results.
The configuration used determines the task that will be ran.
- Returns:
- A tuple containing a boolean indicating success and optionally the results
of running the dbt command.
- Parameters:
command (str)
upload_dbt_project (bool)
delete_before_upload (bool)
replace_on_upload (bool)
artifacts (Iterable[str] | None)
env_vars (Dict[str, Any] | None)
write_perf_info (bool)
- Return type:
- setup_dbt_logging(debug)[source]
Setup dbt logging.
Starting with dbt v1, dbt initializes two loggers: default_file and default_stdout. As these are initialized by the CLI app, we need to initialize them here.
- Parameters:
debug (bool | None)
- upload_dbt_project(project_dir, destination, replace=False, delete_before=False)[source]
Push a dbt project from a given project_dir.
This operation is delegated to a DbtFSHook. An optional connection id is supported for remotes that require it.
- Parameters:
project_dir (URLLike)
destination (URLLike)
replace (bool)
delete_before (bool)
- Return type:
None
- class airflow_dbt_python.hooks.dbt.DbtTaskResult(success, run_results, artifacts)[source]
A tuple returned after a dbt task executes.
- Parameters:
success (bool)
run_results (Optional[RunResult])
artifacts (dict[str, Any])
- success
Whether the task succeeded or not.
- Type:
bool
- run_results
Results from the dbt task, if available.
- Type:
Optional[RunResult]
- artifacts
A dictionary of saved dbt artifacts. It may be empty.
- Type:
dict[str, Any]
- artifacts: dict[str, Any]
Alias for field number 2
- run_results: RunResult | None
Alias for field number 1
- success: bool
Alias for field number 0
- class airflow_dbt_python.hooks.dbt.DbtTemporaryDirectory(suffix=None, prefix=None, dir=None, ignore_cleanup_errors=True)[source]
A wrapper on TemporaryDirectory for older versions of Python.
Support for ignore_cleanup_errors was added in Python 3.10. There is a very obscure error that can happen when cleaning up a directory, even though everything should be cleaned. We would like to use ignore_cleanup_errors to provide clean up on a best-effort basis. For the time being, we are addressing this only for Python>=3.10.