Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Support for Custom Pool Selection in dbt-Cosmos for Airflow #1325

Open
1 task done
wornjs opened this issue Nov 15, 2024 · 4 comments
Open
1 task done

[Feature] Support for Custom Pool Selection in dbt-Cosmos for Airflow #1325

wornjs opened this issue Nov 15, 2024 · 4 comments
Labels
area:config Related to configuration, like YAML files, environment variables, or executer configuration do-not-stale Related to stale job and dosubot enhancement New feature or request triage-needed Items need to be reviewed / assigned to milestone

Comments

@wornjs
Copy link
Contributor

wornjs commented Nov 15, 2024

Description

Title: Support for Custom Pool Selection in dbt-Cosmos for Airflow

Description:
Currently, when using dbt-Cosmos, we are limited to a specific pool in Airflow. However, we would like to schedule models based on their characteristics, as follows:

Pool Size 1: For heavy tasks to be processed one at a time, dedicating DW resources exclusively to a single task.
Pool Size 10: For lightweight tasks, allowing multiple queries to be executed concurrently.
If a model does not specify a pool, it should use the default pool. However, if a pool is defined in the model’s configuration, that pool should be applied to the task.

  - name: model_name
    description: description
    meta:
      owner: "owner"
      cosmos: 
         pool: custom_pool

This feature would allow for better resource management and parallelization based on the workload requirements of each model.

Additionally, it would be useful to allow more flexible configurations under meta.cosmos, such as specifying multiple values for dbt-Cosmos use. For example, setting a custom warehouse in Snowflake.

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

  • Yes, I am willing to submit a PR!
@wornjs wornjs added enhancement New feature or request triage-needed Items need to be reviewed / assigned to milestone labels Nov 15, 2024
@dosubot dosubot bot added the area:config Related to configuration, like YAML files, environment variables, or executer configuration label Nov 15, 2024
@pankajastro
Copy link
Contributor

This looks cool! Please let me know when you have something ready; I would love to review and test it.

@pankajastro pankajastro added the do-not-stale Related to stale job and dosubot label Nov 18, 2024
@pankajastro
Copy link
Contributor

Hey @wornjs, would you like to draft a PR for this? I'm happy to test and merge it.

@wornjs wornjs mentioned this issue Nov 25, 2024
Open
2 tasks
@wornjs
Copy link
Contributor Author

wornjs commented Nov 25, 2024

@pankajastro
Sorry for the delay. Could you take a look at this PR
#1339

@pankajastro
Copy link
Contributor

@pankajastro Sorry for the delay. Could you take a look at this PR #1339

Hey, @wornjs, Thank you for the PR, I'll review it soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:config Related to configuration, like YAML files, environment variables, or executer configuration do-not-stale Related to stale job and dosubot enhancement New feature or request triage-needed Items need to be reviewed / assigned to milestone
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pankajastro @wornjs