Bump python sdk version

[eric-wang-1990]

Upgrade python sdk version from 0.17.0 to 0.36.0
Create DatabricksCredentialManager to managed credentials, under the hood use Config from databricks.sdk.core to manage the auth part which comes from the Python SDK.
For dbt we support 4 different auth mode:

• token : When user explicitly passed in a token=xxxx
• external_browser: When user pass in auth_type=oauth with clientid=xxx and no client secret
• oauth-m2m: When user pass in clientid=xxx and clientsecret=yyy
• azure-client-secret: When user pass in azure_client_id=xxx and azure_client_secret=yyy and your host is an azure host

For oauth-m2m and azure-client-secret we set a preferred auth sequence based on if the clientSecret starts with "dose", so we can reduce the number of false trying as much as possible.

Remove the set/get sharedPassword part, since the python sdk already handles that, but with one defect which I already raise a PR against: databricks/databricks-sdk-py#823

With this change one thing that will no longer work is if customer are using non-databricks OAuth client for U2M case it will not work, since the python sdk does not support modify redirectUrl/scopes.

Description

Checklist

• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

[eric-wang-1990]

Is it safe to remove it seems databricks cli will not use .netrc anymore? https://docs.databricks.com/en/dev-tools/cli/profiles.html @benc-db @jackyhu-db

[jackyhu-db]

I think .netrc is used by python requests which might be used by databricks-sdk code as well?

[eric-wang-1990]

Yeah you're right, seems dbt-databricks use requests library, even though databricks-sdk does not rely on .netrc, the local requests library made from dbt-databricks still read from .netrc without this fix. Will keep it as is.

[benc-db]

yeah, the .netrc thing bit us in the past.

[eric-wang-1990]

I am just honoring the existing behavior. If there are multiple connections using the same DatabricksCredentials instance, there is a shared resource _config which can have concurrent write. @benc-db I think you introduce the original lock, is it still needed?

[benc-db]

Looks like something is off in the python path, with those E2E tests failing.