Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Iceberg connect test #24290

Open
wants to merge 3 commits into
base: dev
Choose a base branch
from
Open

Iceberg connect test #24290

wants to merge 3 commits into from
+204 −9

Conversation

mmaslankaprv
Copy link
Member

@mmaslankaprv mmaslankaprv commented Nov 25, 2024 •
edited
Loading

Added an example test using Redpanda Connect to populate Iceberg enabled topic with data and then verify the translation correctness.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.3.x
  • v24.2.x
  • v24.1.x

Release Notes

  • none

@mmaslankaprv mmaslankaprv force-pushed the iceberg-connect-test branch 2 times, most recently from 719e1e7 to c5c44ff Compare November 26, 2024 10:32
@mmaslankaprv
Copy link
Member Author

/dt

@mmaslankaprv mmaslankaprv marked this pull request as ready for review November 27, 2024 13:17
@mmaslankaprv
tests: added escaping table names in queries
e6412e6 
Some characters that may be used in topic names must be treated specialy
in SQL. Table names must be quoted when they contains the special
character. Quotation mark is specific to query engine. Added quotation
for table names for query engines used in tests.

Signed-off-by: Michał Maślanka <[email protected]>
@mmaslankaprv
tests: added queries for partition count and distinct offset count
e524a0d 
Signed-off-by: Michał Maślanka <[email protected]>
@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Nov 27, 2024 •
edited
Loading

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58878#01936e11-9188-4310-ac15-3bb092bc217c:

"rptest.tests.datalake.simple_connect_test.RedpandaConnectIcebergTest.test_translating_avro_serialized_records.cloud_storage_type=CloudStorageType.S3"

non flaky failures in https://buildkite.com/redpanda/redpanda/builds/58898#01936ee5-e216-4a27-85ea-74b00865d3ae:

"rptest.transactions.compaction_e2e_test.CompactionE2EIdempotencyTest.test_basic_compaction.initial_cleanup_policy=compact.workload=Workload.TX_UNIQUE_KEYS"

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Nov 27, 2024 •
edited
Loading

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58878#01936e0d-a706-47ad-a0d0-48f9423cfe50
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58878#01936e11-918d-4a97-bc9b-60a2085de43d
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/58898#01936ee3-1d97-44ab-a3e7-f40ebac3db07

@vbotbuildovich
Copy link
Collaborator

Retry command for Build#58878

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/tests/datalake/simple_connect_test.py::RedpandaConnectIcebergTest.test_translating_avro_serialized_records@{"cloud_storage_type":1}

@vbotbuildovich
Copy link
Collaborator

the below tests from https://buildkite.com/redpanda/redpanda/builds/58898#01936e89-baf5-4dca-a7b1-ae7d290fbbdf have failed and will be retried

gtest_raft_rpunit

@vbotbuildovich
Copy link
Collaborator

Retry command for Build#58898

please wait until all jobs are finished before running the slash command

/ci-repeat 1
tests/rptest/transactions/compaction_e2e_test.py::CompactionE2EIdempotencyTest.test_basic_compaction@{"initial_cleanup_policy":"compact","workload":"TX_UNIQUE_KEYS"}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bharathv bharathv Awaiting requested review from bharathv

@andrwng andrwng Awaiting requested review from andrwng

@nvartolomei nvartolomei Awaiting requested review from nvartolomei

@ztlpn ztlpn Awaiting requested review from ztlpn

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mmaslankaprv @vbotbuildovich