[server][controller] Add MaterializedViewWriter and support view writers in L/F #1296
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xunyin8
force-pushed
the
RePartitionViewWriter
branch
2 times, most recently
from
e43850a
to
a711263
Compare
xunyin8
force-pushed
the
RePartitionViewWriter
branch
from
a711263
to
b9e01be
Compare
…ers in L/F
1. View writers will be invoked in L/F SIT too instead of only in A/A SIT. We rely on
view config validation to ensure views that do require A/A are only added to stores
with A/A enabled.
2. This PR only includes creation of materialized view topics, writing of data
records and control messages to the materialized view topics in server and controller.
- Materialized view topics are created during version creation time along with other
view topics.
- SOP is sent during view topic creation time with same chunking and compression
configs as the store version.
- EOP is sent when servers have reported EOP in every partition.
- Incremental push control messages SOIP and EOIP are not propagated to the view topic
for now because the end to end incremental push tracking story for view topics is
not clear yet. Store owners will likely just disable the requirement to wait for
view consumers to fully ingest the incremental push.
- Ingestion heartbeats will be propagated in a broadcast manner. See implementation
for details.
- Version swap for CDC users will be implemented in a separate PR to keep this PR
somewhat short for review.
3. TODO: one pending issue to be resolved is that during processing of batch records
in the native replication source fabric, where we consume local VT, a leader transfer
could result in missing records in the materialized view topic. This is because we
don't do any global checkpointing across leader and followers when consuming local VT.
xunyin8
force-pushed
the
RePartitionViewWriter
branch
2 times, most recently
from
24a7c24
to
0abfa3e
Compare
xunyin8
force-pushed
the
RePartitionViewWriter
branch
2 times, most recently
from
60f9774
to
791712b
Compare
xunyin8
force-pushed
the
RePartitionViewWriter
branch
from
791712b
to
bcee9bc
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[server][controller] Add MaterializedViewWriter and support view writers in L/F
View writers will be invoked in L/F SIT too instead of only in A/A SIT. We rely on view config validation to ensure views that do require A/A are only added to stores with A/A enabled.
This PR only includes creation of materialized view topics, writing of data records and control messages to the materialized view topics in server and controller.
How was this PR tested?
Unit and integration tests
Does this PR introduce any user-facing changes?