Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[server][controller] Add MaterializedViewWriter and support view writers in L/F #1296

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Commits on Nov 22, 2024

  1. [server][controller] Add MaterializedViewWriter and support view writ…

    …ers in L/F
    
    1. View writers will be invoked in L/F SIT too instead of only in A/A SIT. We rely on
    view config validation to ensure views that do require A/A are only added to stores
    with A/A enabled.
    
    2. This PR only includes creation of materialized view topics, writing of data
    records and control messages to the materialized view topics  in server and controller.
      - Materialized view topics are created during version creation time along with other
        view topics.
      - SOP is sent during view topic creation time with same chunking and compression
        configs as the store version.
      - EOP is sent when servers have reported EOP in every partition.
      - Incremental push control messages SOIP and EOIP are not propagated to the view topic
        for now because the end to end incremental push tracking story for view topics is
        not clear yet. Store owners will likely just disable the requirement to wait for
        view consumers to fully ingest the incremental push.
      - Ingestion heartbeats will be propagated in a broadcast manner. See implementation
        for details.
      - Version swap for CDC users will be implemented in a separate PR to keep this PR
        somewhat short for review.
    
    3. TODO: one pending issue to be resolved is that during processing of batch records
    in the native replication source fabric, where we consume local VT, a leader transfer
    could result in missing records in the materialized view topic. This is because we
    don't do any global checkpointing across leader and followers when consuming local VT.
    xunyin8 committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    198c3ae View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d180bd6 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2024

  1. Configuration menu
    Copy the full SHA
    6b541a1 View commit details
    Browse the repository at this point in the history

Commits on Nov 25, 2024

  1. Added unit tests

    xunyin8 committed Nov 25, 2024
    Configuration menu
    Copy the full SHA
    bcee9bc View commit details
    Browse the repository at this point in the history