Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fast csp.Struct to Arrow record batch conversion #286

Open
timkpaine opened this issue Jun 19, 2024 · 0 comments
Open

Add fast csp.Struct to Arrow record batch conversion #286

timkpaine opened this issue Jun 19, 2024 · 0 comments
Labels
type: feature Issues and PRs related to new features

Comments

@timkpaine
Copy link
Member

Now that @arhamchopra has taken care of fast json conversion, we should move on to fast Apache Arrow conversion. We should have 2 mechanisms for conversion:

  • converting an individual struct, necessary but probably less efficient overall
  • collecting lists of structs and converting to record batches on either a timer or after a certain window size

Native arrow conversion will make certain use cases substantially faster, including in our csp-based webserver framework, but should also have benefits for other adapters like Kafka and websockets.

@timkpaine timkpaine added the type: feature Issues and PRs related to new features label Jun 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: feature Issues and PRs related to new features
Projects
None yet
Development

No branches or pull requests

1 participant