-
Notifications
You must be signed in to change notification settings - Fork 27
Open
Description
Does fenic provide data source APIs to integrate with the data storage? When a user runs a SQL that is meant to read a Hudi table, query engine will parse the SQL and push down the target columns and predicates to storage engine for returning a minimized list of files. For example, Hudi-rs supports integration with query engines through table and file group APIs (https://github.com/apache/hudi-rs?tab=readme-ov-file#query-engine-integration). Hudi-rs table APIs can be used by fenic to get a list of Hudi's file slices for further query planning, and use Hudi-rs file group APIs to read file slices and get Arrow's RecordBatches at execution.
rohitrastogi
Metadata
Metadata
Assignees
Labels
No labels