trait SupportsPushDownVariantExtractions extends ScanBuilder
A mix-in interface for ScanBuilder. Data sources can implement this interface to
support pushing down variant field extraction operations to the data source.
When variant columns are accessed with specific field extractions (e.g., variant_get, try_variant_get), the optimizer can push these extractions down to the data source. The data source can then read only the required fields from variant columns, reducing I/O and improving performance.
Each VariantExtraction in the input array represents one field extraction operation.
Data sources should examine each extraction and determine which ones can be handled efficiently.
The return value is a boolean array of the same length, where each element indicates whether
the corresponding extraction was accepted.
- Annotations
- @Experimental()
- Source
- SupportsPushDownVariantExtractions.java
- Since
4.1.0
- Alphabetic
- By Inheritance
- SupportsPushDownVariantExtractions
- ScanBuilder
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def build(): Scan
- Definition Classes
- ScanBuilder
- abstract def pushVariantExtractions(extractions: Array[VariantExtraction]): Array[Boolean]
Pushes down variant field extractions to the data source.
Pushes down variant field extractions to the data source.
Each element in the input array represents one field extraction operation from a variant column. Data sources should examine each extraction and determine whether it can be pushed down based on the data source's capabilities (e.g., supported data types, path complexity, etc.).
The return value is a boolean array of the same length as the input array, where each element indicates whether the corresponding extraction was accepted:
- true: The extraction will be handled by the data source
- false: The extraction will be handled by Spark after reading
Data sources can choose to accept all, some, or none of the extractions. Spark will handle any extractions that are not pushed down.
- extractions
Array of variant extractions, one per field extraction operation
- returns
Boolean array indicating which extractions were accepted (same length as input)
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)