Package org.apache.spark.sql.connector.catalog.functions

Interface ReducibleFunction<I,O>

Type Parameters:
I - reducer function input type
O - reducer function output type
@Evolving public interface ReducibleFunction<I,O>
Base class for user-defined functions that can be 'reduced' on another function. A function f_source(x) is 'reducible' on another function f_target(x) if
  • There exists a reducer function r(x) such that r(f_source(x)) = f_target(x) for all input x, or
  • More generally, there exists reducer functions r1(x) and r2(x) such that r1(f_source(x)) = r2(f_target(x)) for all input x.

Examples:

  • Bucket functions where one side has reducer
    • f_source(x) = bucket(4, x)
    • f_target(x) = bucket(2, x)
    • r(x) = x % 2
  • Bucket functions where both sides have reducer
    • f_source(x) = bucket(16, x)
    • f_target(x) = bucket(12, x)
    • r1(x) = x % 4
    • r2(x) = x % 4
  • Date functions
    • f_source(x) = days(x)
    • f_target(x) = hours(x)
    • r(x) = x / 24
Since:
4.0.0

  • Method Summary

    Modifier and Type
    Method
    Description
    default Reducer<I,O>
    reducer(int thisNumBuckets, ReducibleFunction<?,?> otherBucketFunction, int otherNumBuckets)
    This method is for the bucket function.
    default Reducer<I,O>
    reducer(ReducibleFunction<?,?> otherFunction)
    This method is for all other functions.

  • Method Details

    • reducer

      default Reducer<I,O> reducer(int thisNumBuckets, ReducibleFunction<?,?> otherBucketFunction, int otherNumBuckets)
      This method is for the bucket function. If this bucket function is 'reducible' on another bucket function, return the Reducer function.

      For example, to return reducer for reducing f_source = bucket(4, x) on f_target = bucket(2, x)

      • thisBucketFunction = bucket
      • thisNumBuckets = 4
      • otherBucketFunction = bucket
      • otherNumBuckets = 2
      Parameters:
      thisNumBuckets - parameter for this function
      otherBucketFunction - the other parameterized function
      otherNumBuckets - parameter for the other function
      Returns:
      a reduction function if it is reducible, null if not

    • reducer

      default Reducer<I,O> reducer(ReducibleFunction<?,?> otherFunction)
      This method is for all other functions. If this function is 'reducible' on another function, return the Reducer function.

      Example of reducing f_source = days(x) on f_target = hours(x)

      • thisFunction = days
      • otherFunction = hours
      Parameters:
      otherFunction - the other function
      Returns:
      a reduction function if it is reducible, null if not.