Package org.apache.spark.sql.pipelines.graph

Class PipelineUpdateContextImpl

Object

org.apache.spark.sql.pipelines.graph.PipelineUpdateContextImpl

All Implemented Interfaces:

PipelineUpdateContext

public class PipelineUpdateContextImpl
extends Object
implements PipelineUpdateContext

An implementation of the PipelineUpdateContext trait used in production. param: unresolvedGraph The graph (unresolved) to be executed in this update. param: eventCallback A callback function to be called when an event is added to the event buffer. param: refreshTables Filter for which tables should be refreshed when performing this update. param: fullRefreshTables Filter for which tables should be full refreshed when performing this update.

Constructor Summary

Constructors

Constructor	Description
PipelineUpdateContextImpl(DataflowGraph unresolvedGraph, scala.Function1<PipelineEvent,scala.runtime.BoxedUnit> eventCallback, TableFilter refreshTables, TableFilter fullRefreshTables, String storageRoot)

Method Summary

Modifier and Type	Method	Description
scala.Function1<PipelineEvent,scala.runtime.BoxedUnit>	eventCallback()	Callback to invoke for internal events that are emitted during a run of a pipeline.
FlowProgressEventLogger	flowProgressEventLogger()	Emits internal flow progress events into the event buffer.
TableFilter	fullRefreshTables()	Filter for which tables should be full refreshed when performing this update.
PipelineExecution	pipelineExecution()	Defines operations relates to end to end execution of a `DataflowGraph`.
TableFilter	refreshTables()	Filter for which tables should be refreshed when performing this update.
FlowFilter	resetCheckpointFlows()
org.apache.spark.sql.classic.SparkSession	spark()	The SparkSession for this update.
String	storageRoot()	The root storage location for pipeline metadata, including checkpoints for streaming flows.
DataflowGraph	unresolvedGraph()	The unresolved graph for this update.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.sql.pipelines.graph.PipelineUpdateContext

refreshFlows

Constructor Details

PipelineUpdateContextImpl

public PipelineUpdateContextImpl(DataflowGraph unresolvedGraph,
 scala.Function1<PipelineEvent,scala.runtime.BoxedUnit> eventCallback,
 TableFilter refreshTables,
 TableFilter fullRefreshTables,
 String storageRoot)

Method Details

pipelineExecution

public PipelineExecution pipelineExecution()

Description copied from interface: PipelineUpdateContext

Defines operations relates to end to end execution of a `DataflowGraph`.

Specified by:

pipelineExecution in interface PipelineUpdateContext

unresolvedGraph

public DataflowGraph unresolvedGraph()

Description copied from interface: PipelineUpdateContext

The unresolved graph for this update.

Specified by:

unresolvedGraph in interface PipelineUpdateContext

eventCallback

public scala.Function1<PipelineEvent,scala.runtime.BoxedUnit> eventCallback()

Description copied from interface: PipelineUpdateContext

Callback to invoke for internal events that are emitted during a run of a pipeline.

Specified by:

eventCallback in interface PipelineUpdateContext

refreshTables

public TableFilter refreshTables()

Description copied from interface: PipelineUpdateContext

Filter for which tables should be refreshed when performing this update.

Specified by:

refreshTables in interface PipelineUpdateContext

fullRefreshTables

public TableFilter fullRefreshTables()

Description copied from interface: PipelineUpdateContext

Filter for which tables should be full refreshed when performing this update.

Specified by:

fullRefreshTables in interface PipelineUpdateContext

storageRoot

public String storageRoot()

Description copied from interface: PipelineUpdateContext

The root storage location for pipeline metadata, including checkpoints for streaming flows.

Specified by:

storageRoot in interface PipelineUpdateContext

Returns:

(undocumented)

spark

public org.apache.spark.sql.classic.SparkSession spark()

Description copied from interface: PipelineUpdateContext

The SparkSession for this update.

Specified by:

spark in interface PipelineUpdateContext

flowProgressEventLogger

public FlowProgressEventLogger flowProgressEventLogger()

Description copied from interface: PipelineUpdateContext

Emits internal flow progress events into the event buffer.

Specified by:

flowProgressEventLogger in interface PipelineUpdateContext

resetCheckpointFlows

public FlowFilter resetCheckpointFlows()

Specified by:

resetCheckpointFlows in interface PipelineUpdateContext