== Physical Plan ==
AdaptiveSparkPlan (11)
+- == Final Plan ==
ResultQueryStage (7), Statistics(sizeInBytes=8.0 EiB)
+- ObjectHashAggregate (6)
+- ShuffleQueryStage (5), Statistics(sizeInBytes=10.6 KiB, rowCount=50)
+- Exchange (4)
+- ObjectHashAggregate (3)
+- * Project (2)
+- * Scan ExistingRDD Delta Table State #2 - hdlfs://9b46f9d5-066e-49af-bc5c-7af37534ca4d.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-plant-plant/_delta_log (1)
+- == Initial Plan ==
ObjectHashAggregate (10)
+- Exchange (9)
+- ObjectHashAggregate (8)
+- Project (2)
+- Scan ExistingRDD Delta Table State #2 - hdlfs://9b46f9d5-066e-49af-bc5c-7af37534ca4d.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-plant-plant/_delta_log (1)
(1) Scan ExistingRDD Delta Table State #2 - hdlfs://9b46f9d5-066e-49af-bc5c-7af37534ca4d.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-plant-plant/_delta_log [codegen id : 1]
Output [10]: [txn#2920, add#2921, remove#2922, metaData#2923, protocol#2924, cdc#2925, checkpointMetadata#2926, sidecar#2927, domainMetadata#2928, commitInfo#2929]
Arguments: [txn#2920, add#2921, remove#2922, metaData#2923, protocol#2924, cdc#2925, checkpointMetadata#2926, sidecar#2927, domainMetadata#2928, commitInfo#2929], Delta Table State #2 - hdlfs://9b46f9d5-066e-49af-bc5c-7af37534ca4d.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-plant-plant/_delta_log MapPartitionsRDD[162] at $anonfun$recordDeltaOperationInternal$1 at DatabricksLogging.scala:128, ExistingRDD, UnknownPartitioning(0)
(2) Project [codegen id : 1]
Output [6]: [txn#2920, add#2921, remove#2922, metaData#2923, protocol#2924, domainMetadata#2928]
Input [10]: [txn#2920, add#2921, remove#2922, metaData#2923, protocol#2924, cdc#2925, checkpointMetadata#2926, sidecar#2927, domainMetadata#2928, commitInfo#2929]
(3) ObjectHashAggregate
Input [6]: [txn#2920, add#2921, remove#2922, metaData#2923, protocol#2924, domainMetadata#2928]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#2928, 0, 0), partial_collect_set(txn#2920, 0, 0), partial_count(protocol#2924), partial_sum(add#2921.size), partial_last(metaData#2923, true), partial_count(metaData#2923), partial_last(protocol#2924, true), partial_count(remove#2922), partial_count(add#2921), partial_count(txn#2920)]
Aggregate Attributes [12]: [buf#3030, buf#3031, count#3032L, sum#3033L, last#3034, valueSet#3035, count#3036L, last#3037, valueSet#3038, count#3039L, count#3040L, count#3041L]
Results [12]: [buf#3042, buf#3043, count#3044L, sum#3045L, last#3046, valueSet#3047, count#3048L, last#3049, valueSet#3050, count#3051L, count#3052L, count#3053L]
(4) Exchange
Input [12]: [buf#3042, buf#3043, count#3044L, sum#3045L, last#3046, valueSet#3047, count#3048L, last#3049, valueSet#3050, count#3051L, count#3052L, count#3053L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=1236]
(5) ShuffleQueryStage
Output [12]: [buf#3042, buf#3043, count#3044L, sum#3045L, last#3046, valueSet#3047, count#3048L, last#3049, valueSet#3050, count#3051L, count#3052L, count#3053L]
Arguments: 0
(6) ObjectHashAggregate
Input [12]: [buf#3042, buf#3043, count#3044L, sum#3045L, last#3046, valueSet#3047, count#3048L, last#3049, valueSet#3050, count#3051L, count#3052L, count#3053L]
Keys: []
Functions [10]: [collect_set(domainMetadata#2928, 0, 0), collect_set(txn#2920, 0, 0), count(protocol#2924), sum(add#2921.size), last(metaData#2923, true), count(metaData#2923), last(protocol#2924, true), count(remove#2922), count(add#2921), count(txn#2920)]
Aggregate Attributes [10]: [collect_set(domainMetadata#2928, 0, 0)#2976, collect_set(txn#2920, 0, 0)#2977, count(protocol#2924)#2978L, sum(add#2921.size)#2979L, last(metaData#2923) ignore nulls#2980, count(metaData#2923)#2981L, last(protocol#2924) ignore nulls#2982, count(remove#2922)#2983L, count(add#2921)#2984L, count(txn#2920)#2985L]
Results [14]: [collect_set(domainMetadata#2928, 0, 0)#2976 AS domainMetadata#2961, null AS fileSizeHistogram#2962, collect_set(txn#2920, 0, 0)#2977 AS setTransactions#2963, null AS numDeletedRecordsOpt#2964, count(protocol#2924)#2978L AS numOfProtocol#2965L, coalesce(sum(add#2921.size)#2979L, 0) AS sizeInBytes#2966L, last(metaData#2923) ignore nulls#2980 AS metadata#2967, count(metaData#2923)#2981L AS numOfMetadata#2968L, null AS deletedRecordCountsHistogramOpt#2969, null AS numDeletionVectorsOpt#2970, last(protocol#2924) ignore nulls#2982 AS protocol#2971, count(remove#2922)#2983L AS numOfRemoves#2972L, count(add#2921)#2984L AS numOfFiles#2973L, count(txn#2920)#2985L AS numOfSetTransactions#2974L]
(7) ResultQueryStage
Output [14]: [domainMetadata#2961, fileSizeHistogram#2962, setTransactions#2963, numDeletedRecordsOpt#2964, numOfProtocol#2965L, sizeInBytes#2966L, metadata#2967, numOfMetadata#2968L, deletedRecordCountsHistogramOpt#2969, numDeletionVectorsOpt#2970, protocol#2971, numOfRemoves#2972L, numOfFiles#2973L, numOfSetTransactions#2974L]
Arguments: 1
(8) ObjectHashAggregate
Input [6]: [txn#2920, add#2921, remove#2922, metaData#2923, protocol#2924, domainMetadata#2928]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#2928, 0, 0), partial_collect_set(txn#2920, 0, 0), partial_count(protocol#2924), partial_sum(add#2921.size), partial_last(metaData#2923, true), partial_count(metaData#2923), partial_last(protocol#2924, true), partial_count(remove#2922), partial_count(add#2921), partial_count(txn#2920)]
Aggregate Attributes [12]: [buf#3030, buf#3031, count#3032L, sum#3033L, last#3034, valueSet#3035, count#3036L, last#3037, valueSet#3038, count#3039L, count#3040L, count#3041L]
Results [12]: [buf#3042, buf#3043, count#3044L, sum#3045L, last#3046, valueSet#3047, count#3048L, last#3049, valueSet#3050, count#3051L, count#3052L, count#3053L]
(9) Exchange
Input [12]: [buf#3042, buf#3043, count#3044L, sum#3045L, last#3046, valueSet#3047, count#3048L, last#3049, valueSet#3050, count#3051L, count#3052L, count#3053L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=1225]
(10) ObjectHashAggregate
Input [12]: [buf#3042, buf#3043, count#3044L, sum#3045L, last#3046, valueSet#3047, count#3048L, last#3049, valueSet#3050, count#3051L, count#3052L, count#3053L]
Keys: []
Functions [10]: [collect_set(domainMetadata#2928, 0, 0), collect_set(txn#2920, 0, 0), count(protocol#2924), sum(add#2921.size), last(metaData#2923, true), count(metaData#2923), last(protocol#2924, true), count(remove#2922), count(add#2921), count(txn#2920)]
Aggregate Attributes [10]: [collect_set(domainMetadata#2928, 0, 0)#2976, collect_set(txn#2920, 0, 0)#2977, count(protocol#2924)#2978L, sum(add#2921.size)#2979L, last(metaData#2923) ignore nulls#2980, count(metaData#2923)#2981L, last(protocol#2924) ignore nulls#2982, count(remove#2922)#2983L, count(add#2921)#2984L, count(txn#2920)#2985L]
Results [14]: [collect_set(domainMetadata#2928, 0, 0)#2976 AS domainMetadata#2961, null AS fileSizeHistogram#2962, collect_set(txn#2920, 0, 0)#2977 AS setTransactions#2963, null AS numDeletedRecordsOpt#2964, count(protocol#2924)#2978L AS numOfProtocol#2965L, coalesce(sum(add#2921.size)#2979L, 0) AS sizeInBytes#2966L, last(metaData#2923) ignore nulls#2980 AS metadata#2967, count(metaData#2923)#2981L AS numOfMetadata#2968L, null AS deletedRecordCountsHistogramOpt#2969, null AS numDeletionVectorsOpt#2970, last(protocol#2924) ignore nulls#2982 AS protocol#2971, count(remove#2922)#2983L AS numOfRemoves#2972L, count(add#2921)#2984L AS numOfFiles#2973L, count(txn#2920)#2985L AS numOfSetTransactions#2974L]
(11) AdaptiveSparkPlan
Output [14]: [domainMetadata#2961, fileSizeHistogram#2962, setTransactions#2963, numDeletedRecordsOpt#2964, numOfProtocol#2965L, sizeInBytes#2966L, metadata#2967, numOfMetadata#2968L, deletedRecordCountsHistogramOpt#2969, numDeletionVectorsOpt#2970, protocol#2971, numOfRemoves#2972L, numOfFiles#2973L, numOfSetTransactions#2974L]
Arguments: isFinalPlan=true