== Physical Plan ==
AdaptiveSparkPlan (11)
+- == Final Plan ==
ResultQueryStage (7), Statistics(sizeInBytes=8.0 EiB)
+- ObjectHashAggregate (6)
+- ShuffleQueryStage (5), Statistics(sizeInBytes=10.6 KiB, rowCount=50)
+- Exchange (4)
+- ObjectHashAggregate (3)
+- * Project (2)
+- * Scan ExistingRDD Delta Table State #2 - hdlfs://9b46f9d5-066e-49af-bc5c-7af37534ca4d.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-plant-plant/_delta_log (1)
+- == Initial Plan ==
ObjectHashAggregate (10)
+- Exchange (9)
+- ObjectHashAggregate (8)
+- Project (2)
+- Scan ExistingRDD Delta Table State #2 - hdlfs://9b46f9d5-066e-49af-bc5c-7af37534ca4d.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-plant-plant/_delta_log (1)
(1) Scan ExistingRDD Delta Table State #2 - hdlfs://9b46f9d5-066e-49af-bc5c-7af37534ca4d.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-plant-plant/_delta_log [codegen id : 1]
Output [10]: [txn#27707, add#27708, remove#27709, metaData#27710, protocol#27711, cdc#27712, checkpointMetadata#27713, sidecar#27714, domainMetadata#27715, commitInfo#27716]
Arguments: [txn#27707, add#27708, remove#27709, metaData#27710, protocol#27711, cdc#27712, checkpointMetadata#27713, sidecar#27714, domainMetadata#27715, commitInfo#27716], Delta Table State #2 - hdlfs://9b46f9d5-066e-49af-bc5c-7af37534ca4d.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-plant-plant/_delta_log MapPartitionsRDD[1437] at $anonfun$recordDeltaOperationInternal$1 at DatabricksLogging.scala:128, ExistingRDD, UnknownPartitioning(0)
(2) Project [codegen id : 1]
Output [6]: [txn#27707, add#27708, remove#27709, metaData#27710, protocol#27711, domainMetadata#27715]
Input [10]: [txn#27707, add#27708, remove#27709, metaData#27710, protocol#27711, cdc#27712, checkpointMetadata#27713, sidecar#27714, domainMetadata#27715, commitInfo#27716]
(3) ObjectHashAggregate
Input [6]: [txn#27707, add#27708, remove#27709, metaData#27710, protocol#27711, domainMetadata#27715]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#27715, 0, 0), partial_collect_set(txn#27707, 0, 0), partial_count(protocol#27711), partial_sum(add#27708.size), partial_last(metaData#27710, true), partial_count(metaData#27710), partial_last(protocol#27711, true), partial_count(remove#27709), partial_count(add#27708), partial_count(txn#27707)]
Aggregate Attributes [12]: [buf#27817, buf#27818, count#27819L, sum#27820L, last#27821, valueSet#27822, count#27823L, last#27824, valueSet#27825, count#27826L, count#27827L, count#27828L]
Results [12]: [buf#27829, buf#27830, count#27831L, sum#27832L, last#27833, valueSet#27834, count#27835L, last#27836, valueSet#27837, count#27838L, count#27839L, count#27840L]
(4) Exchange
Input [12]: [buf#27829, buf#27830, count#27831L, sum#27832L, last#27833, valueSet#27834, count#27835L, last#27836, valueSet#27837, count#27838L, count#27839L, count#27840L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=113223]
(5) ShuffleQueryStage
Output [12]: [buf#27829, buf#27830, count#27831L, sum#27832L, last#27833, valueSet#27834, count#27835L, last#27836, valueSet#27837, count#27838L, count#27839L, count#27840L]
Arguments: 0
(6) ObjectHashAggregate
Input [12]: [buf#27829, buf#27830, count#27831L, sum#27832L, last#27833, valueSet#27834, count#27835L, last#27836, valueSet#27837, count#27838L, count#27839L, count#27840L]
Keys: []
Functions [10]: [collect_set(domainMetadata#27715, 0, 0), collect_set(txn#27707, 0, 0), count(protocol#27711), sum(add#27708.size), last(metaData#27710, true), count(metaData#27710), last(protocol#27711, true), count(remove#27709), count(add#27708), count(txn#27707)]
Aggregate Attributes [10]: [collect_set(domainMetadata#27715, 0, 0)#27763, collect_set(txn#27707, 0, 0)#27764, count(protocol#27711)#27765L, sum(add#27708.size)#27766L, last(metaData#27710) ignore nulls#27767, count(metaData#27710)#27768L, last(protocol#27711) ignore nulls#27769, count(remove#27709)#27770L, count(add#27708)#27771L, count(txn#27707)#27772L]
Results [14]: [collect_set(domainMetadata#27715, 0, 0)#27763 AS domainMetadata#27748, null AS fileSizeHistogram#27749, collect_set(txn#27707, 0, 0)#27764 AS setTransactions#27750, null AS numDeletedRecordsOpt#27751, count(protocol#27711)#27765L AS numOfProtocol#27752L, coalesce(sum(add#27708.size)#27766L, 0) AS sizeInBytes#27753L, last(metaData#27710) ignore nulls#27767 AS metadata#27754, count(metaData#27710)#27768L AS numOfMetadata#27755L, null AS deletedRecordCountsHistogramOpt#27756, null AS numDeletionVectorsOpt#27757, last(protocol#27711) ignore nulls#27769 AS protocol#27758, count(remove#27709)#27770L AS numOfRemoves#27759L, count(add#27708)#27771L AS numOfFiles#27760L, count(txn#27707)#27772L AS numOfSetTransactions#27761L]
(7) ResultQueryStage
Output [14]: [domainMetadata#27748, fileSizeHistogram#27749, setTransactions#27750, numDeletedRecordsOpt#27751, numOfProtocol#27752L, sizeInBytes#27753L, metadata#27754, numOfMetadata#27755L, deletedRecordCountsHistogramOpt#27756, numDeletionVectorsOpt#27757, protocol#27758, numOfRemoves#27759L, numOfFiles#27760L, numOfSetTransactions#27761L]
Arguments: 1
(8) ObjectHashAggregate
Input [6]: [txn#27707, add#27708, remove#27709, metaData#27710, protocol#27711, domainMetadata#27715]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#27715, 0, 0), partial_collect_set(txn#27707, 0, 0), partial_count(protocol#27711), partial_sum(add#27708.size), partial_last(metaData#27710, true), partial_count(metaData#27710), partial_last(protocol#27711, true), partial_count(remove#27709), partial_count(add#27708), partial_count(txn#27707)]
Aggregate Attributes [12]: [buf#27817, buf#27818, count#27819L, sum#27820L, last#27821, valueSet#27822, count#27823L, last#27824, valueSet#27825, count#27826L, count#27827L, count#27828L]
Results [12]: [buf#27829, buf#27830, count#27831L, sum#27832L, last#27833, valueSet#27834, count#27835L, last#27836, valueSet#27837, count#27838L, count#27839L, count#27840L]
(9) Exchange
Input [12]: [buf#27829, buf#27830, count#27831L, sum#27832L, last#27833, valueSet#27834, count#27835L, last#27836, valueSet#27837, count#27838L, count#27839L, count#27840L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=113212]
(10) ObjectHashAggregate
Input [12]: [buf#27829, buf#27830, count#27831L, sum#27832L, last#27833, valueSet#27834, count#27835L, last#27836, valueSet#27837, count#27838L, count#27839L, count#27840L]
Keys: []
Functions [10]: [collect_set(domainMetadata#27715, 0, 0), collect_set(txn#27707, 0, 0), count(protocol#27711), sum(add#27708.size), last(metaData#27710, true), count(metaData#27710), last(protocol#27711, true), count(remove#27709), count(add#27708), count(txn#27707)]
Aggregate Attributes [10]: [collect_set(domainMetadata#27715, 0, 0)#27763, collect_set(txn#27707, 0, 0)#27764, count(protocol#27711)#27765L, sum(add#27708.size)#27766L, last(metaData#27710) ignore nulls#27767, count(metaData#27710)#27768L, last(protocol#27711) ignore nulls#27769, count(remove#27709)#27770L, count(add#27708)#27771L, count(txn#27707)#27772L]
Results [14]: [collect_set(domainMetadata#27715, 0, 0)#27763 AS domainMetadata#27748, null AS fileSizeHistogram#27749, collect_set(txn#27707, 0, 0)#27764 AS setTransactions#27750, null AS numDeletedRecordsOpt#27751, count(protocol#27711)#27765L AS numOfProtocol#27752L, coalesce(sum(add#27708.size)#27766L, 0) AS sizeInBytes#27753L, last(metaData#27710) ignore nulls#27767 AS metadata#27754, count(metaData#27710)#27768L AS numOfMetadata#27755L, null AS deletedRecordCountsHistogramOpt#27756, null AS numDeletionVectorsOpt#27757, last(protocol#27711) ignore nulls#27769 AS protocol#27758, count(remove#27709)#27770L AS numOfRemoves#27759L, count(add#27708)#27771L AS numOfFiles#27760L, count(txn#27707)#27772L AS numOfSetTransactions#27761L]
(11) AdaptiveSparkPlan
Output [14]: [domainMetadata#27748, fileSizeHistogram#27749, setTransactions#27750, numDeletedRecordsOpt#27751, numOfProtocol#27752L, sizeInBytes#27753L, metadata#27754, numOfMetadata#27755L, deletedRecordCountsHistogramOpt#27756, numDeletionVectorsOpt#27757, protocol#27758, numOfRemoves#27759L, numOfFiles#27760L, numOfSetTransactions#27761L]
Arguments: isFinalPlan=true