From 93ca28385d4033c54d8c516d7db86449a065aba4 Mon Sep 17 00:00:00 2001 From: Alexander Slesarenko Date: Tue, 2 Jul 2024 16:19:31 +0100 Subject: [PATCH 01/12] eip47: first draft --- eip-0047.md | 284 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 284 insertions(+) create mode 100644 eip-0047.md diff --git a/eip-0047.md b/eip-0047.md new file mode 100644 index 00000000..fb26ae49 --- /dev/null +++ b/eip-0047.md @@ -0,0 +1,284 @@ + +# EIP-0048: Pooled Transaction Inputs + +## Background and Motivation + +In the current implementation of the Ergo blockchain, each transaction spends a list of +UTXOs (called Boxes) given by their boxId. These input boxes need to be selected by an +off-chain selector, which is part of the transaction builder in the wallet. Once the +transaction is built and signed, these concrete Boxes are set in stone and cannot be +changed without rebuilding the transaction and re-signing it again. + +### The Problem + +This creates challenges when multiple transactions need to be created from the same wallet +in short period of time in response to external events. Two transactions cannot share the +same input box, so the wallet must track which boxes are already spent and which are not, +selecting non-overlapping subsets of UTXOs. However, this is not always feasible, +especially when UTXOs are consolidated into a single box, resulting in a situation where +only one transaction can be created from that single UTXO and other transactions has to be +chained. + +This problem is exacerbated in scenarios requiring responsiveness. For +example, if a wallet participates in two or more DeFi protocols and needs to react to +multiple events quickly, it cannot chain transactions and has to submit them quickly in +response to the events. In such cases, the consolidation of UTXOs +can become a bottleneck, limiting the number of transactions that can be executed in +parallel. Moreover, in a chain of transactions, any subsequent transaction needs to know +the boxIds of the boxes created by the previous transaction, requiring the entire chain to +be built with knowledge of all boxIds, both existing and future, which rules out any +possibility for conditional chaining. + +### Illustrative Example + +Consider Alice, a user who actively participates in multiple DeFi protocols on the Ergo +blockchain. Each protocol requires Alice to submit transactions in response to various +events, such as staking rewards, yield farming updates, or automated trading triggers. +These interactions need to be executed promptly to maximize her returns and maintain her +positions. + +Scenario: + - Alice stakes some ERGs in a DeFi staking pool, which periodically pays out rewards. + - he also participates in a yield farming protocol, which requires her to move funds + between different liquidity pools based on real-time market conditions. + - Additionally, Alice uses an automated trading bot that submits transactions based on + market signals. + +Each of these DeFi protocols requires Alice to submit transactions spending from her +wallet in response to the protocol-specific events. However, these transactions cannot be chained +together as they are built independently by different dApps. If Alice's wallet has +consolidated its funds into a single UTXO for storage efficiency, she can only process one +transaction at a time, leading to delays and potentially missed opportunities. + +For example, when a staking reward is paid out, Alice needs to quickly reinvest +these rewards into the yield farming protocol. Simultaneously, the trading bot may need to +execute a trade based on a market signal. If all these transactions depend on the same +UTXO, only one can be processed at a time, creating a bottleneck. + +### The Solution + +To address these challenges, we propose the introduction of Pooled Transaction Inputs. +This new mechanism allows transactions to specify input pools from which UTXOs can be +selected dynamically, rather than pre-selecting specific UTXOs. By introducing a new data +structure called `PooledInput`, we enable transactions to define inputs to spend without +specifying concrete UTXOs at the time of transaction creation. + +With `PooledInput`, Alice can define her transactions to spend from a pool of UTXOs +associated with her wallet's address, specifying the amount to be spent and optional +selection filters. This way, multiple transactions can be created by different dApps and +processed in parallel, each spending from the same pool of UTXOs and without +pre-determined boxIds. The Ergo node will select ("unpool") the appropriate UTXOs for +spending during transaction processing, ensuring deterministic and conflict-free execution. + +In Alice's case, this means she can handle staking rewards, yield farming updates, and +automated trades simultaneously, improving her DeFi interaction efficiency and +responsiveness. This solution allows her to maximize her returns and maintain her +positions across multiple protocols without the risk of transaction delays or conflicts. + +By implementing Pooled Transaction Inputs, we aim to enhance the flexibility and +efficiency of UTXO management in Ergo, supporting higher throughput and more responsive +user experience. + +## Proposed Solution + +To address the challenges identified in the Background and Motivation section, we propose +the introduction of a new transaction input type called `PooledInput`. This new data +structure and its associated protocol modifications will enable transactions to +dynamically select UTXOs from a specified pool, enhancing the flexibility and efficiency +of transaction creation and execution on the Ergo blockchain. + +### Key Components of the Solution + +1. **PooledInput Data Structure** which contains the following fields: + - `address: ErgoAddress`: Specifies the address from which UTXOs should be selected. + - `filter: Option[Predicate]`: An optional predicate to filter the UTXOs within the pool, + allowing for more granular selection criteria. + - `value: Long`: The total value to be spent from the pool. + +2. **Transaction Format Extension:** + - Extend the existing transaction format to include a new field `pools: List[PooledInput]`, + which is a list of `PooledInput`. + +3. **Validation and Processing Logic:** + - Modify the Ergo node to support the new transaction format with the `pools` field. + - During transaction processing, the Ergo node will select UTXOs from the specified + address pool that satisfy the filter and have a total value greater than or equal to + the `value` field of the `PooledInput`. + +4. **Deterministic Box Selection:** + - Implement a deterministic selection algorithm to ensure all nodes select the same UTXOs + from the pool. This can be achieved by sorting the pool boxes by their order of inclusion + into the UTXO set (which is defined by the global order of transactions in the + blockchain). The first boxes from the sorted list are selected until the total value + meets or exceeds the required `value`. + +5. **Change Output Handling:** + - After selecting the required UTXOs, the transaction will generate a change output with + the remaining value (total value of selected UTXOs minus the `value` field of the + `PooledInput`). This change output will be appended to the list of transaction + outputs and the change amount will be returned to the address associated with the pool and + thus can be immediately consumed by the next transaction spending from the same pool. + +6. **Transaction Concretization:** + - Introduce the process of `input unpooling` during block + validation. This involves determining the exact UTXOs (concrete boxes) to be spent based + on the state of the UTXO set at the transaction's position in the block. This also + includes adding the change output to the transaction outputs. + +7. **Support for Multiple PooledInputs:** + - Allow transactions to include multiple `PooledInputs` in the `pools` list. The unpooling + process will handle each `PooledInput` sequentially, in the order they appear in the + list. + +### Implementation Details + +#### PooledInput Data Structure + The `PooledInput` structure can be defined in Scala as follows: + ```scala + case class PooledInput( + address: SigmaProp, // defines the pool where UTXOs are selected from + filter: Option[Predicate], // Optional filter to select specific UTXOs + value: Long, // Total value to be spent from the pool + signature: Array[Byte] // Signature for the `address` sigma proposition + ) + ``` + - `SigmaProp` type: aka Sigma Proposition, specifies the source address of the UTXOs (including multisig). + - `Predicate` type: A filter condition that UTXOs must satisfy to be considered for selection. + This is implemented as ErgoTree lambda of type `Box => Boolean`, i.e. it takes a box as + argument and can filter UTXOs boxes based on their registers. + - `value`: The total amount of ERGs to be covered by inputs spent from the pool. + - `signature`: The signature for the `address` sigma proposition, ensuring the transaction + is authorized to spend from the pool. + + The two fields `address` and `filter` define the UTXO pool as a list of boxes as if + returned by the following query: + ```sql + SELECT * FROM UTXO as U + WHERE U.ergoTree = ErgoTree.fromSigmaProp(address) and filter(U) + ``` + +#### Transaction Format Extension + Extend the transaction format to include the `pools` field: + ```scala + case class ErgoTransaction( + inputs: Seq[Input], // List of concrete inputs to spend + dataInputs: Seq[DataInput], // List of data inputs + pools: Seq[PooledInput], // List of pooled inputs (UTXO pools) to spend from + outputs: Seq[BoxCandidate] // List of output boxes (not including change outputs created for each pooled input) + ) + ``` + - `pools`: A list of `PooledInput` that each specify the list of UTXOs to be spent from + each pool during transaction processing. + - When the `pools` is empty (i.e., no pooled inputs), the transaction behaves as a + standard transaction with concrete inputs and data inputs. + - When the `pools` list contain a simple P2PK address, then the corresponding signature + should be present in the `signature` field of the `PooledInput`. + - When the `pools` list contain addresses belonging to different wallets, +then properly generated signatures has to be provided in the `signature` fields of the +corresponding `PooledInput`s. + +#### Validation and Processing Logic + During transaction validation, the Ergo node will: + - **Parse the `pools` field**: + ```scala + val pooledInputs = transaction.pools + ``` + - **For each `PooledInput`, select UTXOs** from the specified address that satisfy the + filter and meet the required value: + ```scala + pooledInputs.foreach { pooledInput => + val candidateUtxos = getUtxosFromAddress(pooledInput.address).filter(pooledInput.filter.getOrElse(_ => true)) + val selectedUtxos = selectUtxos(candidateUtxos, pooledInput.value) + transaction.inputs.appendAll(selectedUtxos.map(createInput)) + } + ``` + - `getUtxosFromAddress(address)`: Retrieves all UTXOs associated with the given address. + - `filter(predicate)`: Applies the optional filter to the UTXOs. + - `selectUtxos(utxos, value)`: Selects UTXOs from the filtered list until the total + value meets or exceeds the specified value. + +#### Extension of UTXO Storage + +To support the deterministic selection of UTXOs from a pool, we need to extend the UTXO +storage with additional information about each UTXO. Specifically, we need to store the +`blockHeight` and `outputIndex` for each UTXO as described below in order to be able to +determine the order of UTXOs in the pool. + +The UTXO storage needs to be extended to include the following additional fields for each +UTXO: + +- `blockHeight`: The height of the block in which the UTXO was created. This is calculated + as the height of the block where UTXO is created. + +- `outputIndex`: This is calculated as the index of the output in the list of outputs of + the full block that created the UTXO. + +These fields are calculated in the `UtxoState.applyModifier` method during the processing +of each `Insert` operation. The `Insert` operation is submitted to the +`persistentProver.performOneOperation` method, which updates the UTXO set in the +persistent storage. + +#### Deterministic Box Selection + The selection algorithm will retrieve UTXOs in order of their inclusion into the UTXO set + using `blockHeight` and `outputIndex` fields. The algorithm can be implemented as follows: + ```scala + def selectUtxos(utxos: Seq[Utxo], requiredValue: Long): Seq[Utxo] = { + val sortedUtxos = utxos.sortBy(u => (u.blockHeight, u.outputIndex)) + var selectedUtxos = Seq.empty[Utxo] + var totalSelectedValue = 0L + + for (utxo <- sortedUtxos if totalSelectedValue < requiredValue) { + selectedUtxos = selectedUtxos :+ utxo + totalSelectedValue += utxo.value + } + + selectedUtxos + } + ``` + - `sortBy(u => (u.blockHeight, u.outputIndex))`: Sorts UTXOs first by blockHeight and then by outputIndex. + - The selected UTXOs are those whose cumulative value meets or exceeds the required value. + +#### Change Output Handling + After selecting the UTXOs, an optional change output is created to return the remaining + coins to the pool. This is done as follows: + ```scala + def addChangeOutput(selectedUtxos: Seq[Utxo], pooledInput: PooledInput, transaction: ErgoTransaction): Unit = { + val totalSelectedValue = selectedUtxos.map(_.value).sum + val changeValue = totalSelectedValue - pooledInput.value + if (changeValue > 0) { + val changeOutput = createChangeOutput(changeValue, pooledInput.address) + transaction.outputs.append(changeOutput) + } + } + ``` + - `totalSelectedValue`: Sum of values of the selected UTXOs. + - `changeValue`: Difference between the total selected value and the required value. + - `createChangeOutput(value, address)`: Creates a change output box with the specified + value and address, note the use of the same address as the pool. + +#### Transaction Concretization + When a transaction have at least one `PooledInput`, the transaction needs to be + concretized, i.e. the exact UTXOs to be spent need to be determined along with the change + outputs. This process is called `input unpooling` and is done during block application and + validation. The concretization process can be implemented as follows: + ```scala + def concretizeTransaction(tx: ErgoTransaction, utxoState: UtxoState): Unit = { + tx.pools.flatMap { pooledInput => + val candidateUtxos = utxoState + .getUtxos(pooledInput.address) + .filter(pooledInput.filter.getOrElse(_ => true)) + val selectedUtxos = selectUtxos(candidateUtxos, pooledInput.value) + val concretizedInputs = selectedUtxos.map(u => createInput(u)) + transaction.inputs.appendAll(concretizedInputs) + addChangeOutput(selectedUtxos, pooledInput, transaction) + } + } + ``` + - `concretizeTransaction`: Resolves `PooledInputs` into concrete inputs + based on the current UTXO state and adds change outputs. + - `createInput(utxo)`: Converts a UTXO into a transaction input. + - Note, the method can handle multiple `PooledInput` by sequentially processing each + entry in the `pools` list: + - This ensures that each `PooledInput` is processed in the order they appear, with + deterministic UTXO selection and conflict-free execution. + From ae57c015dfd7fb6c365c064767eef16f1815631b Mon Sep 17 00:00:00 2001 From: Alexander Slesarenko Date: Wed, 3 Jul 2024 12:15:46 +0100 Subject: [PATCH 02/12] eip47: example added --- eip-0047.md | 144 ++++++++++++++++++++++++++++++++++++---- eip-0047/entities.puml | 51 ++++++++++++++ eip-0047/er-diagram.uml | 0 3 files changed, 182 insertions(+), 13 deletions(-) create mode 100644 eip-0047/entities.puml create mode 100644 eip-0047/er-diagram.uml diff --git a/eip-0047.md b/eip-0047.md index fb26ae49..4e829509 100644 --- a/eip-0047.md +++ b/eip-0047.md @@ -17,17 +17,16 @@ same input box, so the wallet must track which boxes are already spent and which selecting non-overlapping subsets of UTXOs. However, this is not always feasible, especially when UTXOs are consolidated into a single box, resulting in a situation where only one transaction can be created from that single UTXO and other transactions has to be -chained. +chained which by itself creates undesirable dependency on the parent transaction so that +cancellation/delay of the parent transaction will also cancel/delay the child transaction. This problem is exacerbated in scenarios requiring responsiveness. For -example, if a wallet participates in two or more DeFi protocols and needs to react to -multiple events quickly, it cannot chain transactions and has to submit them quickly in +example, if a user participates in two or more DeFi protocols involving some form of +transaction queueing and needs to react to multiple events quickly, user cannot chain +transactions belonging to different DeFi protocols, but has to submit them quickly in response to the events. In such cases, the consolidation of UTXOs can become a bottleneck, limiting the number of transactions that can be executed in -parallel. Moreover, in a chain of transactions, any subsequent transaction needs to know -the boxIds of the boxes created by the previous transaction, requiring the entire chain to -be built with knowledge of all boxIds, both existing and future, which rules out any -possibility for conditional chaining. +parallel. ### Illustrative Example @@ -39,7 +38,7 @@ positions. Scenario: - Alice stakes some ERGs in a DeFi staking pool, which periodically pays out rewards. - - he also participates in a yield farming protocol, which requires her to move funds + - She also participates in a yield farming protocol, which requires her to move funds between different liquidity pools based on real-time market conditions. - Additionally, Alice uses an automated trading bot that submits transactions based on market signals. @@ -48,13 +47,18 @@ Each of these DeFi protocols requires Alice to submit transactions spending from wallet in response to the protocol-specific events. However, these transactions cannot be chained together as they are built independently by different dApps. If Alice's wallet has consolidated its funds into a single UTXO for storage efficiency, she can only process one -transaction at a time, leading to delays and potentially missed opportunities. +transaction at a time, leading to delays and potentially missed opportunities. For example, when a staking reward is paid out, Alice needs to quickly reinvest these rewards into the yield farming protocol. Simultaneously, the trading bot may need to execute a trade based on a market signal. If all these transactions depend on the same UTXO, only one can be processed at a time, creating a bottleneck. +Alternatively, Alice can de-consolidate her UTXOs, but this additional step will worsen +user experience and increase time and cost of transactions. Even if implemented by wallet, +such add-hoc consolidation/de-consolidation once in a while will bottleneck Alice in her +DeFi experience. + ### The Solution To address these challenges, we propose the introduction of Pooled Transaction Inputs. @@ -67,17 +71,20 @@ With `PooledInput`, Alice can define her transactions to spend from a pool of UT associated with her wallet's address, specifying the amount to be spent and optional selection filters. This way, multiple transactions can be created by different dApps and processed in parallel, each spending from the same pool of UTXOs and without -pre-determined boxIds. The Ergo node will select ("unpool") the appropriate UTXOs for -spending during transaction processing, ensuring deterministic and conflict-free execution. +pre-determined boxIds. The Ergo node will select (or _unpool_) the appropriate UTXOs for +spending during transaction processing, ensuring deterministic and conflict-free execution +on L1 protocol level. In Alice's case, this means she can handle staking rewards, yield farming updates, and automated trades simultaneously, improving her DeFi interaction efficiency and responsiveness. This solution allows her to maximize her returns and maintain her -positions across multiple protocols without the risk of transaction delays or conflicts. +positions across multiple protocols without the risk of transaction conflicts even in the +presence of queuing, delays or cancellations at individual protocols. By implementing Pooled Transaction Inputs, we aim to enhance the flexibility and efficiency of UTXO management in Ergo, supporting higher throughput and more responsive -user experience. +user experience especially in the context of cross-DeFi interactions. More on this in the +Benefits section. ## Proposed Solution @@ -282,3 +289,114 @@ persistent storage. - This ensures that each `PooledInput` is processed in the order they appear, with deterministic UTXO selection and conflict-free execution. +### Example: Alice's Multi-Step DeFi Workflow +Let's create a detailed example involving Alice's participation in multiple DeFi +protocols, requiring her to sign transactions and manage her UTXOs efficiently using the +Pooled Transaction Inputs mechanism. + +#### Context +Alice is an active participant in the DeFi ecosystem on the Ergo blockchain. She engages +in multiple protocols, including staking, yield farming, and automated trading. She needs +to handle these interactions efficiently to maximize her returns. + +#### Current Step-by-Step Workflow with explicit UTXO selection + +1. **Initial Setup:** + - **Wallet:** Alice's wallet contains the following UTXOs: + UTXO1: 100 ERG + +2. **Staking:** + - **Event:** Alice decides to stake 40 ERG in a DeFi staking pool that periodically pays out rewards. + - **Transaction 1:** Alice creates a transaction to stake 40 ERG. + - **Inputs:** UTXO1 (100 ERG) + - **Outputs:** + - StakeOutput: 40 ERG to the staking pool + - ChangeOutput: 60 ERG back to Alice's wallet + - Wallet: UTXO2: 60 ERG (unconfirmed) + +3. **Yield Farming:** + - **Event:** Alice wants to participate in a yield farming protocol that requires 35 ERG. + - **Transaction 2:** Alice creates chained transaction to deposit 35 ERG into the yield farming pool + referencing the change output from the previous transaction. + - **Inputs:** UTXO2 (60 ERG) (unconfirmed) + - **Outputs:** + - YieldFarmOutput: 35 ERG to the yield farming pool + - ChangeOutput: 25 ERG back to Alice's wallet + - Wallet: UTXO3: 25 ERG (unconfirmed) + +4. **Automated Trading:** + - **Event:** Alice's automated trading bot detects a market signal and needs to execute a trade with 20 ERG. + - **Transaction 3:** The bot creates a transaction to execute the trade. + - **Inputs:** UTXO3 (25 ERG) (unconfirmed) + - **Outputs:** + - TradeOutput: 20 ERG to the trading pool + - ChangeOutput: 5 ERG back to Alice's wallet + - Wallet: UTXO4: 5 ERG (unconfirmed) + +Note, in order for Alice to execute the above transaction in parallel, they all need to be +chained together. This creates hard dependence between the transactions and requires all +the previous (parent) transactions to be successful for child transactions to also be +successful. This rules out any possibility for cancellation or transaction queueing on +DeFi protocol side. If Transaction 1 is cancelled/delayed by the DeFi protocol, then +the whole chain of transactions will be also cancelled/delayed. + +#### Same Workflow with Pooled Inputs +1. **Initial Setup:** + - **Wallet:** Alice's wallet contains the following UTXOs: + UTXO1: 100 ERG + +2. **Staking:** + - **Transaction 1:** Alice creates a transaction to stake 40 ERG. + - **PooledInput** + - address: Alice's Wallet Address + - value: 40 ERG + - signature: Alice's Signature + - **Outputs:** + - StakeOutput: 40 ERG to the staking pool + +3. **Yield Farming:** + - **Transaction 2:** Alice creates chained transaction to deposit 35 ERG into the yield farming pool + - **PooledInput** + - address: Alice's Wallet Address + - value: 35 ERG + - signature: Alice's Signature + - **Outputs:** + - YieldFarmOutput: 35 ERG to the yield farming pool + +3. **Automated Trading:** + - **Event:** Alice's automated trading bot detects a market signal and needs to execute a trade with 20 ERG. + - **Transaction 3:** The bot creates a transaction to execute the trade. + - **PooledInput** + - address: Alice's Wallet Address + - value: 20 ERG + - signature: Alice's Signature + - **Outputs:** + - TradeOutput: 20 ERG to the trading pool + +#### Benefits of Pooled Transaction Inputs +- wallet doesn't need to chain transactions and select UTXOs explicitly +- no need to add change outputs explicitly +- transactions can be sent in any order and don't depend on each other +- any transaction can be cancelled/delayed/queued without affecting others + +#### Conclusion + +The introduction of Pooled Transaction Inputs significantly streamlines the process of +managing multiple DeFi interactions for users like Alice. By allowing transactions to +specify input pools rather than fixed UTXOs, Alice can engage in various DeFi protocols +simultaneously without the need for chaining transactions or managing UTXO dependencies. +This approach not only enhances the flexibility and efficiency of transaction creation, but +also reduces the risk of transaction conflicts caused by delays/queueing/cancellations of +transactions, enabling a more responsive and fluid DeFi experience on UTXO blockchain +platform such as Ergo. + +Furthermore, Pooled Transaction Inputs mitigate the challenges associated with UTXO +consolidation, allowing users to maximize their participation in the DeFi ecosystem +without being bottlenecked by transaction processing limitations. This proposal +represents a step forward in the scalability and usability of the Ergo +blockchain, paving the way for more complex and dynamic DeFi applications. + +Pooled Transaction Inputs simplifies UTXO management in the wallet software, by allowing +transactions to draw from UTXO pools. This eliminates the need +for fixed UTXO selection, enhancing transaction flexibility and efficiency. As a result, +users enjoy a smoother DeFi experience on the Ergo blockchain. diff --git a/eip-0047/entities.puml b/eip-0047/entities.puml new file mode 100644 index 00000000..e8120da3 --- /dev/null +++ b/eip-0047/entities.puml @@ -0,0 +1,51 @@ +@startuml +entity Transaction { + +transactionId: String + +inputs: List + +outputs: List + +pools: List +} + +entity PooledInput { + +address: ErgoAddress + +filter: Option + +value: Long + +signature: Array +} + +entity UTXO { + +utxoId: String + +value: Long + +blockHeight: Int + +outputIndex: Int +} + +entity Wallet { + +walletId: String + +address: ErgoAddress + +utxos: List +} + +entity ErgoNode { + +nodeId: String +} + +entity Block { + +blockId: String + +previousBlockId: String + +blockHeight: Int + +transactions: List +} +entity Address { +} +Transaction "1" -- "0..*" PooledInput : pools +Transaction "1" -- "0..*" UTXO : inputs +Transaction "1" -- "1..*" UTXO : outputs +Wallet "1" -- "*" Address : has access to +Wallet "1..*" -- "1" ErgoNode : interacts with +ErgoNode "1" -- "*" Block : creates +ErgoNode "1" -- "*" Transaction : processes +Block "1" -- "1..*" Transaction : contains +PooledInput "*" -- "1" Address : spends from +UTXO "0..*" -- "1" Block : created in +@enduml diff --git a/eip-0047/er-diagram.uml b/eip-0047/er-diagram.uml new file mode 100644 index 00000000..e69de29b From 2fb18d0d54685be7606ee4f1529685d88eb4f785 Mon Sep 17 00:00:00 2001 From: Alexander Slesarenko Date: Thu, 4 Jul 2024 11:10:00 +0100 Subject: [PATCH 03/12] eip47: solution to avoid double spending --- eip-0047.md | 179 ++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 130 insertions(+), 49 deletions(-) diff --git a/eip-0047.md b/eip-0047.md index 4e829509..9e4387f3 100644 --- a/eip-0047.md +++ b/eip-0047.md @@ -86,6 +86,32 @@ efficiency of UTXO management in Ergo, supporting higher throughput and more res user experience especially in the context of cross-DeFi interactions. More on this in the Benefits section. +### Additional Problem: Double-Spending Risk with PooledInputs + +While PooledInputs offer significant benefits, they also introduce a potential risk (if +not addressed) of double-spending. Specifically, when a transaction contains only +PooledInputs, it can be submitted multiple times to the blockchain through different +network nodes, potentially spending from the same UTXO pools. This happens because the +`messageToSign` for different submissions of the same transaction will be the same, making +the signature valid for each attempt. This loophole can be exploited for double-spending +attacks. + +The classical solution to this problem involves incorporating a nonce in the +`messageToSign` that increments with each transaction submission. However, this approach +is impractical for UTXO blockchains due to several reasons: +- The nonce is not part of the UTXO. +- There is no place to store the previous nonce in the UTXO state. +- Introducing a nonce would require significant changes to the UTXO model. +- The nonce would need to be persistently stored, adding inefficiency and requiring additional storage. +- Each transaction submission would require updating the nonce, increasing processing overhead. +- Wallet software would need to track the nonce for each UTXO pool, adding complexity. +- The transaction format and signing process would need to change to include the nonce, + necessitating wallet software updates. + +In this proposal we address this issue by imposing a constraint on the usage of +PooledInputs and describe a convention for UTXO management that will make this proposal +both safe and also practical to implement. More on this in the Proposed Solution section. + ## Proposed Solution To address the challenges identified in the Background and Motivation section, we propose @@ -137,6 +163,43 @@ of transaction creation and execution on the Ergo blockchain. process will handle each `PooledInput` sequentially, in the order they appear in the list. +### Solution to Double-Spending Risk + +To address the risk of double-spending attacks with PooledInputs, we propose a constraint: +a transaction with PooledInputs must include at least one concrete input. This ensures +that each transaction has a unique piece of data in its `messageToSign`, preventing +duplicate submissions. The Ergo node will enforce this constraint during transaction +validation. + +However, in many practical cases, transactions with PooledInputs may not naturally have +concrete inputs, as seen in the examples described earlier. To address this, we propose a +convention for UTXO management supported by wallet software, which can be regarded as a +wallet standard (can be published as separate EIP motivated by this proposal). + +#### Wallet Standard for PooledInputs + +1. **PoolHolderBox**: + - When creating a transaction with PooledInputs, the wallet should include a dummy input + box called `PoolHolderBox` with a minimal value (pre-defined constant + `PoolHolderBoxValue` of nanoERG). This box is spendable from the wallet's address + and serves solely as a placeholder to ensure the transaction has at least one + concrete input. + +2. **Maintaining PoolHolderBoxes**: + - Every transaction spending a `PoolHolderBox` should also create a new one. This + maintains a continuous supply of `PoolHolderBoxes` for future transactions and + preserves the input/output balance. + - Initially, the wallet software should create a sufficient number of `PoolHolderBoxes` to + enable the PooledInputs feature. Subsequent management of these boxes will be + handled internally by the wallet, ensuring user-friendly operation. + +3. **User Experience**: + - The wallet can prompt the user to enable the "Pooled Inputs" feature once enough ERGs + are available, creating the necessary `PoolHolderBoxes` in a single transaction. + - istinguish `PoolHolderBoxes` from other boxes, they should have a specific register + value (e.g., `R4 = 0x012345`). The wallet can identify available `PoolHolderBoxes` + using (wallet's address, `R4 = 0x012345`, value == `PoolHolderBoxValue`). + ### Implementation Details #### PooledInput Data Structure @@ -341,62 +404,80 @@ DeFi protocol side. If Transaction 1 is cancelled/delayed by the DeFi protocol, the whole chain of transactions will be also cancelled/delayed. #### Same Workflow with Pooled Inputs -1. **Initial Setup:** - - **Wallet:** Alice's wallet contains the following UTXOs: - UTXO1: 100 ERG -2. **Staking:** - - **Transaction 1:** Alice creates a transaction to stake 40 ERG. - - **PooledInput** - - address: Alice's Wallet Address - - value: 40 ERG - - signature: Alice's Signature - - **Outputs:** +Note: +- to ensure the security and uniqueness of transactions involving PooledInputs, +Alice's wallet will include a `PoolHolderBox` in each transaction. +- many holder boxes can be created initially to enable PooledInputs feature in the wallet +- the number of holder boxes defines the maximum number of transactions that can be created + in parallel without using chaining. + +1. **Initial Setup**: + - **Wallet**: Alice's wallet contains the following UTXOs: + - UTXO1: 100 ERG + - PoolHolderBox1: 0.001 ERG (pre-defined constant `PoolHolderBoxValue`) + - PoolHolderBox2: 0.001 ERG + - PoolHolderBox3: 0.001 ERG + +2. **Staking**: + - **Transaction 1**: Alice creates a transaction to stake 40 ERG. + - **PooledInput**: + - Address: Alice's Wallet Address + - Value: 40 ERG + - Signature: Alice's Signature + - **Concrete Input**: PoolHolderBox1 (0.001 ERG) + - **Outputs**: - StakeOutput: 40 ERG to the staking pool + - PoolHolderBox4: 0.001 ERG to Alice's wallet + +3. **Yield Farming**: + - **Transaction 2**: Alice creates a transaction to deposit 35 ERG into the yield farming pool. + - **PooledInput**: + - Address: Alice's Wallet Address + - Value: 35 ERG + - Signature: Alice's Signature + - **Concrete Input**: PoolHolderBox2 (0.001 ERG) + - **Outputs**: + - YieldFarmOutput: 35 ERG to the yield farming pool + - PoolHolderBox5: 0.001 ERG to Alice's wallet + +4. **Automated Trading**: + - **Transaction 3**: The bot creates a transaction to execute a trade with 20 ERG. + - **PooledInput**: + - Address: Alice's Wallet Address + - Value: 20 ERG + - Signature: Alice's Signature + - **Concrete Input**: PoolHolderBox3 (0.001 ERG) + - **Outputs**: + - TradeOutput: 20 ERG to the trading pool + - PoolHolderBox6: 0.001 ERG to Alice's wallet -3. **Yield Farming:** - - **Transaction 2:** Alice creates chained transaction to deposit 35 ERG into the yield farming pool - - **PooledInput** - - address: Alice's Wallet Address - - value: 35 ERG - - signature: Alice's Signature - - **Outputs:** - - YieldFarmOutput: 35 ERG to the yield farming pool - -3. **Automated Trading:** - - **Event:** Alice's automated trading bot detects a market signal and needs to execute a trade with 20 ERG. - - **Transaction 3:** The bot creates a transaction to execute the trade. - - **PooledInput** - - address: Alice's Wallet Address - - value: 20 ERG - - signature: Alice's Signature - - **Outputs:** - - TradeOutput: 20 ERG to the trading pool +This workflow ensures that each transaction has a unique `messageToSign`, preventing +double-spending attacks while maintaining the flexibility and efficiency of PooledInputs. #### Benefits of Pooled Transaction Inputs - wallet doesn't need to chain transactions and select UTXOs explicitly -- no need to add change outputs explicitly -- transactions can be sent in any order and don't depend on each other +- no need to add change outputs explicitly (the change is handled by the network on the protocol level) +- transactions can be sent in any order (or all at once) and don't depend on each other - any transaction can be cancelled/delayed/queued without affecting others #### Conclusion -The introduction of Pooled Transaction Inputs significantly streamlines the process of -managing multiple DeFi interactions for users like Alice. By allowing transactions to -specify input pools rather than fixed UTXOs, Alice can engage in various DeFi protocols -simultaneously without the need for chaining transactions or managing UTXO dependencies. -This approach not only enhances the flexibility and efficiency of transaction creation, but -also reduces the risk of transaction conflicts caused by delays/queueing/cancellations of -transactions, enabling a more responsive and fluid DeFi experience on UTXO blockchain -platform such as Ergo. - -Furthermore, Pooled Transaction Inputs mitigate the challenges associated with UTXO -consolidation, allowing users to maximize their participation in the DeFi ecosystem -without being bottlenecked by transaction processing limitations. This proposal -represents a step forward in the scalability and usability of the Ergo -blockchain, paving the way for more complex and dynamic DeFi applications. - -Pooled Transaction Inputs simplifies UTXO management in the wallet software, by allowing -transactions to draw from UTXO pools. This eliminates the need -for fixed UTXO selection, enhancing transaction flexibility and efficiency. As a result, -users enjoy a smoother DeFi experience on the Ergo blockchain. +The introduction of Pooled Transaction Inputs and the use of `PoolHolderBox` significantly +streamline the process of managing multiple DeFi interactions for users like Alice. By +allowing transactions to specify input pools rather than fixed UTXOs, and ensuring each +transaction includes at least one concrete input, users can engage in various DeFi +protocols simultaneously without chaining transactions or managing UTXO dependencies. This +approach not only enhances transaction flexibility and efficiency but also prevents +double-spending attacks, enabling a more secure and responsive DeFi experience on UTXO +blockchain platform such as Ergo. + +Furthermore, Pooled Transaction Inputs and `PoolHolderBox` mitigate challenges associated +with UTXO consolidation, allowing users to maximize their participation in the DeFi +ecosystem without transaction processing limitations. This proposal represents a +significant step forward in the scalability and usability of the Ergo blockchain, paving +the way for more complex and dynamic DeFi applications. + +The implementation of Pooled Transaction Inputs and `PoolHolderBox` as a wallet convention +simplifies UTXO management, ensuring that transactions are both secure and efficient. This +user-friendly approach enhances the DeFi experience on the Ergo blockchain. From ad5c31de676ca3f519271c2075c4e8d4a9e54185 Mon Sep 17 00:00:00 2001 From: Alexander Slesarenko Date: Sun, 7 Jul 2024 14:12:44 +0100 Subject: [PATCH 04/12] eip47: extending implementation with indexDb --- eip-0047.md | 109 ++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 88 insertions(+), 21 deletions(-) diff --git a/eip-0047.md b/eip-0047.md index 9e4387f3..57a00a22 100644 --- a/eip-0047.md +++ b/eip-0047.md @@ -145,20 +145,32 @@ of transaction creation and execution on the Ergo blockchain. blockchain). The first boxes from the sorted list are selected until the total value meets or exceeds the required `value`. -5. **Change Output Handling:** +5. **LevelDB Integration for Efficient Box Selection:** + + - To efficiently manage UTXOs and ensure deterministic box selection, we propose leveraging + LevelDB's ordered key-value storage. This approach maintains UTXOs in an ordered fashion + based on `(address, blockHeight, outputIndex)`, where the value stored is the `boxId` of + the UTXO. + - Key Design: + - Key: `(address, blockHeight, outputIndex)` formatted as a string to maintain order. + - Value: `boxId` stored as bytes. + - New Database Instance: + - `indexDb`: Stores the ordered keys with the UTXO `boxId` as the value. + +6. **Change Output Handling:** - After selecting the required UTXOs, the transaction will generate a change output with the remaining value (total value of selected UTXOs minus the `value` field of the `PooledInput`). This change output will be appended to the list of transaction outputs and the change amount will be returned to the address associated with the pool and thus can be immediately consumed by the next transaction spending from the same pool. -6. **Transaction Concretization:** +7. **Transaction Concretization:** - Introduce the process of `input unpooling` during block validation. This involves determining the exact UTXOs (concrete boxes) to be spent based on the state of the UTXO set at the transaction's position in the block. This also includes adding the change output to the transaction outputs. -7. **Support for Multiple PooledInputs:** +8. **Support for Multiple PooledInputs:** - Allow transactions to include multiple `PooledInputs` in the `pools` list. The unpooling process will handle each `PooledInput` sequentially, in the order they appear in the list. @@ -289,24 +301,79 @@ of each `Insert` operation. The `Insert` operation is submitted to the persistent storage. #### Deterministic Box Selection - The selection algorithm will retrieve UTXOs in order of their inclusion into the UTXO set - using `blockHeight` and `outputIndex` fields. The algorithm can be implemented as follows: - ```scala - def selectUtxos(utxos: Seq[Utxo], requiredValue: Long): Seq[Utxo] = { - val sortedUtxos = utxos.sortBy(u => (u.blockHeight, u.outputIndex)) - var selectedUtxos = Seq.empty[Utxo] - var totalSelectedValue = 0L - - for (utxo <- sortedUtxos if totalSelectedValue < requiredValue) { - selectedUtxos = selectedUtxos :+ utxo - totalSelectedValue += utxo.value - } - selectedUtxos - } - ``` - - `sortBy(u => (u.blockHeight, u.outputIndex))`: Sorts UTXOs first by blockHeight and then by outputIndex. - - The selected UTXOs are those whose cumulative value meets or exceeds the required value. +To ensure deterministic and efficient selection of UTXOs for a given `PooledInput`, we use +LevelDB to efficiently (i.e. O(log(n)) time) store and retrieve UTXOs in an ordered manner +(using lexically ordered keys). + +##### UTXO Storage and Key Design + +Each UTXO is stored with a key consisting of `(address, blockHeight, outputIndex)` and the +`boxId` as the value. The UTXO data itself is stored separately in the +`PersistentBatchAVLProver` (field in `UtxoState`) with the `boxId` as the key. + +```scala +def createKey(address: String, blockHeight: Long, outputIndex: Int): String = { + f"$address%s-$blockHeight%010d-$outputIndex%05d" +} +``` + +##### Adding and Removing UTXOs + +UTXOs are added to and removed from both the `indexDb` and the `PersistentBatchAVLProver`. +The logic is as follows: + +```scala +val stateChanges = ErgoState.stateChanges(block) +stateChanges.foreach { change => change match { + case (Insert(k, v), utxo) => + val key = createKey(utxo.address, utxo.blockHeight, utxo.outputIndex) + indexDb.put(key, utxo.boxId) + persistentProver.performOneOperation(Insert(k, v)) // existing code: add UTXO to the persistent storage + case Remove(boxId) => + val utxo = persistentProver.unauthenticatedLookup(boxId).get + val key = createKey(utxo.address, utxo.blockHeight, utxo.outputIndex) + indexDb.delete(key) + persistentProver.performOneOperation(Remove(boxId)) +}} +``` +##### Selecting UTXOs for Unpooling + +To select UTXOs for a given PooledInput, we use an iterator to retrieve UTXOs in order +until the required value is met. The logic can be implemented as follows: + +```scala +def selectUtxos(address: String, requiredValue: Long): Seq[UTXO] = { + val iter = indexDb.iterator() + val prefix = f"$address%s-" + iter.seek(prefix.getBytes) + var selectedUtxos = Seq.empty[UTXO] + var totalSelectedValue = 0L + + while (iter.hasNext && totalSelectedValue < requiredValue) { + val entry = iter.next() + val keyStr = new String(entry.getKey) + if (keyStr.startsWith(prefix)) { + val boxId = new String(entry.getValue) + val utxoBytes = persistentProver.unauthenticatedLookup(boxId).get + val utxo = deserialize(utxoBytes) + + selectedUtxos = selectedUtxos :+ utxo + totalSelectedValue += utxo.value + } + } + + iter.close() + + if (totalSelectedValue >= requiredValue) { + selectedUtxos + } else { + Seq.empty[UTXO] // Or throw an error if the required value cannot be met + } +} +``` +By leveraging LevelDB's ordered storage, we ensure efficient and deterministic selection +of UTXOs for unpooling. #### Change Output Handling After selecting the UTXOs, an optional change output is created to return the remaining @@ -403,7 +470,7 @@ successful. This rules out any possibility for cancellation or transaction queue DeFi protocol side. If Transaction 1 is cancelled/delayed by the DeFi protocol, then the whole chain of transactions will be also cancelled/delayed. -#### Same Workflow with Pooled Inputs +#### New step-by-step Workflow with Pooled Inputs Note: - to ensure the security and uniqueness of transactions involving PooledInputs, From 50af1cbca8b954c637c3416d54c5c853eadcba0a Mon Sep 17 00:00:00 2001 From: Alexander Slesarenko Date: Sun, 7 Jul 2024 20:34:44 +0100 Subject: [PATCH 05/12] eip47: removing filter field --- eip-0047.md | 192 ++++++++++++++++++++++++++-------------------------- 1 file changed, 96 insertions(+), 96 deletions(-) diff --git a/eip-0047.md b/eip-0047.md index 57a00a22..c635f32d 100644 --- a/eip-0047.md +++ b/eip-0047.md @@ -1,3 +1,9 @@ +- Author: aslesarenko +- Status: Proposed +- Created: 07-July-2024 +- Last edited: 07-July-2024 +- License: CC0 +- Forking: soft-fork # EIP-0048: Pooled Transaction Inputs @@ -12,13 +18,15 @@ changed without rebuilding the transaction and re-signing it again. ### The Problem This creates challenges when multiple transactions need to be created from the same wallet -in short period of time in response to external events. Two transactions cannot share the +in short period of time in response to external events (e.g. when participating in multiple +DeFi protocols). Two transactions cannot share the same input box, so the wallet must track which boxes are already spent and which are not, selecting non-overlapping subsets of UTXOs. However, this is not always feasible, especially when UTXOs are consolidated into a single box, resulting in a situation where only one transaction can be created from that single UTXO and other transactions has to be chained which by itself creates undesirable dependency on the parent transaction so that -cancellation/delay of the parent transaction will also cancel/delay the child transaction. +cancellation/delaying/queueing of the parent transaction will also cancel/delay/queue the +child transaction. This problem is exacerbated in scenarios requiring responsiveness. For example, if a user participates in two or more DeFi protocols involving some form of @@ -59,59 +67,32 @@ user experience and increase time and cost of transactions. Even if implemented such add-hoc consolidation/de-consolidation once in a while will bottleneck Alice in her DeFi experience. -### The Solution +### The Solution Idea -To address these challenges, we propose the introduction of Pooled Transaction Inputs. +To address these challenges, we propose the introduction of _Pooled Transaction Inputs_. This new mechanism allows transactions to specify input pools from which UTXOs can be selected dynamically, rather than pre-selecting specific UTXOs. By introducing a new data -structure called `PooledInput`, we enable transactions to define inputs to spend without +structure called `PooledInput`, we enable transactions to define _inputs to spend_ without specifying concrete UTXOs at the time of transaction creation. With `PooledInput`, Alice can define her transactions to spend from a pool of UTXOs -associated with her wallet's address, specifying the amount to be spent and optional -selection filters. This way, multiple transactions can be created by different dApps and -processed in parallel, each spending from the same pool of UTXOs and without -pre-determined boxIds. The Ergo node will select (or _unpool_) the appropriate UTXOs for -spending during transaction processing, ensuring deterministic and conflict-free execution -on L1 protocol level. +associated with her wallet's address, specifying the amount to be spent. This way, +multiple transactions can be created by different dApps and processed in parallel, each +spending from the same pool of UTXOs and without pre-determined boxIds. The Ergo node will +select (or _unpool_) the appropriate UTXOs for spending during transaction processing, +ensuring deterministic and conflict-free execution on L1 protocol level. In Alice's case, this means she can handle staking rewards, yield farming updates, and automated trades simultaneously, improving her DeFi interaction efficiency and -responsiveness. This solution allows her to maximize her returns and maintain her -positions across multiple protocols without the risk of transaction conflicts even in the -presence of queuing, delays or cancellations at individual protocols. +responsiveness. This solution allows her to create transactions simultaneously across +multiple dApps without the risk of transaction conflicts even in the +presence of queuing, delays or cancellations at individual applications. By implementing Pooled Transaction Inputs, we aim to enhance the flexibility and efficiency of UTXO management in Ergo, supporting higher throughput and more responsive user experience especially in the context of cross-DeFi interactions. More on this in the Benefits section. -### Additional Problem: Double-Spending Risk with PooledInputs - -While PooledInputs offer significant benefits, they also introduce a potential risk (if -not addressed) of double-spending. Specifically, when a transaction contains only -PooledInputs, it can be submitted multiple times to the blockchain through different -network nodes, potentially spending from the same UTXO pools. This happens because the -`messageToSign` for different submissions of the same transaction will be the same, making -the signature valid for each attempt. This loophole can be exploited for double-spending -attacks. - -The classical solution to this problem involves incorporating a nonce in the -`messageToSign` that increments with each transaction submission. However, this approach -is impractical for UTXO blockchains due to several reasons: -- The nonce is not part of the UTXO. -- There is no place to store the previous nonce in the UTXO state. -- Introducing a nonce would require significant changes to the UTXO model. -- The nonce would need to be persistently stored, adding inefficiency and requiring additional storage. -- Each transaction submission would require updating the nonce, increasing processing overhead. -- Wallet software would need to track the nonce for each UTXO pool, adding complexity. -- The transaction format and signing process would need to change to include the nonce, - necessitating wallet software updates. - -In this proposal we address this issue by imposing a constraint on the usage of -PooledInputs and describe a convention for UTXO management that will make this proposal -both safe and also practical to implement. More on this in the Proposed Solution section. - ## Proposed Solution To address the challenges identified in the Background and Motivation section, we propose @@ -124,8 +105,6 @@ of transaction creation and execution on the Ergo blockchain. 1. **PooledInput Data Structure** which contains the following fields: - `address: ErgoAddress`: Specifies the address from which UTXOs should be selected. - - `filter: Option[Predicate]`: An optional predicate to filter the UTXOs within the pool, - allowing for more granular selection criteria. - `value: Long`: The total value to be spent from the pool. 2. **Transaction Format Extension:** @@ -135,14 +114,14 @@ of transaction creation and execution on the Ergo blockchain. 3. **Validation and Processing Logic:** - Modify the Ergo node to support the new transaction format with the `pools` field. - During transaction processing, the Ergo node will select UTXOs from the specified - address pool that satisfy the filter and have a total value greater than or equal to - the `value` field of the `PooledInput`. + address pool and have a total value greater than or equal to the `value` field of the + `PooledInput`. 4. **Deterministic Box Selection:** - Implement a deterministic selection algorithm to ensure all nodes select the same UTXOs from the pool. This can be achieved by sorting the pool boxes by their order of inclusion into the UTXO set (which is defined by the global order of transactions in the - blockchain). The first boxes from the sorted list are selected until the total value + blockchain). The first boxes from the ordered list are selected until the total value meets or exceeds the required `value`. 5. **LevelDB Integration for Efficient Box Selection:** @@ -175,42 +154,69 @@ of transaction creation and execution on the Ergo blockchain. process will handle each `PooledInput` sequentially, in the order they appear in the list. +### Avoiding Double-Spending with PooledInputs + +While PooledInputs offer flexibility, they also introduce a potential risk (if +not addressed) of double-spending. Specifically, when a transaction contains only +PooledInputs, it can be submitted multiple times to the blockchain through different +network nodes, potentially spending from the same UTXO pools. This is possible because the +`messageToSign` for different submissions of the same transaction will be the same, making +the signature valid for each attempt. This loophole can be exploited for double-spending +attacks. + +The classical solution to this problem involves incorporating a _nonce_ in the +`messageToSign` that increments with each transaction submission. However, this approach +is impractical for UTXO blockchains due to several reasons: +- The nonce is not part of the UTXO. +- There is no place to store the previous nonce in the UTXO state. +- Introducing a nonce would require significant changes to the UTXO model. +- The nonce would need to be persistently stored, adding inefficiency and requiring additional storage. +- Each transaction submission would require updating the nonce, which introduces a shared state. +- Wallet software would need to track the nonce for each UTXO pool, adding complexity. +- The transaction format and signing process would need to change to include the nonce, + necessitating wallet software updates. + +In this proposal, instead of using nonce, we address this issue by imposing a constraint +on the usage of PooledInputs and describe a convention for UTXO management that will make +this proposal both safe and also practical to implement. More on this in the Proposed +Solution section. + ### Solution to Double-Spending Risk -To address the risk of double-spending attacks with PooledInputs, we propose a constraint: -a transaction with PooledInputs must include at least one concrete input. This ensures -that each transaction has a unique piece of data in its `messageToSign`, preventing -duplicate submissions. The Ergo node will enforce this constraint during transaction -validation. +To address the possibility of double-spending attacks with PooledInputs, we propose a +constraint: a transaction with PooledInputs must include at least one concrete input. This +ensures that each transaction has a unique piece of data in its `messageToSign`, +preventing duplicate submissions. The Ergo node will enforce this constraint during +transaction validation. -However, in many practical cases, transactions with PooledInputs may not naturally have +However, in many practical use cases, transactions with PooledInputs may not naturally have concrete inputs, as seen in the examples described earlier. To address this, we propose a convention for UTXO management supported by wallet software, which can be regarded as a wallet standard (can be published as separate EIP motivated by this proposal). #### Wallet Standard for PooledInputs -1. **PoolHolderBox**: +1. **PoolAccessBox**: - When creating a transaction with PooledInputs, the wallet should include a dummy input - box called `PoolHolderBox` with a minimal value (pre-defined constant - `PoolHolderBoxValue` of nanoERG). This box is spendable from the wallet's address + box called `PoolAccessBox` with a minimal value (pre-defined constant + `PoolAccessBoxValue` of nanoERG). This box is spendable from the wallet's address and serves solely as a placeholder to ensure the transaction has at least one - concrete input. + concrete input and thus have access to input pools. -2. **Maintaining PoolHolderBoxes**: - - Every transaction spending a `PoolHolderBox` should also create a new one. This - maintains a continuous supply of `PoolHolderBoxes` for future transactions and +2. **Maintaining PoolAccessBoxes**: + - Every transaction spending `PoolAccessBox` should also create a new one in its outputs. + This maintains a continuous supply of `PoolAccessBoxes` for future transactions and preserves the input/output balance. - - Initially, the wallet software should create a sufficient number of `PoolHolderBoxes` to + - Initially, the wallet software should create a sufficient number of `PoolAccessBoxes` to enable the PooledInputs feature. Subsequent management of these boxes will be handled internally by the wallet, ensuring user-friendly operation. 3. **User Experience**: - The wallet can prompt the user to enable the "Pooled Inputs" feature once enough ERGs - are available, creating the necessary `PoolHolderBoxes` in a single transaction. - - istinguish `PoolHolderBoxes` from other boxes, they should have a specific register - value (e.g., `R4 = 0x012345`). The wallet can identify available `PoolHolderBoxes` - using (wallet's address, `R4 = 0x012345`, value == `PoolHolderBoxValue`). + are available, creating the necessary `PoolAccessBoxes` in a single transaction. + - To distinguish `PoolAccessBoxes` from other boxes, they can have a specific register + value (e.g., `R4 = 0x012345`). The wallet can identify available `PoolAccessBoxes` + using (wallet's address, `R4 = 0x012345`, value == `PoolAccessBoxValue`). ### Implementation Details @@ -219,24 +225,19 @@ wallet standard (can be published as separate EIP motivated by this proposal). ```scala case class PooledInput( address: SigmaProp, // defines the pool where UTXOs are selected from - filter: Option[Predicate], // Optional filter to select specific UTXOs value: Long, // Total value to be spent from the pool signature: Array[Byte] // Signature for the `address` sigma proposition ) ``` - - `SigmaProp` type: aka Sigma Proposition, specifies the source address of the UTXOs (including multisig). - - `Predicate` type: A filter condition that UTXOs must satisfy to be considered for selection. - This is implemented as ErgoTree lambda of type `Box => Boolean`, i.e. it takes a box as - argument and can filter UTXOs boxes based on their registers. + - `SigmaProp` type: aka Sigma Proposition, specifies the source address of the UTXOs (in its simplest form it is P2PK, but can also be multisig and more complex sigma trees). - `value`: The total amount of ERGs to be covered by inputs spent from the pool. - `signature`: The signature for the `address` sigma proposition, ensuring the transaction is authorized to spend from the pool. - The two fields `address` and `filter` define the UTXO pool as a list of boxes as if - returned by the following query: + The `address` field defines the UTXO pool as a list of boxes as if returned by the following query: ```sql SELECT * FROM UTXO as U - WHERE U.ergoTree = ErgoTree.fromSigmaProp(address) and filter(U) + WHERE U.ergoTree = ErgoTree.fromSigmaProp(address) ``` #### Transaction Format Extension @@ -258,26 +259,26 @@ wallet standard (can be published as separate EIP motivated by this proposal). - When the `pools` list contain addresses belonging to different wallets, then properly generated signatures has to be provided in the `signature` fields of the corresponding `PooledInput`s. + - Technically, the format versioning is done using reusing one of the counts which is greater than + MaxArrayLength (e.g. 100000), for example data inputs count. This is similar to how + versioning is done in BlockTransactionsSerializer. #### Validation and Processing Logic + During transaction validation, the Ergo node will: - - **Parse the `pools` field**: + - **Get the list of PooledInputs**: ```scala val pooledInputs = transaction.pools ``` - - **For each `PooledInput`, select UTXOs** from the specified address that satisfy the - filter and meet the required value: + - **For each `PooledInput`, select UTXOs** from the specified address that sum up to the required value: ```scala pooledInputs.foreach { pooledInput => - val candidateUtxos = getUtxosFromAddress(pooledInput.address).filter(pooledInput.filter.getOrElse(_ => true)) - val selectedUtxos = selectUtxos(candidateUtxos, pooledInput.value) + val selectedUtxos = selectUtxos(pooledInput.address, pooledInput.value) transaction.inputs.appendAll(selectedUtxos.map(createInput)) } ``` - - `getUtxosFromAddress(address)`: Retrieves all UTXOs associated with the given address. - - `filter(predicate)`: Applies the optional filter to the UTXOs. - - `selectUtxos(utxos, value)`: Selects UTXOs from the filtered list until the total - value meets or exceeds the specified value. + - `selectUtxos(address, value)`: Selects UTXOs from the address until the total + value meets or exceeds the specified value (see the `def` below). #### Extension of UTXO Storage @@ -403,7 +404,6 @@ of UTXOs for unpooling. tx.pools.flatMap { pooledInput => val candidateUtxos = utxoState .getUtxos(pooledInput.address) - .filter(pooledInput.filter.getOrElse(_ => true)) val selectedUtxos = selectUtxos(candidateUtxos, pooledInput.value) val concretizedInputs = selectedUtxos.map(u => createInput(u)) transaction.inputs.appendAll(concretizedInputs) @@ -474,17 +474,17 @@ the whole chain of transactions will be also cancelled/delayed. Note: - to ensure the security and uniqueness of transactions involving PooledInputs, -Alice's wallet will include a `PoolHolderBox` in each transaction. -- many holder boxes can be created initially to enable PooledInputs feature in the wallet -- the number of holder boxes defines the maximum number of transactions that can be created +Alice's wallet will include a `PoolAccessBox` in each transaction. +- many access boxes can be created initially to enable PooledInputs feature in the wallet +- the number of access boxes defines the maximum number of transactions that can be created in parallel without using chaining. 1. **Initial Setup**: - **Wallet**: Alice's wallet contains the following UTXOs: - UTXO1: 100 ERG - - PoolHolderBox1: 0.001 ERG (pre-defined constant `PoolHolderBoxValue`) - - PoolHolderBox2: 0.001 ERG - - PoolHolderBox3: 0.001 ERG + - PoolAccessBox1: 0.001 ERG (pre-defined constant `PoolAccessBoxValue`) + - PoolAccessBox2: 0.001 ERG + - PoolAccessBox3: 0.001 ERG 2. **Staking**: - **Transaction 1**: Alice creates a transaction to stake 40 ERG. @@ -492,10 +492,10 @@ Alice's wallet will include a `PoolHolderBox` in each transaction. - Address: Alice's Wallet Address - Value: 40 ERG - Signature: Alice's Signature - - **Concrete Input**: PoolHolderBox1 (0.001 ERG) + - **Concrete Input**: PoolAccessBox1 (0.001 ERG) - **Outputs**: - StakeOutput: 40 ERG to the staking pool - - PoolHolderBox4: 0.001 ERG to Alice's wallet + - PoolAccessBox4: 0.001 ERG to Alice's wallet 3. **Yield Farming**: - **Transaction 2**: Alice creates a transaction to deposit 35 ERG into the yield farming pool. @@ -503,10 +503,10 @@ Alice's wallet will include a `PoolHolderBox` in each transaction. - Address: Alice's Wallet Address - Value: 35 ERG - Signature: Alice's Signature - - **Concrete Input**: PoolHolderBox2 (0.001 ERG) + - **Concrete Input**: PoolAccessBox2 (0.001 ERG) - **Outputs**: - YieldFarmOutput: 35 ERG to the yield farming pool - - PoolHolderBox5: 0.001 ERG to Alice's wallet + - PoolAccessBox5: 0.001 ERG to Alice's wallet 4. **Automated Trading**: - **Transaction 3**: The bot creates a transaction to execute a trade with 20 ERG. @@ -514,10 +514,10 @@ Alice's wallet will include a `PoolHolderBox` in each transaction. - Address: Alice's Wallet Address - Value: 20 ERG - Signature: Alice's Signature - - **Concrete Input**: PoolHolderBox3 (0.001 ERG) + - **Concrete Input**: PoolAccessBox3 (0.001 ERG) - **Outputs**: - TradeOutput: 20 ERG to the trading pool - - PoolHolderBox6: 0.001 ERG to Alice's wallet + - PoolAccessBox6: 0.001 ERG to Alice's wallet This workflow ensures that each transaction has a unique `messageToSign`, preventing double-spending attacks while maintaining the flexibility and efficiency of PooledInputs. @@ -530,7 +530,7 @@ double-spending attacks while maintaining the flexibility and efficiency of Pool #### Conclusion -The introduction of Pooled Transaction Inputs and the use of `PoolHolderBox` significantly +The introduction of Pooled Transaction Inputs and the use of `PoolAccessBox` significantly streamline the process of managing multiple DeFi interactions for users like Alice. By allowing transactions to specify input pools rather than fixed UTXOs, and ensuring each transaction includes at least one concrete input, users can engage in various DeFi @@ -539,12 +539,12 @@ approach not only enhances transaction flexibility and efficiency but also preve double-spending attacks, enabling a more secure and responsive DeFi experience on UTXO blockchain platform such as Ergo. -Furthermore, Pooled Transaction Inputs and `PoolHolderBox` mitigate challenges associated +Furthermore, Pooled Transaction Inputs and `PoolAccessBox` mitigate challenges associated with UTXO consolidation, allowing users to maximize their participation in the DeFi ecosystem without transaction processing limitations. This proposal represents a significant step forward in the scalability and usability of the Ergo blockchain, paving the way for more complex and dynamic DeFi applications. -The implementation of Pooled Transaction Inputs and `PoolHolderBox` as a wallet convention +The implementation of Pooled Transaction Inputs and `PoolAccessBox` as a wallet convention simplifies UTXO management, ensuring that transactions are both secure and efficient. This user-friendly approach enhances the DeFi experience on the Ergo blockchain. From de096129d147285dcb6a06f6b6549038b99b6eaf Mon Sep 17 00:00:00 2001 From: Alexander Slesarenko Date: Sun, 7 Jul 2024 22:13:05 +0100 Subject: [PATCH 06/12] eip47: finalized --- eip-0047.md | 139 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 93 insertions(+), 46 deletions(-) diff --git a/eip-0047.md b/eip-0047.md index c635f32d..78beac27 100644 --- a/eip-0047.md +++ b/eip-0047.md @@ -280,31 +280,10 @@ corresponding `PooledInput`s. - `selectUtxos(address, value)`: Selects UTXOs from the address until the total value meets or exceeds the specified value (see the `def` below). -#### Extension of UTXO Storage - -To support the deterministic selection of UTXOs from a pool, we need to extend the UTXO -storage with additional information about each UTXO. Specifically, we need to store the -`blockHeight` and `outputIndex` for each UTXO as described below in order to be able to -determine the order of UTXOs in the pool. - -The UTXO storage needs to be extended to include the following additional fields for each -UTXO: - -- `blockHeight`: The height of the block in which the UTXO was created. This is calculated - as the height of the block where UTXO is created. - -- `outputIndex`: This is calculated as the index of the output in the list of outputs of - the full block that created the UTXO. - -These fields are calculated in the `UtxoState.applyModifier` method during the processing -of each `Insert` operation. The `Insert` operation is submitted to the -`persistentProver.performOneOperation` method, which updates the UTXO set in the -persistent storage. - #### Deterministic Box Selection To ensure deterministic and efficient selection of UTXOs for a given `PooledInput`, we use -LevelDB to efficiently (i.e. O(log(n)) time) store and retrieve UTXOs in an ordered manner +LevelDB to efficiently (i.e. logarithmic time) store and retrieve UTXOs in an ordered manner (using lexically ordered keys). ##### UTXO Storage and Key Design @@ -313,41 +292,96 @@ Each UTXO is stored with a key consisting of `(address, blockHeight, outputIndex `boxId` as the value. The UTXO data itself is stored separately in the `PersistentBatchAVLProver` (field in `UtxoState`) with the `boxId` as the key. +The following method can be used to create the ordered key for a given UTXO: ```scala def createKey(address: String, blockHeight: Long, outputIndex: Int): String = { f"$address%s-$blockHeight%010d-$outputIndex%05d" } ``` +Thus, the keys are ordered first by `address`, then by `blockHeight`, and finally by `outputIndex`. + +#### Obtaining UTXO Ordering Details + +To support the deterministic selection of UTXOs from a pool, we need to obtain for each UTXO +the additional information about its address and ordering. Specifically, we need to compute +`address`, `blockHeight` and `outputIndex` for each UTXO in order to be able to determine +the order of UTXOs in the pool. + +```scala +case class UtxoInfo(addressOpt: Option[String], blockHeight: Int, outputIndex: Int) +``` + +- `addressOpt`: This is the optional pool's address associated with the UTXO. +- +- `blockHeight`: This is calculated as the height of the block where UTXO is created. + +- `outputIndex`: This is calculated as the index of the output in the list of outputs of + the full block that created the UTXO. + +These fields are calculated in the `UtxoState.stateChanges` method during the processing +of the full block. ##### Adding and Removing UTXOs UTXOs are added to and removed from both the `indexDb` and the `PersistentBatchAVLProver`. The logic is as follows: +1. Block is processed and a list of state changes is obtained. Each state change is a pair + of `(Operation, UtxoInfo)`, where `Operation` is either `Insert` or `Remove` and + `UtxoInfo` is a case class containing the UTXO details, which can be unambiguously + obtained during the block processing. + +```scala +val stateChanges = ErgoState.stateChanges(block) // compute state changes for the block +``` + +2. Once the block is validated and the state changes are obtained, they are processed as + follows: ```scala -val stateChanges = ErgoState.stateChanges(block) -stateChanges.foreach { change => change match { - case (Insert(k, v), utxo) => - val key = createKey(utxo.address, utxo.blockHeight, utxo.outputIndex) +stateChanges.foreach { + case (Insert(boxId, boxBytes), utxo) => + utxo.addressOpt.foreach { address => + // When UTXO can be related to a pool, add it to the indexDb + val key = createKey(address, utxo.blockHeight, utxo.outputIndex) indexDb.put(key, utxo.boxId) - persistentProver.performOneOperation(Insert(k, v)) // existing code: add UTXO to the persistent storage - case Remove(boxId) => - val utxo = persistentProver.unauthenticatedLookup(boxId).get - val key = createKey(utxo.address, utxo.blockHeight, utxo.outputIndex) + } + // add UTXO to the persistent storage (existing code of the current version) + persistentProver.performOneOperation(Insert(boxId, boxBytes)) + case (Remove(boxId), utxo) => + utxo.addressOpt.foreach { address => + // When UTXO is spent from a pool, remove it to the indexDb + val key = createKey(address, utxo.blockHeight, utxo.outputIndex) indexDb.delete(key) - persistentProver.performOneOperation(Remove(boxId)) -}} + } + // remove UTXO from the persistent storage (existing code of the current version) + persistentProver.performOneOperation(Remove(boxId)) +} ``` + +#### Extension of UTXO Storage + +To efficiently obtain UtxoInfo for each UTXO that is being spent, we need to extend the UTXO +storage with additional information. Specifically, we need to store the +`blockHeight` and `outputIndex` for each new UTXO that is created. + +These fields can be retrieved later in the `UtxoState.stateChanges` method for each input +box, during the processing of a full block. + +To store this pair of parameters, the `value` field of the `Insert` operation, which is +submitted to the `persistentProver.performOneOperation` method need to be extended with +the above information (serialized UTXO bytes followed by serialized `blockHeight` and +`outputIndex`). + ##### Selecting UTXOs for Unpooling -To select UTXOs for a given PooledInput, we use an iterator to retrieve UTXOs in order +To select UTXOs for a given PooledInput, we use an iterator to retrieve UTXOs in the order until the required value is met. The logic can be implemented as follows: ```scala def selectUtxos(address: String, requiredValue: Long): Seq[UTXO] = { val iter = indexDb.iterator() val prefix = f"$address%s-" - iter.seek(prefix.getBytes) + iter.seek(prefix.getBytes) // position the iterator at the first UTXO with the given address var selectedUtxos = Seq.empty[UTXO] var totalSelectedValue = 0L @@ -402,9 +436,7 @@ of UTXOs for unpooling. ```scala def concretizeTransaction(tx: ErgoTransaction, utxoState: UtxoState): Unit = { tx.pools.flatMap { pooledInput => - val candidateUtxos = utxoState - .getUtxos(pooledInput.address) - val selectedUtxos = selectUtxos(candidateUtxos, pooledInput.value) + val selectedUtxos = selectUtxos(pooledInput.address, pooledInput.value) val concretizedInputs = selectedUtxos.map(u => createInput(u)) transaction.inputs.appendAll(concretizedInputs) addChangeOutput(selectedUtxos, pooledInput, transaction) @@ -419,6 +451,10 @@ of UTXOs for unpooling. - This ensures that each `PooledInput` is processed in the order they appear, with deterministic UTXO selection and conflict-free execution. +Transaction concretization ensures that each transaction is processed with concrete inputs +as if it was a standard transaction fully constructed and signed off-chain. This also +means PooledInput mechanism is fully transparent to the smart contracts evaluation. + ### Example: Alice's Multi-Step DeFi Workflow Let's create a detailed example involving Alice's participation in multiple DeFi protocols, requiring her to sign transactions and manage her UTXOs efficiently using the @@ -463,7 +499,7 @@ to handle these interactions efficiently to maximize her returns. - ChangeOutput: 5 ERG back to Alice's wallet - Wallet: UTXO4: 5 ERG (unconfirmed) -Note, in order for Alice to execute the above transaction in parallel, they all need to be +Note, in order for Alice to execute the above transaction at the same time, they all need to be chained together. This creates hard dependence between the transactions and requires all the previous (parent) transactions to be successful for child transactions to also be successful. This rules out any possibility for cancellation or transaction queueing on @@ -518,26 +554,37 @@ Alice's wallet will include a `PoolAccessBox` in each transaction. - **Outputs**: - TradeOutput: 20 ERG to the trading pool - PoolAccessBox6: 0.001 ERG to Alice's wallet - + - Wallet: UTXO: 5 ERG; PAB4: 0.001 ERG; PAB5: 0.001 ERG; PAB6: 0.001 ERG (unconfirmed) + This workflow ensures that each transaction has a unique `messageToSign`, preventing double-spending attacks while maintaining the flexibility and efficiency of PooledInputs. #### Benefits of Pooled Transaction Inputs + - wallet doesn't need to chain transactions and select UTXOs explicitly - no need to add change outputs explicitly (the change is handled by the network on the protocol level) - transactions can be sent in any order (or all at once) and don't depend on each other - any transaction can be cancelled/delayed/queued without affecting others +- resolves the trade-off between UTXO consolidation and transaction efficiency, it became + possible to consolidate UTXOs to a single box and still be able to participate in multiple + DeFi protocols simultaneously and independently +- combined with atomic transaction chains (see EIP-46) enables atomic transaction + composition, which will allow to avoid transaction conflicts in a systematic and generic + way +- enables complex atomic transaction DAGs to be submitted to the mempool + and then process by the network without requiring wallet to manage the dependencies, or + even be connected #### Conclusion The introduction of Pooled Transaction Inputs and the use of `PoolAccessBox` significantly -streamline the process of managing multiple DeFi interactions for users like Alice. By -allowing transactions to specify input pools rather than fixed UTXOs, and ensuring each -transaction includes at least one concrete input, users can engage in various DeFi -protocols simultaneously without chaining transactions or managing UTXO dependencies. This -approach not only enhances transaction flexibility and efficiency but also prevents -double-spending attacks, enabling a more secure and responsive DeFi experience on UTXO -blockchain platform such as Ergo. +streamline the process of managing multiple DeFi simultaneous interactions for users like +Alice. By allowing transactions to specify input pools rather than fixed UTXOs, and +ensuring each transaction includes at least one concrete input, users can engage in +various DeFi protocols simultaneously without chaining transactions or managing UTXO +dependencies. This approach not only enhances transaction flexibility and efficiency but +also prevents double-spending attacks, enabling a more secure and responsive DeFi +experience on UTXO blockchain platform such as Ergo. Furthermore, Pooled Transaction Inputs and `PoolAccessBox` mitigate challenges associated with UTXO consolidation, allowing users to maximize their participation in the DeFi From b5542e6a21f8e3a7ed7a5b16c90e8a82bf919cb1 Mon Sep 17 00:00:00 2001 From: Alexander Slesarenko Date: Mon, 8 Jul 2024 15:21:42 +0100 Subject: [PATCH 07/12] eip47: fixes and cleanup --- eip-0047.md | 105 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 66 insertions(+), 39 deletions(-) diff --git a/eip-0047.md b/eip-0047.md index 78beac27..e2f3a8b1 100644 --- a/eip-0047.md +++ b/eip-0047.md @@ -1,12 +1,40 @@ - Author: aslesarenko - Status: Proposed -- Created: 07-July-2024 -- Last edited: 07-July-2024 +- Created: 07.07.2024 +- Last edited: 07.07.2024 - License: CC0 -- Forking: soft-fork +- Forking: soft-fork required # EIP-0048: Pooled Transaction Inputs +- [Background and Motivation](#background-and-motivation) + - [The Problem](#the-problem) + - [Illustrative Example](#illustrative-example) + - [The Solution Idea](#the-solution-idea) +- [Proposed Solution](#proposed-solution) + - [Key Components of the Solution](#key-components-of-the-solution) + - [Avoiding Double-Spending with PooledInputs](#avoiding-double-spending-with-pooledinputs) + - [Solution to Double-Spending Risk](#solution-to-double-spending-risk) + - [Wallet Standard for PooledInputs](#wallet-standard-for-pooledinputs) + - [Implementation Details](#implementation-details) + - [PooledInput Data Structure](#pooledinput-data-structure) + - [Transaction Format Extension](#transaction-format-extension) + - [Validation and Processing Logic](#validation-and-processing-logic) + - [Deterministic Box Selection](#deterministic-box-selection) + - [UTXO Storage and Key Design](#utxo-storage-and-key-design) + - [Obtaining UTXO Ordering Details](#obtaining-utxo-ordering-details) + - [Adding and Removing UTXOs](#adding-and-removing-utxos) + - [Extension of UTXO Storage](#extension-of-utxo-storage) + - [Selecting UTXOs for Unpooling](#selecting-utxos-for-unpooling) + - [Change Output Handling](#change-output-handling) + - [Transaction Concretization](#transaction-concretization) +- [Example: Alice's Multi-Step DeFi Workflow](#example-alices-multi-step-defi-workflow) + - [Context](#context) + - [Current Step-by-Step Workflow with explicit UTXO selection](#current-step-by-step-workflow-with-explicit-utxo-selection) + - [New step-by-step Workflow with Pooled Inputs](#new-step-by-step-workflow-with-pooled-inputs) +- [Benefits of Pooled Transaction Inputs](#benefits-of-pooled-transaction-inputs) +- [Conclusion](#conclusion) + ## Background and Motivation In the current implementation of the Ergo blockchain, each transaction spends a list of @@ -18,23 +46,24 @@ changed without rebuilding the transaction and re-signing it again. ### The Problem This creates challenges when multiple transactions need to be created from the same wallet -in short period of time in response to external events (e.g. when participating in multiple -DeFi protocols). Two transactions cannot share the -same input box, so the wallet must track which boxes are already spent and which are not, -selecting non-overlapping subsets of UTXOs. However, this is not always feasible, -especially when UTXOs are consolidated into a single box, resulting in a situation where -only one transaction can be created from that single UTXO and other transactions has to be -chained which by itself creates undesirable dependency on the parent transaction so that -cancellation/delaying/queueing of the parent transaction will also cancel/delay/queue the -child transaction. - -This problem is exacerbated in scenarios requiring responsiveness. For -example, if a user participates in two or more DeFi protocols involving some form of -transaction queueing and needs to react to multiple events quickly, user cannot chain -transactions belonging to different DeFi protocols, but has to submit them quickly in -response to the events. In such cases, the consolidation of UTXOs -can become a bottleneck, limiting the number of transactions that can be executed in -parallel. +in short period of time in response to external events (e.g. when participating in +multiple DeFi protocols). Two transactions cannot share the same input box, so the wallet +must track which boxes are already spent and which are not, selecting non-overlapping +subsets of UTXOs. + +However, this is not always feasible, especially when UTXOs are consolidated into a single +box, resulting in a situation where only one transaction can be created from that single +UTXO and other transactions has to be chained. The chaining by itself creates undesirable +dependency on the parent transaction so that cancellation/delaying/queueing of the parent +transaction will also cancel/delay/queue the child transaction. + +This problem is exacerbated in scenarios requiring responsiveness. For example, if a user +participates in two or more DeFi protocols involving some form of transaction queueing and +needs to react to multiple events quickly, user cannot chain transactions belonging to +different DeFi protocols (to avoid dependencies), but has to submit them simultaneously in +response to the events. In such cases, the consolidation of UTXOs can become a bottleneck, +limiting the number of transactions that can be executed in parallel without undesirable +chaining. ### Illustrative Example @@ -42,7 +71,7 @@ Consider Alice, a user who actively participates in multiple DeFi protocols on t blockchain. Each protocol requires Alice to submit transactions in response to various events, such as staking rewards, yield farming updates, or automated trading triggers. These interactions need to be executed promptly to maximize her returns and maintain her -positions. +positions, which can lead to multiple simultaneous, but independent transactions. Scenario: - Alice stakes some ERGs in a DeFi staking pool, which periodically pays out rewards. @@ -52,20 +81,18 @@ Scenario: market signals. Each of these DeFi protocols requires Alice to submit transactions spending from her -wallet in response to the protocol-specific events. However, these transactions cannot be chained -together as they are built independently by different dApps. If Alice's wallet has +wallet in response to the protocol-specific events. However, chaining of these transactions +together is undesirable as they are built independently by different dApps. If Alice's wallet has consolidated its funds into a single UTXO for storage efficiency, she can only process one -transaction at a time, leading to delays and potentially missed opportunities. - -For example, when a staking reward is paid out, Alice needs to quickly reinvest -these rewards into the yield farming protocol. Simultaneously, the trading bot may need to -execute a trade based on a market signal. If all these transactions depend on the same -UTXO, only one can be processed at a time, creating a bottleneck. +transaction at a time, leading to delays and potentially missed opportunities. +Alternatively, Alice can chain transactions, dealing with undesirable dependencies and +risks of cancellation/delaying/queueing of the parent transaction affecting the whole chain. Alternatively, Alice can de-consolidate her UTXOs, but this additional step will worsen -user experience and increase time and cost of transactions. Even if implemented by wallet, +user experience and increase time and cost of transactions. Even if implemented by a wallet, such add-hoc consolidation/de-consolidation once in a while will bottleneck Alice in her -DeFi experience. +DeFi experience. Moreover, not being implemented on network protocol level, automatic +de-consolidation has to be enforced across all wallets to enable consistent user experience. ### The Solution Idea @@ -300,7 +327,7 @@ def createKey(address: String, blockHeight: Long, outputIndex: Int): String = { ``` Thus, the keys are ordered first by `address`, then by `blockHeight`, and finally by `outputIndex`. -#### Obtaining UTXO Ordering Details +##### Obtaining UTXO Ordering Details To support the deterministic selection of UTXOs from a pool, we need to obtain for each UTXO the additional information about its address and ordering. Specifically, we need to compute @@ -372,7 +399,7 @@ submitted to the `persistentProver.performOneOperation` method need to be extend the above information (serialized UTXO bytes followed by serialized `blockHeight` and `outputIndex`). -##### Selecting UTXOs for Unpooling +#### Selecting UTXOs for Unpooling To select UTXOs for a given PooledInput, we use an iterator to retrieve UTXOs in the order until the required value is met. The logic can be implemented as follows: @@ -455,17 +482,17 @@ Transaction concretization ensures that each transaction is processed with concr as if it was a standard transaction fully constructed and signed off-chain. This also means PooledInput mechanism is fully transparent to the smart contracts evaluation. -### Example: Alice's Multi-Step DeFi Workflow +## Example: Alice's Multi-Step DeFi Workflow Let's create a detailed example involving Alice's participation in multiple DeFi protocols, requiring her to sign transactions and manage her UTXOs efficiently using the Pooled Transaction Inputs mechanism. -#### Context +### Context Alice is an active participant in the DeFi ecosystem on the Ergo blockchain. She engages in multiple protocols, including staking, yield farming, and automated trading. She needs to handle these interactions efficiently to maximize her returns. -#### Current Step-by-Step Workflow with explicit UTXO selection +### Current Step-by-Step Workflow with explicit UTXO selection 1. **Initial Setup:** - **Wallet:** Alice's wallet contains the following UTXOs: @@ -506,7 +533,7 @@ successful. This rules out any possibility for cancellation or transaction queue DeFi protocol side. If Transaction 1 is cancelled/delayed by the DeFi protocol, then the whole chain of transactions will be also cancelled/delayed. -#### New step-by-step Workflow with Pooled Inputs +### New step-by-step Workflow with Pooled Inputs Note: - to ensure the security and uniqueness of transactions involving PooledInputs, @@ -559,7 +586,7 @@ Alice's wallet will include a `PoolAccessBox` in each transaction. This workflow ensures that each transaction has a unique `messageToSign`, preventing double-spending attacks while maintaining the flexibility and efficiency of PooledInputs. -#### Benefits of Pooled Transaction Inputs +## Benefits of Pooled Transaction Inputs - wallet doesn't need to chain transactions and select UTXOs explicitly - no need to add change outputs explicitly (the change is handled by the network on the protocol level) @@ -589,7 +616,7 @@ experience on UTXO blockchain platform such as Ergo. Furthermore, Pooled Transaction Inputs and `PoolAccessBox` mitigate challenges associated with UTXO consolidation, allowing users to maximize their participation in the DeFi ecosystem without transaction processing limitations. This proposal represents a -significant step forward in the scalability and usability of the Ergo blockchain, paving +step forward in the scalability and usability of the Ergo blockchain, paving the way for more complex and dynamic DeFi applications. The implementation of Pooled Transaction Inputs and `PoolAccessBox` as a wallet convention From 1791bebb34f0aba710c93123358d4cc0d606c52e Mon Sep 17 00:00:00 2001 From: Alexander Slesarenko Date: Mon, 8 Jul 2024 16:19:51 +0100 Subject: [PATCH 08/12] eip47: Related Work and References --- eip-0047.md | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 2 deletions(-) diff --git a/eip-0047.md b/eip-0047.md index e2f3a8b1..8267eca3 100644 --- a/eip-0047.md +++ b/eip-0047.md @@ -311,7 +311,8 @@ corresponding `PooledInput`s. To ensure deterministic and efficient selection of UTXOs for a given `PooledInput`, we use LevelDB to efficiently (i.e. logarithmic time) store and retrieve UTXOs in an ordered manner -(using lexically ordered keys). +(using lexically ordered keys). This will allow to implement simple FIFO selection of +UTXOs [1] on-chain, as part of Layer 1 protocol. ##### UTXO Storage and Key Design @@ -602,7 +603,49 @@ double-spending attacks while maintaining the flexibility and efficiency of Pool and then process by the network without requiring wallet to manage the dependencies, or even be connected -#### Conclusion +## Related Work + +To answer the question regarding prior work that proposes to move coin selection from +off-chain to on-chain and make it part of Layer 1 protocol, we conducted a search for +relevant publications in the blockchain space. Here are some notable examples and +discussions: + +**Research Papers and Proposals** + + Various academic papers discuss coin selection + strategies, focusing on optimizing transaction costs and sizes. These discussions often + highlight the limitations of off-chain coin selection and suggest improvements, some of + which could potentially be integrated into the protocol layer. For example, the paper "A + Survey on Coin Selection Algorithms in UTXO-based Blockchains" provides a comprehensive + review and categorizes existing algorithms, potentially paving the way for on-chain + implementations [1]. + +**UTXO Management and Coin Selection**: + + *MACS Algorithm*: This algorithm, although not explicitly moving coin selection +on-chain, addresses many foundational issues with UTXO management by optimizing +transaction costs and sizes, enhancing privacy, and efficiently managing the UTXO pool. It +considers the age of transactions to promote the spending of older UTXOs, which can be +seen as a step towards more dynamic and protocol-level coin management [2]. + +**Dynamic UTXO Selection**: + + Utreexo by MIT Digital Currency Initiative: Utreexo introduces a dynamic hash-based +accumulator for UTXO management that operates in a more efficient manner compared to +traditional methods. It doesn't explicitly move coin selection on-chain but proposes +significant changes to how UTXOs are managed and verified, indirectly affecting the coin +selection process [3]. + +Despite these advancements, it appears that explicitly moving the entire coin selection +process on-chain as a part of the Layer 1 protocol is still a relatively novel idea. The +existing research and proposals primarily focus on optimizing the current off-chain +processes or introducing hybrid approaches. + +For a deeper dive into the specifics, you can explore the links listed in the References +section. These sources provide foundational insights and potential directions for future +research and implementation in the area of coin selection. + +## Conclusion The introduction of Pooled Transaction Inputs and the use of `PoolAccessBox` significantly streamline the process of managing multiple DeFi simultaneous interactions for users like @@ -622,3 +665,8 @@ the way for more complex and dynamic DeFi applications. The implementation of Pooled Transaction Inputs and `PoolAccessBox` as a wallet convention simplifies UTXO management, ensuring that transactions are both secure and efficient. This user-friendly approach enhances the DeFi experience on the Ergo blockchain. + +## References +1. [A Survey on Coin Selection Algorithms in UTXO-based Blockchains](https://cs.paperswithcode.com/paper/a-survey-on-coin-selection-algorithms-in-utxo) +2. [MACS: A new approach to multi-asset coin selection](https://cardanofoundation.org/blog/macs-a-new-approach-to-multi-asset-coin-selection) +3. [Utreexo: A dynamic hash-based accumulator optimized for the Bitcoin UTXO set](https://dci.mit.edu/dci-news/2019/6/6/utreexo-a-dynamic-hash-based-accumulator-optimized-for-the-bitcoin-utxo-set) From 5d599495e2cb49037fb9bce83115d8e4c8dfbb0e Mon Sep 17 00:00:00 2001 From: Alexander Slesarenko Date: Mon, 8 Jul 2024 17:59:31 +0100 Subject: [PATCH 09/12] eip47: more proof reading --- eip-0047.md | 199 +++++++++++++++++++++++++++++----------------------- 1 file changed, 110 insertions(+), 89 deletions(-) diff --git a/eip-0047.md b/eip-0047.md index 8267eca3..979f04aa 100644 --- a/eip-0047.md +++ b/eip-0047.md @@ -1,3 +1,5 @@ +# EIP-0048: Pooled Transaction Inputs + - Author: aslesarenko - Status: Proposed - Created: 07.07.2024 @@ -5,8 +7,7 @@ - License: CC0 - Forking: soft-fork required -# EIP-0048: Pooled Transaction Inputs - +## Contents - [Background and Motivation](#background-and-motivation) - [The Problem](#the-problem) - [Illustrative Example](#illustrative-example) @@ -15,24 +16,25 @@ - [Key Components of the Solution](#key-components-of-the-solution) - [Avoiding Double-Spending with PooledInputs](#avoiding-double-spending-with-pooledinputs) - [Solution to Double-Spending Risk](#solution-to-double-spending-risk) - - [Wallet Standard for PooledInputs](#wallet-standard-for-pooledinputs) - - [Implementation Details](#implementation-details) - - [PooledInput Data Structure](#pooledinput-data-structure) - - [Transaction Format Extension](#transaction-format-extension) - - [Validation and Processing Logic](#validation-and-processing-logic) - - [Deterministic Box Selection](#deterministic-box-selection) - - [UTXO Storage and Key Design](#utxo-storage-and-key-design) - - [Obtaining UTXO Ordering Details](#obtaining-utxo-ordering-details) - - [Adding and Removing UTXOs](#adding-and-removing-utxos) - - [Extension of UTXO Storage](#extension-of-utxo-storage) - - [Selecting UTXOs for Unpooling](#selecting-utxos-for-unpooling) - - [Change Output Handling](#change-output-handling) - - [Transaction Concretization](#transaction-concretization) + - [Wallet Standard for PooledInputs](#wallet-standard-for-pooledinputs) +- [Implementation Details](#implementation-details) + - [PooledInput Data Structure](#pooledinput-data-structure) + - [Transaction Format Extension](#transaction-format-extension) + - [Validation and Processing Logic](#validation-and-processing-logic) + - [Deterministic Box Selection](#deterministic-box-selection) + - [Ordered access to UTXO Storage](#ordered-access-to-utxo-storage) + - [Obtaining UTXO Ordering Details](#obtaining-utxo-ordering-details) + - [Adding and Removing UTXOs](#adding-and-removing-utxos) + - [Extension of UTXO Storage](#extension-of-utxo-storage) + - [Selecting UTXOs for Unpooling](#selecting-utxos-for-unpooling) + - [Change Output Handling](#change-output-handling) + - [Transaction Concretization](#transaction-concretization) - [Example: Alice's Multi-Step DeFi Workflow](#example-alices-multi-step-defi-workflow) - [Context](#context) - [Current Step-by-Step Workflow with explicit UTXO selection](#current-step-by-step-workflow-with-explicit-utxo-selection) - [New step-by-step Workflow with Pooled Inputs](#new-step-by-step-workflow-with-pooled-inputs) - [Benefits of Pooled Transaction Inputs](#benefits-of-pooled-transaction-inputs) +- [Related Work](#related-work) - [Conclusion](#conclusion) ## Background and Motivation @@ -131,7 +133,7 @@ of transaction creation and execution on the Ergo blockchain. ### Key Components of the Solution 1. **PooledInput Data Structure** which contains the following fields: - - `address: ErgoAddress`: Specifies the address from which UTXOs should be selected. + - `address: SigmaProp`: Specifies the address from which UTXOs should be selected. - `value: Long`: The total value to be spent from the pool. 2. **Transaction Format Extension:** @@ -146,25 +148,23 @@ of transaction creation and execution on the Ergo blockchain. 4. **Deterministic Box Selection:** - Implement a deterministic selection algorithm to ensure all nodes select the same UTXOs - from the pool. This can be achieved by sorting the pool boxes by their order of inclusion - into the UTXO set (which is defined by the global order of transactions in the - blockchain). The first boxes from the ordered list are selected until the total value - meets or exceeds the required `value`. + from the pool. This can be achieved by using FIFO selector [1], i.e. selecting the + pool boxes by their order of inclusion into the UTXO set (which is defined by the + global order of transactions in the blockchain). The boxes from the ordered pool + are selected until the total value meets or exceeds the required `value`. 5. **LevelDB Integration for Efficient Box Selection:** - - To efficiently manage UTXOs and ensure deterministic box selection, we propose leveraging + - To efficiently manage UTXOs and ensure deterministic box selection, we propose leveraging LevelDB's ordered key-value storage. This approach maintains UTXOs in an ordered fashion based on `(address, blockHeight, outputIndex)`, where the value stored is the `boxId` of - the UTXO. - - Key Design: - - Key: `(address, blockHeight, outputIndex)` formatted as a string to maintain order. - - Value: `boxId` stored as bytes. - - New Database Instance: - - `indexDb`: Stores the ordered keys with the UTXO `boxId` as the value. + the UTXO: + - Record Key: `(address, blockHeight, outputIndex)` formatted as a string to maintain order. + - Record Value: `boxId` stored as bytes. + - New Database Instance `indexDb` to store the ordered keys with the UTXO `boxId` as the value. 6. **Change Output Handling:** - - After selecting the required UTXOs, the transaction will generate a change output with + - After selecting the required UTXOs, the change output will be added with the remaining value (total value of selected UTXOs minus the `value` field of the `PooledInput`). This change output will be appended to the list of transaction outputs and the change amount will be returned to the address associated with the pool and @@ -203,32 +203,33 @@ is impractical for UTXO blockchains due to several reasons: - The transaction format and signing process would need to change to include the nonce, necessitating wallet software updates. -In this proposal, instead of using nonce, we address this issue by imposing a constraint -on the usage of PooledInputs and describe a convention for UTXO management that will make -this proposal both safe and also practical to implement. More on this in the Proposed -Solution section. +In this proposal, instead of using nonce, we address this issue by imposing a protocol +level constraint on the usage of PooledInputs and describe a wallet level convention for +UTXO management that will make this proposal both safe and also practical to implement. +More on this in the Proposed Solution section. ### Solution to Double-Spending Risk To address the possibility of double-spending attacks with PooledInputs, we propose a -constraint: a transaction with PooledInputs must include at least one concrete input. This -ensures that each transaction has a unique piece of data in its `messageToSign`, -preventing duplicate submissions. The Ergo node will enforce this constraint during -transaction validation. +constraint: _a transaction with PooledInputs must include at least one concrete input_. +This ensures that each transaction has a unique piece of data in its `messageToSign` +(concrete box id which is unique), preventing duplicate submissions. The Ergo node will +enforce this constraint during transaction validation. However, in many practical use cases, transactions with PooledInputs may not naturally have concrete inputs, as seen in the examples described earlier. To address this, we propose a convention for UTXO management supported by wallet software, which can be regarded as a wallet standard (can be published as separate EIP motivated by this proposal). -#### Wallet Standard for PooledInputs +### Wallet Standard for PooledInputs 1. **PoolAccessBox**: - - When creating a transaction with PooledInputs, the wallet should include a dummy input - box called `PoolAccessBox` with a minimal value (pre-defined constant - `PoolAccessBoxValue` of nanoERG). This box is spendable from the wallet's address - and serves solely as a placeholder to ensure the transaction has at least one - concrete input and thus have access to input pools. + - When creating a transaction with PooledInputs (when there is no concrete input), + the wallet should include a dummy input box called `PoolAccessBox` with a minimal value + (pre-defined constant `PoolAccessBoxValue` of nanoERG). This box is spendable from the + wallet's address and serves solely as a placeholder to ensure the transaction has at least + one concrete input and thus can have pooled inputs (have access to the _input pools_ + feature). 2. **Maintaining PoolAccessBoxes**: - Every transaction spending `PoolAccessBox` should also create a new one in its outputs. @@ -245,9 +246,9 @@ wallet standard (can be published as separate EIP motivated by this proposal). value (e.g., `R4 = 0x012345`). The wallet can identify available `PoolAccessBoxes` using (wallet's address, `R4 = 0x012345`, value == `PoolAccessBoxValue`). -### Implementation Details +## Implementation Details -#### PooledInput Data Structure +### PooledInput Data Structure The `PooledInput` structure can be defined in Scala as follows: ```scala case class PooledInput( @@ -266,8 +267,9 @@ wallet standard (can be published as separate EIP motivated by this proposal). SELECT * FROM UTXO as U WHERE U.ergoTree = ErgoTree.fromSigmaProp(address) ``` +The ordering of the boxes in the pool is defined below. -#### Transaction Format Extension +### Transaction Format Extension Extend the transaction format to include the `pools` field: ```scala case class ErgoTransaction( @@ -286,11 +288,13 @@ wallet standard (can be published as separate EIP motivated by this proposal). - When the `pools` list contain addresses belonging to different wallets, then properly generated signatures has to be provided in the `signature` fields of the corresponding `PooledInput`s. - - Technically, the format versioning is done using reusing one of the counts which is greater than - MaxArrayLength (e.g. 100000), for example data inputs count. This is similar to how - versioning is done in BlockTransactionsSerializer. + - Technically, the format versioning can be done by reusing one of the counts (e.g. data + input count). When the count is greater than MaxArrayLength (e.g. 100000) then it means + the transaction contains `pools` field, otherwise the list is empty, which means the + transaction is indistinguishable from the current protocol version. + This is similar to how versioning is done in BlockTransactionsSerializer. -#### Validation and Processing Logic +### Validation and Processing Logic During transaction validation, the Ergo node will: - **Get the list of PooledInputs**: @@ -304,17 +308,18 @@ corresponding `PooledInput`s. transaction.inputs.appendAll(selectedUtxos.map(createInput)) } ``` - - `selectUtxos(address, value)`: Selects UTXOs from the address until the total - value meets or exceeds the specified value (see the `def` below). + - `selectUtxos(address, value)`: Selects UTXOs from the pool (by the address) until the total + value meets or exceeds the specified value (see the `def selectUtxos` below). -#### Deterministic Box Selection +### Deterministic Box Selection To ensure deterministic and efficient selection of UTXOs for a given `PooledInput`, we use LevelDB to efficiently (i.e. logarithmic time) store and retrieve UTXOs in an ordered manner -(using lexically ordered keys). This will allow to implement simple FIFO selection of -UTXOs [1] on-chain, as part of Layer 1 protocol. +(using lexically ordered keys). This is an implementation detail and just one of the +possible ways to implement simple FIFO selection of UTXOs [1] on-chain, as part of Layer 1 +protocol. -##### UTXO Storage and Key Design +#### Ordered access to UTXO Storage Each UTXO is stored with a key consisting of `(address, blockHeight, outputIndex)` and the `boxId` as the value. The UTXO data itself is stored separately in the @@ -327,8 +332,10 @@ def createKey(address: String, blockHeight: Long, outputIndex: Int): String = { } ``` Thus, the keys are ordered first by `address`, then by `blockHeight`, and finally by `outputIndex`. +Such defined keys will allow to create an iterator and seek to the first UTXO with the +given address (see below). -##### Obtaining UTXO Ordering Details +#### Obtaining UTXO Ordering Details To support the deterministic selection of UTXOs from a pool, we need to obtain for each UTXO the additional information about its address and ordering. Specifically, we need to compute @@ -339,17 +346,19 @@ the order of UTXOs in the pool. case class UtxoInfo(addressOpt: Option[String], blockHeight: Int, outputIndex: Int) ``` -- `addressOpt`: This is the optional pool's address associated with the UTXO. -- +- `addressOpt`: This is the optional pool's address associated with the UTXO. It is + optional because not all UTXOs can be related to a pool (i.e. most boxes with PayToScript + addresses). + - `blockHeight`: This is calculated as the height of the block where UTXO is created. - `outputIndex`: This is calculated as the index of the output in the list of outputs of the full block that created the UTXO. -These fields are calculated in the `UtxoState.stateChanges` method during the processing -of the full block. +These fields can be calculated in the `UtxoState.stateChanges` method during the processing +of the full block (see below). -##### Adding and Removing UTXOs +#### Adding and Removing UTXOs UTXOs are added to and removed from both the `indexDb` and the `PersistentBatchAVLProver`. The logic is as follows: @@ -368,18 +377,22 @@ val stateChanges = ErgoState.stateChanges(block) // compute state changes for th ```scala stateChanges.foreach { case (Insert(boxId, boxBytes), utxo) => - utxo.addressOpt.foreach { address => - // When UTXO can be related to a pool, add it to the indexDb - val key = createKey(address, utxo.blockHeight, utxo.outputIndex) - indexDb.put(key, utxo.boxId) + utxo.addressOpt match { + case Some(address) => + // When UTXO can be related to a pool, add it to the indexDb + val key = createKey(address, utxo.blockHeight, utxo.outputIndex) + indexDb.put(key, utxo.boxId) + case _ => } // add UTXO to the persistent storage (existing code of the current version) persistentProver.performOneOperation(Insert(boxId, boxBytes)) case (Remove(boxId), utxo) => - utxo.addressOpt.foreach { address => - // When UTXO is spent from a pool, remove it to the indexDb - val key = createKey(address, utxo.blockHeight, utxo.outputIndex) - indexDb.delete(key) + utxo.addressOpt match { + case Some(address) => + // When UTXO is spent from a pool, remove it to the indexDb + val key = createKey(address, utxo.blockHeight, utxo.outputIndex) + indexDb.delete(key) + case _ => } // remove UTXO from the persistent storage (existing code of the current version) persistentProver.performOneOperation(Remove(boxId)) @@ -397,10 +410,9 @@ box, during the processing of a full block. To store this pair of parameters, the `value` field of the `Insert` operation, which is submitted to the `persistentProver.performOneOperation` method need to be extended with -the above information (serialized UTXO bytes followed by serialized `blockHeight` and -`outputIndex`). +the above information (serialized `blockHeight` and `outputIndex`). -#### Selecting UTXOs for Unpooling +### Selecting UTXOs for Unpooling To select UTXOs for a given PooledInput, we use an iterator to retrieve UTXOs in the order until the required value is met. The logic can be implemented as follows: @@ -435,10 +447,10 @@ def selectUtxos(address: String, requiredValue: Long): Seq[UTXO] = { } } ``` -By leveraging LevelDB's ordered storage, we ensure efficient and deterministic selection -of UTXOs for unpooling. +By leveraging LevelDB's ordered storage, we can implement efficient and deterministic FIFO +selection [1] of UTXOs for unpooling. -#### Change Output Handling +### Change Output Handling After selecting the UTXOs, an optional change output is created to return the remaining coins to the pool. This is done as follows: ```scala @@ -456,13 +468,13 @@ of UTXOs for unpooling. - `createChangeOutput(value, address)`: Creates a change output box with the specified value and address, note the use of the same address as the pool. -#### Transaction Concretization +### Transaction Concretization When a transaction have at least one `PooledInput`, the transaction needs to be concretized, i.e. the exact UTXOs to be spent need to be determined along with the change outputs. This process is called `input unpooling` and is done during block application and validation. The concretization process can be implemented as follows: ```scala - def concretizeTransaction(tx: ErgoTransaction, utxoState: UtxoState): Unit = { + def concretizeTransaction(tx: ErgoTransaction): Unit = { tx.pools.flatMap { pooledInput => val selectedUtxos = selectUtxos(pooledInput.address, pooledInput.value) val concretizedInputs = selectedUtxos.map(u => createInput(u)) @@ -472,16 +484,17 @@ of UTXOs for unpooling. } ``` - `concretizeTransaction`: Resolves `PooledInputs` into concrete inputs - based on the current UTXO state and adds change outputs. + based on FIFO selector and adds change outputs. - `createInput(utxo)`: Converts a UTXO into a transaction input. - Note, the method can handle multiple `PooledInput` by sequentially processing each entry in the `pools` list: - This ensures that each `PooledInput` is processed in the order they appear, with - deterministic UTXO selection and conflict-free execution. + deterministic FIFO UTXO selection and conflict-free execution. Transaction concretization ensures that each transaction is processed with concrete inputs as if it was a standard transaction fully constructed and signed off-chain. This also -means PooledInput mechanism is fully transparent to the smart contracts evaluation. +means PooledInput mechanism is fully transparent to the smart contracts evaluation as the +concretization happens before any contract is executed. ## Example: Alice's Multi-Step DeFi Workflow Let's create a detailed example involving Alice's participation in multiple DeFi @@ -587,8 +600,9 @@ Alice's wallet will include a `PoolAccessBox` in each transaction. This workflow ensures that each transaction has a unique `messageToSign`, preventing double-spending attacks while maintaining the flexibility and efficiency of PooledInputs. -## Benefits of Pooled Transaction Inputs +## Benefits +Here are the key benefits of the proposed Pooled Transaction Inputs mechanism: - wallet doesn't need to chain transactions and select UTXOs explicitly - no need to add change outputs explicitly (the change is handled by the network on the protocol level) - transactions can be sent in any order (or all at once) and don't depend on each other @@ -603,6 +617,14 @@ double-spending attacks while maintaining the flexibility and efficiency of Pool and then process by the network without requiring wallet to manage the dependencies, or even be connected +Here are some of the additional benefits of the proposed protocol level extensions: +- Box.isPoolable property: can be used in contracts to require/forbid specific boxes + (input or output) to be poolable. +- Box.blockHeight property: can be used in contracts to determine exact height when the + box was created (in contrast to creationHeight, which doesn't guarantee the exact height). +- Box.outputIndex property: can be used in contracts to implement ordering logic between + boxes relying on the exact "created before" relation between UTXOs. + ## Related Work To answer the question regarding prior work that proposes to move coin selection from @@ -612,13 +634,12 @@ discussions: **Research Papers and Proposals** - Various academic papers discuss coin selection - strategies, focusing on optimizing transaction costs and sizes. These discussions often - highlight the limitations of off-chain coin selection and suggest improvements, some of - which could potentially be integrated into the protocol layer. For example, the paper "A - Survey on Coin Selection Algorithms in UTXO-based Blockchains" provides a comprehensive - review and categorizes existing algorithms, potentially paving the way for on-chain - implementations [1]. + Various academic papers discuss coin selection strategies, focusing on optimizing + transaction costs and sizes. These discussions often highlight the limitations of + off-chain coin selection and suggest improvements, some of which could potentially be + integrated into the protocol layer. For example, the paper "A Survey on Coin Selection + Algorithms in UTXO-based Blockchains" provides a comprehensive review and categorizes + existing algorithms, potentially paving the way for on-chain implementations [1]. **UTXO Management and Coin Selection**: @@ -652,7 +673,7 @@ streamline the process of managing multiple DeFi simultaneous interactions for u Alice. By allowing transactions to specify input pools rather than fixed UTXOs, and ensuring each transaction includes at least one concrete input, users can engage in various DeFi protocols simultaneously without chaining transactions or managing UTXO -dependencies. This approach not only enhances transaction flexibility and efficiency but +dependencies. This approach not only enhances transaction flexibility and efficiency, but also prevents double-spending attacks, enabling a more secure and responsive DeFi experience on UTXO blockchain platform such as Ergo. From 2f4e73610ec417eb335c93893f4e3cc5c7027682 Mon Sep 17 00:00:00 2001 From: Alexander Slesarenko Date: Mon, 8 Jul 2024 18:04:53 +0100 Subject: [PATCH 10/12] eip47: move example before implementation details --- eip-0047.md | 226 ++++++++++++++++++++++++++-------------------------- 1 file changed, 113 insertions(+), 113 deletions(-) diff --git a/eip-0047.md b/eip-0047.md index 979f04aa..73d23656 100644 --- a/eip-0047.md +++ b/eip-0047.md @@ -14,9 +14,13 @@ - [The Solution Idea](#the-solution-idea) - [Proposed Solution](#proposed-solution) - [Key Components of the Solution](#key-components-of-the-solution) - - [Avoiding Double-Spending with PooledInputs](#avoiding-double-spending-with-pooledinputs) - - [Solution to Double-Spending Risk](#solution-to-double-spending-risk) + - [Double-Spending Problem with PooledInputs](#double-spending-problem-with-pooledinputs) + - [Solution to Double-Spending Problem](#solution-to-double-spending-problem) - [Wallet Standard for PooledInputs](#wallet-standard-for-pooledinputs) +- [Example: Alice's Multi-Step DeFi Workflow](#example-alices-multi-step-defi-workflow) + - [Context](#context) + - [Current Step-by-Step Workflow with explicit UTXO selection](#current-step-by-step-workflow-with-explicit-utxo-selection) + - [New step-by-step Workflow with Pooled Inputs](#new-step-by-step-workflow-with-pooled-inputs) - [Implementation Details](#implementation-details) - [PooledInput Data Structure](#pooledinput-data-structure) - [Transaction Format Extension](#transaction-format-extension) @@ -25,14 +29,10 @@ - [Ordered access to UTXO Storage](#ordered-access-to-utxo-storage) - [Obtaining UTXO Ordering Details](#obtaining-utxo-ordering-details) - [Adding and Removing UTXOs](#adding-and-removing-utxos) - - [Extension of UTXO Storage](#extension-of-utxo-storage) + - [Extension of UTXO Storage](#extension-of-utxo-storage) - [Selecting UTXOs for Unpooling](#selecting-utxos-for-unpooling) - [Change Output Handling](#change-output-handling) - [Transaction Concretization](#transaction-concretization) -- [Example: Alice's Multi-Step DeFi Workflow](#example-alices-multi-step-defi-workflow) - - [Context](#context) - - [Current Step-by-Step Workflow with explicit UTXO selection](#current-step-by-step-workflow-with-explicit-utxo-selection) - - [New step-by-step Workflow with Pooled Inputs](#new-step-by-step-workflow-with-pooled-inputs) - [Benefits of Pooled Transaction Inputs](#benefits-of-pooled-transaction-inputs) - [Related Work](#related-work) - [Conclusion](#conclusion) @@ -181,7 +181,7 @@ of transaction creation and execution on the Ergo blockchain. process will handle each `PooledInput` sequentially, in the order they appear in the list. -### Avoiding Double-Spending with PooledInputs +### Double-Spending Problem with PooledInputs While PooledInputs offer flexibility, they also introduce a potential risk (if not addressed) of double-spending. Specifically, when a transaction contains only @@ -208,7 +208,7 @@ level constraint on the usage of PooledInputs and describe a wallet level conven UTXO management that will make this proposal both safe and also practical to implement. More on this in the Proposed Solution section. -### Solution to Double-Spending Risk +### Solution to Double-Spending Problem To address the possibility of double-spending attacks with PooledInputs, we propose a constraint: _a transaction with PooledInputs must include at least one concrete input_. @@ -246,6 +246,110 @@ wallet standard (can be published as separate EIP motivated by this proposal). value (e.g., `R4 = 0x012345`). The wallet can identify available `PoolAccessBoxes` using (wallet's address, `R4 = 0x012345`, value == `PoolAccessBoxValue`). +## Example: Alice's Multi-Step DeFi Workflow +Let's create a detailed example involving Alice's participation in multiple DeFi +protocols, requiring her to sign transactions and manage her UTXOs efficiently using the +Pooled Transaction Inputs mechanism. + +### Context +Alice is an active participant in the DeFi ecosystem on the Ergo blockchain. She engages +in multiple protocols, including staking, yield farming, and automated trading. She needs +to handle these interactions efficiently to maximize her returns. + +### Current Step-by-Step Workflow with explicit UTXO selection + +1. **Initial Setup:** + - **Wallet:** Alice's wallet contains the following UTXOs: + UTXO1: 100 ERG + +2. **Staking:** + - **Event:** Alice decides to stake 40 ERG in a DeFi staking pool that periodically pays out rewards. + - **Transaction 1:** Alice creates a transaction to stake 40 ERG. + - **Inputs:** UTXO1 (100 ERG) + - **Outputs:** + - StakeOutput: 40 ERG to the staking pool + - ChangeOutput: 60 ERG back to Alice's wallet + - Wallet: UTXO2: 60 ERG (unconfirmed) + +3. **Yield Farming:** + - **Event:** Alice wants to participate in a yield farming protocol that requires 35 ERG. + - **Transaction 2:** Alice creates chained transaction to deposit 35 ERG into the yield farming pool + referencing the change output from the previous transaction. + - **Inputs:** UTXO2 (60 ERG) (unconfirmed) + - **Outputs:** + - YieldFarmOutput: 35 ERG to the yield farming pool + - ChangeOutput: 25 ERG back to Alice's wallet + - Wallet: UTXO3: 25 ERG (unconfirmed) + +4. **Automated Trading:** + - **Event:** Alice's automated trading bot detects a market signal and needs to execute a trade with 20 ERG. + - **Transaction 3:** The bot creates a transaction to execute the trade. + - **Inputs:** UTXO3 (25 ERG) (unconfirmed) + - **Outputs:** + - TradeOutput: 20 ERG to the trading pool + - ChangeOutput: 5 ERG back to Alice's wallet + - Wallet: UTXO4: 5 ERG (unconfirmed) + +Note, in order for Alice to execute the above transaction at the same time, they all need to be +chained together. This creates hard dependence between the transactions and requires all +the previous (parent) transactions to be successful for child transactions to also be +successful. This rules out any possibility for cancellation or transaction queueing on +DeFi protocol side. If Transaction 1 is cancelled/delayed by the DeFi protocol, then +the whole chain of transactions will be also cancelled/delayed. + +### New step-by-step Workflow with Pooled Inputs + +Note: +- to ensure the security and uniqueness of transactions involving PooledInputs, + Alice's wallet will include a `PoolAccessBox` in each transaction. +- many access boxes can be created initially to enable PooledInputs feature in the wallet +- the number of access boxes defines the maximum number of transactions that can be created + in parallel without using chaining. + +1. **Initial Setup**: + - **Wallet**: Alice's wallet contains the following UTXOs: + - UTXO1: 100 ERG + - PoolAccessBox1: 0.001 ERG (pre-defined constant `PoolAccessBoxValue`) + - PoolAccessBox2: 0.001 ERG + - PoolAccessBox3: 0.001 ERG + +2. **Staking**: + - **Transaction 1**: Alice creates a transaction to stake 40 ERG. + - **PooledInput**: + - Address: Alice's Wallet Address + - Value: 40 ERG + - Signature: Alice's Signature + - **Concrete Input**: PoolAccessBox1 (0.001 ERG) + - **Outputs**: + - StakeOutput: 40 ERG to the staking pool + - PoolAccessBox4: 0.001 ERG to Alice's wallet + +3. **Yield Farming**: + - **Transaction 2**: Alice creates a transaction to deposit 35 ERG into the yield farming pool. + - **PooledInput**: + - Address: Alice's Wallet Address + - Value: 35 ERG + - Signature: Alice's Signature + - **Concrete Input**: PoolAccessBox2 (0.001 ERG) + - **Outputs**: + - YieldFarmOutput: 35 ERG to the yield farming pool + - PoolAccessBox5: 0.001 ERG to Alice's wallet + +4. **Automated Trading**: + - **Transaction 3**: The bot creates a transaction to execute a trade with 20 ERG. + - **PooledInput**: + - Address: Alice's Wallet Address + - Value: 20 ERG + - Signature: Alice's Signature + - **Concrete Input**: PoolAccessBox3 (0.001 ERG) + - **Outputs**: + - TradeOutput: 20 ERG to the trading pool + - PoolAccessBox6: 0.001 ERG to Alice's wallet + - Wallet: UTXO: 5 ERG; PAB4: 0.001 ERG; PAB5: 0.001 ERG; PAB6: 0.001 ERG (unconfirmed) + +This workflow ensures that each transaction has a unique `messageToSign`, preventing +double-spending attacks while maintaining the flexibility and efficiency of PooledInputs. + ## Implementation Details ### PooledInput Data Structure @@ -496,110 +600,6 @@ as if it was a standard transaction fully constructed and signed off-chain. This means PooledInput mechanism is fully transparent to the smart contracts evaluation as the concretization happens before any contract is executed. -## Example: Alice's Multi-Step DeFi Workflow -Let's create a detailed example involving Alice's participation in multiple DeFi -protocols, requiring her to sign transactions and manage her UTXOs efficiently using the -Pooled Transaction Inputs mechanism. - -### Context -Alice is an active participant in the DeFi ecosystem on the Ergo blockchain. She engages -in multiple protocols, including staking, yield farming, and automated trading. She needs -to handle these interactions efficiently to maximize her returns. - -### Current Step-by-Step Workflow with explicit UTXO selection - -1. **Initial Setup:** - - **Wallet:** Alice's wallet contains the following UTXOs: - UTXO1: 100 ERG - -2. **Staking:** - - **Event:** Alice decides to stake 40 ERG in a DeFi staking pool that periodically pays out rewards. - - **Transaction 1:** Alice creates a transaction to stake 40 ERG. - - **Inputs:** UTXO1 (100 ERG) - - **Outputs:** - - StakeOutput: 40 ERG to the staking pool - - ChangeOutput: 60 ERG back to Alice's wallet - - Wallet: UTXO2: 60 ERG (unconfirmed) - -3. **Yield Farming:** - - **Event:** Alice wants to participate in a yield farming protocol that requires 35 ERG. - - **Transaction 2:** Alice creates chained transaction to deposit 35 ERG into the yield farming pool - referencing the change output from the previous transaction. - - **Inputs:** UTXO2 (60 ERG) (unconfirmed) - - **Outputs:** - - YieldFarmOutput: 35 ERG to the yield farming pool - - ChangeOutput: 25 ERG back to Alice's wallet - - Wallet: UTXO3: 25 ERG (unconfirmed) - -4. **Automated Trading:** - - **Event:** Alice's automated trading bot detects a market signal and needs to execute a trade with 20 ERG. - - **Transaction 3:** The bot creates a transaction to execute the trade. - - **Inputs:** UTXO3 (25 ERG) (unconfirmed) - - **Outputs:** - - TradeOutput: 20 ERG to the trading pool - - ChangeOutput: 5 ERG back to Alice's wallet - - Wallet: UTXO4: 5 ERG (unconfirmed) - -Note, in order for Alice to execute the above transaction at the same time, they all need to be -chained together. This creates hard dependence between the transactions and requires all -the previous (parent) transactions to be successful for child transactions to also be -successful. This rules out any possibility for cancellation or transaction queueing on -DeFi protocol side. If Transaction 1 is cancelled/delayed by the DeFi protocol, then -the whole chain of transactions will be also cancelled/delayed. - -### New step-by-step Workflow with Pooled Inputs - -Note: -- to ensure the security and uniqueness of transactions involving PooledInputs, -Alice's wallet will include a `PoolAccessBox` in each transaction. -- many access boxes can be created initially to enable PooledInputs feature in the wallet -- the number of access boxes defines the maximum number of transactions that can be created - in parallel without using chaining. - -1. **Initial Setup**: - - **Wallet**: Alice's wallet contains the following UTXOs: - - UTXO1: 100 ERG - - PoolAccessBox1: 0.001 ERG (pre-defined constant `PoolAccessBoxValue`) - - PoolAccessBox2: 0.001 ERG - - PoolAccessBox3: 0.001 ERG - -2. **Staking**: - - **Transaction 1**: Alice creates a transaction to stake 40 ERG. - - **PooledInput**: - - Address: Alice's Wallet Address - - Value: 40 ERG - - Signature: Alice's Signature - - **Concrete Input**: PoolAccessBox1 (0.001 ERG) - - **Outputs**: - - StakeOutput: 40 ERG to the staking pool - - PoolAccessBox4: 0.001 ERG to Alice's wallet - -3. **Yield Farming**: - - **Transaction 2**: Alice creates a transaction to deposit 35 ERG into the yield farming pool. - - **PooledInput**: - - Address: Alice's Wallet Address - - Value: 35 ERG - - Signature: Alice's Signature - - **Concrete Input**: PoolAccessBox2 (0.001 ERG) - - **Outputs**: - - YieldFarmOutput: 35 ERG to the yield farming pool - - PoolAccessBox5: 0.001 ERG to Alice's wallet - -4. **Automated Trading**: - - **Transaction 3**: The bot creates a transaction to execute a trade with 20 ERG. - - **PooledInput**: - - Address: Alice's Wallet Address - - Value: 20 ERG - - Signature: Alice's Signature - - **Concrete Input**: PoolAccessBox3 (0.001 ERG) - - **Outputs**: - - TradeOutput: 20 ERG to the trading pool - - PoolAccessBox6: 0.001 ERG to Alice's wallet - - Wallet: UTXO: 5 ERG; PAB4: 0.001 ERG; PAB5: 0.001 ERG; PAB6: 0.001 ERG (unconfirmed) - -This workflow ensures that each transaction has a unique `messageToSign`, preventing -double-spending attacks while maintaining the flexibility and efficiency of PooledInputs. - ## Benefits Here are the key benefits of the proposed Pooled Transaction Inputs mechanism: From 0836dd1eca323c6b5fd6b5172c27a465bd4449cd Mon Sep 17 00:00:00 2001 From: Alexander Slesarenko Date: Mon, 8 Jul 2024 18:12:00 +0100 Subject: [PATCH 11/12] eip47: diagrams removed --- eip-0047/entities.puml | 51 ----------------------------------------- eip-0047/er-diagram.uml | 0 2 files changed, 51 deletions(-) delete mode 100644 eip-0047/entities.puml delete mode 100644 eip-0047/er-diagram.uml diff --git a/eip-0047/entities.puml b/eip-0047/entities.puml deleted file mode 100644 index e8120da3..00000000 --- a/eip-0047/entities.puml +++ /dev/null @@ -1,51 +0,0 @@ -@startuml -entity Transaction { - +transactionId: String - +inputs: List - +outputs: List - +pools: List -} - -entity PooledInput { - +address: ErgoAddress - +filter: Option - +value: Long - +signature: Array -} - -entity UTXO { - +utxoId: String - +value: Long - +blockHeight: Int - +outputIndex: Int -} - -entity Wallet { - +walletId: String - +address: ErgoAddress - +utxos: List -} - -entity ErgoNode { - +nodeId: String -} - -entity Block { - +blockId: String - +previousBlockId: String - +blockHeight: Int - +transactions: List -} -entity Address { -} -Transaction "1" -- "0..*" PooledInput : pools -Transaction "1" -- "0..*" UTXO : inputs -Transaction "1" -- "1..*" UTXO : outputs -Wallet "1" -- "*" Address : has access to -Wallet "1..*" -- "1" ErgoNode : interacts with -ErgoNode "1" -- "*" Block : creates -ErgoNode "1" -- "*" Transaction : processes -Block "1" -- "1..*" Transaction : contains -PooledInput "*" -- "1" Address : spends from -UTXO "0..*" -- "1" Block : created in -@enduml diff --git a/eip-0047/er-diagram.uml b/eip-0047/er-diagram.uml deleted file mode 100644 index e69de29b..00000000 From e45b72956afb3c8141e7374824ea84363c517399 Mon Sep 17 00:00:00 2001 From: Alexander Slesarenko Date: Wed, 18 Dec 2024 12:06:38 +0000 Subject: [PATCH 12/12] eip47: typos and improvements --- eip-0047.md | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/eip-0047.md b/eip-0047.md index 73d23656..b14dc468 100644 --- a/eip-0047.md +++ b/eip-0047.md @@ -168,7 +168,8 @@ of transaction creation and execution on the Ergo blockchain. the remaining value (total value of selected UTXOs minus the `value` field of the `PooledInput`). This change output will be appended to the list of transaction outputs and the change amount will be returned to the address associated with the pool and - thus can be immediately consumed by the next transaction spending from the same pool. + thus can be consumed by the next transaction spending from the same pool (respecting FIFO + ordering). 7. **Transaction Concretization:** - Introduce the process of `input unpooling` during block @@ -349,6 +350,8 @@ Note: This workflow ensures that each transaction has a unique `messageToSign`, preventing double-spending attacks while maintaining the flexibility and efficiency of PooledInputs. +All transaction are independent and can be submitted simultaneously without the risk of +cancellation/delaying/queueing of the parent transaction affecting them all. ## Implementation Details @@ -358,15 +361,17 @@ double-spending attacks while maintaining the flexibility and efficiency of Pool case class PooledInput( address: SigmaProp, // defines the pool where UTXOs are selected from value: Long, // Total value to be spent from the pool - signature: Array[Byte] // Signature for the `address` sigma proposition + signature: Array[Byte] // Signature for the `address` sigma proposition proof ) ``` - - `SigmaProp` type: aka Sigma Proposition, specifies the source address of the UTXOs (in its simplest form it is P2PK, but can also be multisig and more complex sigma trees). + - `SigmaProp` type: aka Sigma Proposition, specifies the source address of the UTXOs (in + its simplest form it is P2PK, but can also be multisig and more complex [sigma trees + supported by Ergo](https://hackernoon.com/sigma-protocols-for-the-working-programmer)). - `value`: The total amount of ERGs to be covered by inputs spent from the pool. - `signature`: The signature for the `address` sigma proposition, ensuring the transaction is authorized to spend from the pool. - The `address` field defines the UTXO pool as a list of boxes as if returned by the following query: + The `address` field defines the UTXO _pool_ as a list of boxes as if returned by the following query: ```sql SELECT * FROM UTXO as U WHERE U.ergoTree = ErgoTree.fromSigmaProp(address) @@ -378,7 +383,7 @@ The ordering of the boxes in the pool is defined below. ```scala case class ErgoTransaction( inputs: Seq[Input], // List of concrete inputs to spend - dataInputs: Seq[DataInput], // List of data inputs + dataInputs: Seq[DataInput], // List of data inputs (readonly or reference inputs) pools: Seq[PooledInput], // List of pooled inputs (UTXO pools) to spend from outputs: Seq[BoxCandidate] // List of output boxes (not including change outputs created for each pooled input) ) @@ -393,10 +398,10 @@ The ordering of the boxes in the pool is defined below. then properly generated signatures has to be provided in the `signature` fields of the corresponding `PooledInput`s. - Technically, the format versioning can be done by reusing one of the counts (e.g. data - input count). When the count is greater than MaxArrayLength (e.g. 100000) then it means - the transaction contains `pools` field, otherwise the list is empty, which means the - transaction is indistinguishable from the current protocol version. - This is similar to how versioning is done in BlockTransactionsSerializer. + input count). When the count is greater than MaxArrayLength protocol constant + (e.g. 100000) then it means the transaction contains `pools` field, otherwise the list is empty, + which means the transaction is indistinguishable from the current protocol version. This + is similar to how versioning is done in BlockTransactionsSerializer. ### Validation and Processing Logic @@ -493,7 +498,7 @@ stateChanges.foreach { case (Remove(boxId), utxo) => utxo.addressOpt match { case Some(address) => - // When UTXO is spent from a pool, remove it to the indexDb + // When UTXO is spent from a pool, remove it from the indexDb val key = createKey(address, utxo.blockHeight, utxo.outputIndex) indexDb.delete(key) case _ =>